Control device, imaging mobile device, and control method

Abstract

Ensure that the subject does not move out of frame from the imaging range of the camera mounted on the moving object. [Solution] The control device controls an imaging mobile body 30, which has a mobile body 300 and an imaging means 320 mounted on the mobile body that performs imaging via an optical system including a rotatable optical member 430 that bends the light path from the subject. The control device generates control information for controlling the mobile body and controls the rotational position of the optical member based on the estimated posture of the mobile body after it has been controlled based on the control information.

Description

Technical Field

[0001] The present invention relates to the control of a moving body capable of imaging.

Background Art

[0002] A moving body such as a drone may be equipped with a camera for aerial photography or the like. For example, a camera can be held by a rotating mechanism (such as a gimbal) provided on the moving body, and the imaging range of the camera can be changed by operating the rotating mechanism.

[0003] Also, as disclosed in Patent Document 1, a camera having a bending optical system including a mirror can be used as the camera mounted on the moving body. By rotating the mirror in the bending optical system, the imaging range can be changed, and the influence on the movement and posture of the moving body due to the operation of the rotating mechanism as described above can be avoided.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a camera having a bending optical system is mounted on a moving body with a large change in posture during movement (flight), it is necessary to control the rotation of the mirror so that the subject does not frame out of the imaging range of the camera.

Means for Solving the Problems

[0006] One aspect of the present invention is a control device that controls an imaging mobile body having a mobile body and imaging means mounted on the mobile body and performing imaging via an optical system including a rotatable optical member that bends the optical path from a subject. The control device has a first control means for generating control information for controlling the mobile body and a second control means for controlling the rotational position of the optical member. The second control means is characterized by controlling the rotational position of the optical member based on the estimated posture of the mobile body after it has been controlled based on the control information. An imaging mobile body including the above control device also constitutes another aspect of the present invention.

[0007] Another aspect of the present invention, the control method, is applied to an imaging mobile body having a mobile body and an imaging means mounted on the mobile body that performs imaging via an optical system including a rotatable optical element that bends the optical path from a subject. The control method is characterized by comprising the steps of generating control information for controlling the mobile body and controlling the rotational position of the optical element based on the estimated posture of the mobile body after it has been controlled based on the control information. A program that causes a computer to execute processing according to the above control method also constitutes another aspect of the present invention. [Effects of the Invention]

[0008] According to the present invention, in an imaging mobile device that performs imaging using a bent optical system, it is possible to reduce the likelihood of the subject going out of frame due to changes in posture. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing the configuration of the imaging drone in this embodiment. [Figure 2] A flowchart illustrating the imaging process in the embodiment. [Figure 3] External view of the imaging drone in the embodiment. [Figure 4] A diagram showing the camera configuration in the embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Examples]

[0011] Figures 3(a) and 3(b) show the external appearance of the imaging drone 30 as an imaging mobile device, viewed from above and the side, respectively. The imaging drone 30 comprises a drone 300, which is a mobile flying object, and a camera 320, which is an imaging means. Figure 3(a) also shows a control device (remote control) 390 for the user to remotely control the drone 300 and the camera 320.

[0012] The drone 300 is equipped with propellers 310 at the ends of its four arms, and flies by rotating these propellers 310. The camera 320 is mounted on the underside of the aircraft. The detailed configuration of the camera 320 will be described later.

[0013] The transmitter 390 is a transmitter that sends instructions to the imaging drone 30 in response to user operations. These instructions include instructions regarding the flight of the drone 300, as well as instructions regarding the imaging conditions and imaging range of the camera 320. Alternatively, an automatic instruction device such as a personal computer that remotely controls the imaging drone 30 may be used instead of the transmitter 390.

[0014] Figure 4 shows the configuration of the camera 320. The camera 320 is positioned to extend downward from the body side of the drone 300, and has an image sensor 410 on the body side, with an imaging optical system below the image sensor 410. The image sensor 410 is a photoelectric conversion element such as a CCD sensor or CMOS sensor, and performs photoelectric conversion (imaging) of the subject image formed by the imaging optical system.

[0015] The imaging optical system is a bent optical system that includes a bending mirror 430 as an optical element that bends the optical path of incident light. Of the optical axes of the bent optical system (shown by dashed lines in the figure), the principal optical axis 400a on the image side of the bending mirror 430 is perpendicular to the imaging plane of the imaging sensor 410. Of the optical axes, the objective optical axis 400b on the object side of the bending mirror 430 is directed toward the subject as an object. The bending mirror 430 is rotatable around the pan axis and tilt axis, which will be described later. Note that the optical element that bends the optical path of incident light is not limited to a mirror, but may be other optical elements such as a prism.

[0016] Light from the subject, taken into the bent optical system through the objective window 460, is reflected by the bent mirror 430, passes through the zoom lens 420, and is imaged onto the image sensor 410. The zoom lens 420 changes the imaging angle of view of the camera 320 by moving along the principal optical axis 400a. The imaging angle of view can be changed within a range smaller than 360° horizontally and 180° vertically (for example, 30° to 45° horizontally, and 20° to 30° vertically). Note that the bent optical system does not necessarily have a zoom lens 420; it only needs to have a lens that images the light from the bent mirror 430.

[0017] The first driving means, the horizontal rotation mechanism 440, rotates the objective section 470, which includes the bent mirror 430, the second driving means, the vertical rotation mechanism 450, and the objective window 460, around a pan axis that coincides with the principal optical axis 400a. This changes the imaging range of the camera 320 in the horizontal direction, which is the first direction (panning). The vertical rotation mechanism 450 rotates the bent mirror 430 around a tilt axis that is perpendicular to the principal optical axis 400a and the objective optical axis 400b, so as to change the vertical angle between the principal optical axis 400a and the objective optical axis 400b. This changes the imaging range of the camera 320 in the vertical direction, which is the second direction (tilting).

[0018] FIG. 1 shows the electrical configuration of the imaging drone 30. The drone 300 has an flight control unit 110, a drive unit 120, an attitude sensor 140, an attitude estimator 150, a compensator 160, and an imaging control unit 180 inside the aircraft body. The camera 320 has the imaging sensor 410, the horizontal rotation mechanism 440, and the vertical rotation mechanism 450 described above.

[0019] In the drone 300, the flight control unit 110 as the first control means generates and outputs a flight control amount as control information for controlling the flight of the drone 300 in response to an instruction from the prop 390. The flight control amount is a control amount related to the rotation speed of the propeller 310, the attitude (tilt) of the drone 300, etc. The drive unit 120 flies the drone 300 by rotating the propeller 310 at a rotation speed corresponding to the flight control amount output from the flight control unit 110 or tilting the drone 300.

[0020] The attitude sensor 140 as the detection means is composed of an acceleration sensor, a gyro sensor, a GPS (Global Positioning System) sensor, etc., and detects the attitude of the drone 300. The flight control unit 110 outputs a flight control amount by PID feedback control using a signal corresponding to the detected attitude from the attitude sensor 140.

[0021] The attitude estimator 150 obtains and outputs an attitude estimation value indicating the estimated attitude of the drone 300 after flight control based on the flight control amount from the flight control unit 110. Here, after flight control based on the flight control amount, it includes when flight control is being performed based on the flight control amount. At this time, the flight control unit 110 may read out the attitude estimation value corresponding to the flight control amount from the flight control unit 110 from the table data storing the attitude estimation values for each of the plurality of flight control amounts. Also, the average of the detected attitudes obtained multiple times for the same past flight control amount may be used as the attitude estimation value.

[0022] The imaging control unit 180 outputs a rotation control amount for controlling the rotation of the horizontal rotation mechanism 440 and the vertical rotation mechanism 450 in the camera 320 (i.e., controlling the rotational position of the bending mirror 430). The imaging control unit 180 also controls imaging by the imaging sensor 410. The attitude estimation unit 150 and the imaging control unit 180 constitute a second control means. Furthermore, the flight control unit 110, attitude estimation unit 150, and imaging control unit 180 constitute a control device.

[0023] The compensator 160 adds a compensation value corresponding to the difference between the attitude estimate obtained by the attitude estimation unit 150 and the actual attitude detected by the attitude sensor 140 to the horizontal rotation control amount and vertical rotation control amount output from the imaging control unit 180 to the horizontal rotation mechanism 440 and vertical rotation mechanism 450, respectively. This corrects the change in the imaging range of the camera 320 caused by the attitude change of the drone 300.

[0024] The flowchart in Figure 2 shows the processing (control method) mainly performed by the flight control unit 110, attitude estimation unit 150, compensator 160, and imaging control unit 180 in this embodiment. The flight control unit 110, attitude estimation unit 150, compensator 160, and imaging control unit 180 are configured as computers including a CPU, and they perform this processing according to the program.

[0025] This section describes a case where, in a forward-flying imaging drone 30, the user specifies a hemispherical imaging range of 360° horizontally and 180° vertically downwards via the remote control 390 to start imaging. The horizontal rotation mechanism 440 rotates the objective unit 470 (i.e., the retractable mirror 430) 360° around the pan axis. The vertical rotation mechanism 450 rotates the retractable mirror 430 around the tilt axis between a position where the objective optical axis 400b is perpendicular to the principal optical axis 400a and a position where the reflected light from the subject reaches the imaging surface of the imaging sensor 410 at its limit. As a result, the camera 320 can perform 360° horizontal imaging and hemispherical imaging of 180° vertically downwards. In this case, the camera 320 may obtain a hemispherical image by combining multiple still images obtained by taking still images at multiple imaging positions (for example, at 10° intervals) within each of the 360° horizontal and 180° vertical downward imaging ranges, or it may continuously perform video imaging.

[0026] First, in step S101, the flight control unit 110 receives a forward command from the transmitter 390.

[0027] Next, in step S102, the flight control unit 110 outputs a flight control amount that controls the drone 300 to tilt forward in order to make it fly forward. As a result, the drone 300 starts to tilt forward in accordance with the flight control amount output in step S102.

[0028] As this forward tilt begins, in step S103, the attitude estimation unit 150 outputs an attitude estimate (for example, a forward tilt angle of 10°) obtained from the flight control amount for forward tilt (attitude change) output in step S102.

[0029] Furthermore, in step S104, the compensator 160 outputs a compensation amount corresponding to the difference between the attitude estimate from the attitude estimation unit 150 and the actual attitude detected by the attitude sensor 140.

[0030] Then, in step S105, the imaging control unit 180 adds the compensation amount output in step S104 to the horizontal rotation control amount and vertical rotation control amount corresponding to the multiple imaging positions described above, and outputs this to the horizontal rotation mechanism 440 and the vertical rotation mechanism 450. At this point, the drone 300 completes its forward tilt.

[0031] This controls the horizontal and vertical rotational position of the bent mirror 430 to match the forward tilt angle of the drone 300, thereby preventing the subject from going out of frame within the imaging range. Furthermore, by controlling the horizontal and vertical rotation of the bent mirror 430, high-resolution 360-degree imaging and hemispherical imaging can be performed.

[0032] In this embodiment, the rotational position of the flexed mirror 430 is controlled based on the estimated attitude of the drone 300 after flight control (attitude change) obtained from the flight control amount (control information) of the drone 300, and the attitude detected by the attitude sensor 140.

[0033] In this embodiment, the rotational position of the flexible mirror 430 is not only controlled in response to changes in the attitude of the drone 300 during flight. For example, the rotational position of the flexible mirror 430 is controlled according to the difference between the estimated attitude value (e.g., tilt of 0°) during hovering control in a fixed position in the air and the detected attitude of the drone 300 that has been shaken by external factors such as wind. This prevents the subject from going out of frame from the imaging range even if the attitude of the drone 300 changes due to external factors.

[0034] By adopting a configuration in which the objective lens of the refractive optical system and a small part of the bent mirror are rotated, as in this embodiment, it is possible to avoid the disadvantages of the reaction force generated by operating a mechanism that rotates the entire camera, such as a gimbal, which can affect the flight and attitude of the drone.

[0035] In the above embodiment, we described a case where the rotation position of the flexed mirror is controlled based on the difference between the estimated attitude and the detected attitude of the drone. However, the rotation position of the flexed mirror may also be controlled based solely on the estimated attitude of the drone. For example, such control is possible when there is little difference between the estimated attitude and the actual attitude of the drone, and the previous estimated attitude can be considered as the actual attitude before the current flight control.

[0036] In the above embodiment, the case in which the control device, which consists of an imaging control unit 180 and a flight control unit 110, is built into the imaging drone 30 was described, but the control device may be provided outside the imaging drone. In this case, a personal computer or remote control capable of communicating with the imaging drone can be used as the control device, and the control device transmits the generated flight control amount and rotation control amount to the imaging drone.

[0037] The above embodiments include the following configuration. (Composition 1) A control device for controlling an imaging mobile body, which has a mobile body and imaging means mounted on the mobile body and performing imaging via an optical system including a rotatable optical member that bends the optical path from a subject, A first control means for generating control information for controlling the moving body, It has a second control means for controlling the rotational position of the optical member, The control device is characterized in that the second control means controls the rotational position of the optical member based on the estimated posture of the moving body after it has been controlled based on the control information. (Configuration 2) The system has detection means for detecting the posture of the moving body, The control device according to configuration 1, characterized in that the second control means controls the rotational position of the optical member based on the difference between the estimated posture and the posture detected by the detection means. (Composition 3) The control device according to configuration 1 or 2, characterized in that the first control means generates the control information in response to an instruction input from an instruction device that is operated by a user or that automatically controls the moving body. (Composition 4) The control device according to any one of configurations 1 to 3, characterized in that the second control means controls a first drive means for rotating the optical member in a first direction and a second drive means for rotating the optical member in a second direction different from the first direction. (Composition 5) The control device according to configuration 4, characterized in that the second control means rotates the optical member 360° in the first direction to cause the imaging means to perform 360° imaging. (Composition 6) The control device according to any one of configurations 1 to 5, characterized in that the moving body is an aerial vehicle. (Composition 7) The control device according to configuration 6, characterized in that the first control means controls the aircraft to tilt based on the control information. (Composition 8) A control device as described in any one of configurations 1 to 7, The aforementioned moving body, An imaging mobile body characterized by having the aforementioned imaging means.

[0038] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0039] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of Symbols]

[0040] 30 Imaging Drones 110 Flight Control Unit 140 Attitude Sensors 150 Posture estimation section 180 Imaging Control Unit 300 Drones 320 Camera

Claims

1. A control device for controlling an imaging mobile body, which has a mobile body and imaging means mounted on the mobile body and performing imaging via an optical system including a rotatable optical member that bends the optical path from a subject, A first control means for generating control information for controlling the moving body, It has a second control means for controlling the rotational position of the optical member, The control device is characterized in that the second control means controls the rotational position of the optical member based on the estimated posture of the moving body after it has been controlled based on the control information.

2. The system has detection means for detecting the posture of the moving body, The control device according to claim 1, characterized in that the second control means controls the rotational position of the optical member based on the difference between the estimated posture and the posture detected by the detection means.

3. The control device according to claim 1, characterized in that the first control means generates the control information in response to an instruction input from an instruction device that is operated by a user or that automatically controls the moving body.

4. The control device according to claim 1, characterized in that the second control means controls a first drive means for rotating the optical member in a first direction and a second drive means for rotating the optical member in a second direction different from the first direction.

5. The control device according to claim 4, characterized in that the second control means rotates the optical member 360° in the first direction to cause the imaging means to perform 360° imaging.

6. The control device according to claim 1, characterized in that the moving body is an aircraft.

7. The control device according to claim 6, characterized in that the first control means controls the aircraft to tilt based on the control information.

8. A control device according to any one of claims 1 to 7, The aforementioned moving body, An imaging mobile body characterized by having the aforementioned imaging means.

9. A control method for controlling an imaging mobile body having a mobile body and imaging means mounted on the mobile body and performing imaging via an optical system including a rotatable optical element that bends the optical path from a subject, A step of generating control information for controlling the moving body, A control method characterized by comprising the step of controlling the rotational position of the optical member based on the estimated posture of the moving body after it has been controlled based on the control information.

10. A program characterized by causing a computer to execute a process according to the control method described in claim 9.

Images