Imaging device and its control method, imaging system

Abstract

The present invention provides an imaging device capable of acquiring smoother display images with suppressed image distortion. [Solution] The imaging device 100 can be connected to the mobile body 10 via the gimbal 50, and can take pictures while moving on the mobile body 10. The image sensor 150 is an image sensor in which exposure time differences occur for each pixel row, and can be moved in a direction parallel to the imaging plane by the image sensor drive device 220. Based on the movement information of the mobile body 10 and the imaging device 100, the control unit 180 controls the movement of the image sensor 150 by the image sensor drive device 220 when taking still images and when taking live view images. The control unit 180 controls the amount of movement of the image sensor during exposure when taking still images and when taking live view images.

Description

Technical Field

[0001] The present invention relates to a technique for imaging with a movable imaging device.

Background Art

[0002] Among mobile imaging devices having imaging means, there are drones that can be remotely controlled by a user and unmanned flying objects called UAVs (Unmanned Aerial Vehicles). For example, in drone mapping using a drone equipped with an imaging device, surveying is performed by taking still images. In still image shooting for surveying, while the drone is flying, the shooting lens is directed at the survey target and continuous shooting is performed, so it is possible to improve the efficiency of surveying.

[0003] By the way, the rolling shutter method adopted for an imaging element is a method of performing exposure in order from the upper row and reading out the signal that has been photoelectrically converted. Therefore, there is a possibility of generating image distortion called rolling shutter distortion. Patent Document 1 discloses a correction technique for rolling shutter distortion when the operation speed of panning is non-uniform. Based on the detected camera shake component, control is performed to correct the distortion generated in the image of the subject due to the rolling shutter method using a shake correction means.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, Patent Document 1 does not contain any description regarding the shooting of display images (hereinafter also referred to as live view images) that are output in real time between the shooting of multiple still images. Furthermore, Patent Document 1 does not contain any description regarding the correction of rolling shutter distortion for live view images. If rolling shutter distortion is not corrected for live view images, the image will be distorted. If the amount of drive (movement) of the image sensor becomes too large when outputting live view images, the frame rate of the live view images may decrease. In still image shooting for surveying using drones capable of high-speed flight, users often check the flight status of the drone remotely using live view images, and smooth images at a higher frame rate are required in addition to the correction of rolling shutter distortion. The present invention aims to provide an imaging device capable of acquiring smoother display images with suppressed image distortion. [Means for solving the problem]

[0006] The apparatus of the embodiment of the present invention includes an acquisition means for acquiring movement information relating to the movement of an imaging device, a driving means for moving an image sensor or a lens for image blur correction in a direction parallel to the imaging plane, and a control means for controlling the movement of the image sensor or lens by the driving means according to the movement information during a first shooting to acquire a still image and a second shooting to acquire a display image. When the imaging device is moving while shooting, during the first shooting, the control means calculates a first drive amount and controls the movement of the image sensor or lens by the driving means, and during the second shooting, the control means calculates a second drive amount and controls the movement of the image sensor or lens by the driving means. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an imaging device that can acquire smoother display images with suppressed image distortion. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an imaging system according to an embodiment. [Figure 2] This figure shows the external appearance of the imaging device according to the present invention. [Figure 3] This is a block diagram showing a part of the configuration of the imaging system according to the embodiment. [Figure 4] This block diagram shows other parts of the configuration of the imaging system according to the embodiment. [Figure 5] This is a timing chart illustrating the exposure of the image sensor according to the first embodiment. [Figure 6] This is a flowchart illustrating the process in the first embodiment. [Figure 7] This is a schematic diagram illustrating the movement path of a moving object. [Figure 8] This figure shows a moving object during filming in the embodiment. [Figure 9] This is a schematic diagram illustrating the occurrence of rolling shutter distortion. [Figure 10] This is a timing chart illustrating the distortion correction process in the first embodiment. [Figure 11] This is a timing chart illustrating the distortion correction process in the second embodiment. [Figure 12] This is a timing chart illustrating the exposure of the image sensor according to the third embodiment. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the embodiments, an example is shown in which the mobile imaging device is applied to a surveying aircraft such as a drone, but the present invention is applicable to various imaging systems that have imaging means capable of taking images while moving.

[0010] [First Embodiment] Figure 1 is an external perspective view showing the imaging system 1 of this embodiment. The imaging system 1 consists of a mobile body 10, a gimbal 50, a remote control device 600, and an imaging device 100. The mobile body 10 is an unmanned aerial vehicle (a so-called drone), and the photographer can remotely control the mobile body 10 using the remote control device 600. The mobile body 10 is, for example, a quadcopter equipped with four fan blades. Note that the number of fan blades can be one or more. Furthermore, the mobile body 10 is not limited to a rotary-wing aircraft, but may be configured as a fixed-wing aircraft.

[0011] The gimbal 50 is a connecting device that connects the mobile body 10 and the imaging device 100. The gimbal 50 can rotate and support the imaging device 100 in three axial directions (roll direction, yaw direction, and pitch direction) with respect to the optical axis of the imaging device 100. In addition, various signals and information are transmitted and received between the mobile body 10 and the imaging device 100 via the gimbal 50.

[0012] The remote control device 600 is a controller that remotely controls the mobile body 10 according to the operator's instructions. The remote control device 600 has a display unit 602 and an operation unit 603. The display unit 602 displays setting information for the mobile body 10 and the imaging device 100, position information and movement information of the mobile body 10, still images (recording images) and display images (live view images) captured by the imaging device 100, etc. For example, the operator, who is the user, can understand the control status and check the images by visually checking the displayed image information on the display screen of the display unit 602.

[0013] The control unit 603 has multiple control members operated by the photographer. These include control members for controlling the movement of the mobile body 10, and control members for changing the settings of the mobile body 10 and the imaging device 100. For example, the control unit 603 is composed of multiple buttons and joysticks, and has switches for operating the power of the remote control device 600. The photographer can issue a command to the mobile body 10 to start moving by operating the control unit 603.

[0014] Referring to FIG. 2, the imaging device 100 will be described in detail. FIG. 2 is an external perspective view showing the appearance of the imaging device 100. FIG. 2(A) is an upper front perspective view of the imaging device 100. FIG. 2(B) is a lower rear perspective view of the imaging device 100. The imaging device 100 is, for example, an interchangeable-lens digital camera, and an interchangeable lens 500 can be attached to and detached from its main body. Each part will be described by defining the subject side as the front side. Note that although an imaging system in which the interchangeable lens 500 can be attached to and detached from the main body of the imaging device 100 is illustrated, an imaging system in which a lens is fixed to the main body of the imaging device 100 may also be used.

[0015] The interchangeable lens 500 is a lens unit that can be attached to the front of the main body of the imaging device 100. An imaging optical system having a lens built into the interchangeable lens 500 can form a subject image on an imaging element (FIG. 3: 150) described later. For example, the interchangeable lens 500 is a single-focus lens unit with a focal length of 50 mm. The user may attach a lens unit with a different focal length to the main body of the imaging device 100 according to the shooting situation, or may attach a zoom lens unit with a variable focal length to the main body of the imaging device 100.

[0016] As shown in FIG. 2(B), an imaging device-side interface group 105 is installed on the back of the main body of the imaging device 100. Hereinafter, the interface will be denoted as "IF". The imaging device-side IF group 105 includes a video output terminal 110, a power input terminal 120, a shooting signal input terminal 130, and a movement information input / output terminal 140.

[0017] The video output terminal 110 is a terminal that outputs a video signal indicating the set value of the imaging device 1, a captured still image signal, and a captured live view video signal. For example, the video output terminal 110 is configured as an HDMI (registered trademark, HDMI High-Definition Multimedia Interface) terminal. Alternatively, the video output terminal 110 may be configured as a USB (Universal Serial Bus) Type-C terminal.

[0018] The power input terminal 120 is a terminal that receives power transmitted from the mobile unit 10 via the gimbal 50. The power input terminal 120 is not a general-purpose terminal or a terminal conforming to standards such as USB, but is configured as a terminal dedicated to the imaging device 100.

[0019] The shooting signal input terminal 130 is a terminal that receives a shooting start signal transmitted from the mobile body 10 via the gimbal 50. The imaging device 100 takes still images and live view images when it receives the shooting start signal from the mobile body 10. For example, the shooting signal input terminal 130 is configured as a terminal for a 3.5mm mini plug.

[0020] The movement information input / output terminal 140 is a terminal that transmits and receives movement information between the mobile body 10 and the main body of the imaging device 100 via the gimbal 50. Movement information will be described later.

[0021] The imaging device 100 has a movable body fixing part 155. The movable body fixing part 155 is a screw fixing part for fixing the imaging device 100 to the gimbal 50, and is configured as, for example, a tripod screw. In this case, it is possible to fix it to a device that has a mounting part corresponding to a tripod screw in addition to the gimbal 50.

[0022] The main body of the imaging device 100 has a media cover 145 on its rear. The media cover 145 is openable and closable, allowing the user to open the media cover 145 and insert or remove the recording media (Figure 3:251) described later.

[0023] Next, the imaging system 1 will be described with reference to Figures 3 and 4. Figure 3 is a block diagram showing the configuration of the imaging device 100 and gimbal 50 in the imaging system 1. Figure 4 is a block diagram showing the configuration of the mobile body 10 and remote control device 600 in the imaging system 1.

[0024] The main body of the imaging device 100 (Figure 3) can accommodate interchangeable lenses 500. The main body of the imaging device 100 includes an image sensor 150, an image sensor drive unit 220, a control unit 180, a memory unit 190, a recording media slot 250, a recording media 251, and an imaging device side IF group 105.

[0025] The image sensor 150 is a device having multiple photoelectric conversion elements, and is configured, for example, as a CMOS (complementary metal-oxide-semiconductor) type image sensor. The image sensor 150 converts the subject image formed by the interchangeable lens 500 into an electrical signal using photoelectric conversion and outputs it to the control unit 180. The image sensor drive device 220 consists of a blur detection unit 170, an image sensor movable unit 160, and an image sensor position detection unit 230. The image sensor drive device 220 is capable of driving the image sensor 150 for movement and rotation.

[0026] The blur detection unit 170 has a sensor that detects vibrations applied to the imaging device 100 and outputs a detection signal to the control unit 180. For example, the blur detection unit 170 has an acceleration sensor or an angular velocity sensor such as a gyroscope.

[0027] The movable part 160 of the image sensor has a drive mechanism that is movable relative to the optical axis of the interchangeable lens 500. For example, the control unit 180 can acquire the detection signal from the blur detection unit 170 and, based on the detected value, can control the movement of the image sensor 150 in a direction parallel to the imaging plane. The range of motion of the image sensor 150 is limited by the configuration of the movable part 160 of the image sensor, so there is a limit to the range of motion.

[0028] The image sensor position detection unit 230 detects the position of the image sensor 150 and outputs a position detection signal to the control unit 180. The image sensor position detection unit 230 has a sensor that detects the current position of the image sensor 150 within the movable range of the image sensor 150 driven by the drive mechanism of the image sensor movable part 160.

[0029] With the above configuration, the image sensor drive unit 220 can compensate for the effect on the image caused by vibrations applied to the imaging device 100 by mechanically driving (moving or rotating) the image sensor 150. Details of the method for correcting rolling shutter distortion by the image sensor drive unit 220 will be described later.

[0030] The control unit 180 (Figure 3) controls the imaging device 100. For example, the control unit 180 has a microprocessor and executes various processes according to the program. The memory unit 190 is an information storage device that stores setting information for the imaging device 100 and also stores programs that have been loaded in advance.

[0031] The recording media slot 250 is an IF unit connected to the recording media 251 that transmits still image signals. The recording media 251 is an information storage device that stores data of still images captured by the imaging device 100. For example, the recording media 251 is configured as an SD card, and the recording media slot 250 is an SD card slot.

[0032] The gimbal 50 (Figure 3) has a roll movable part 51, a yaw movable part 52, and a pitch movable part 53. The roll movable part 51 is a rotational drive unit that rotates the imaging device 100 in the roll direction. The yaw movable part 52 is a rotational drive unit that rotates the imaging device 100 in the yaw direction. The pitch movable part 53 is a rotational drive unit that rotates the imaging device 100 in the pitch direction. Cables connecting each terminal of the imaging device side IF group 105 and each terminal of the mobile body side IF group 11 (Figure 4), which will be described later, are installed inside the gimbal 50. Although this example illustrates an imaging system in which the main body of the imaging device 100 is detachably fixed to the gimbal 50, an imaging system in which the main body of the imaging device 100 is not detachably fixed to the gimbal 50 may also be used.

[0033] The mobile unit 10 (Figure 4) includes a mobile unit control unit 21, a mobile unit memory unit 16, a mobile unit side IF group 11, a battery 12, a drive unit 13, a position acquisition unit 14, a wireless communication unit 15, and a mobile unit operation unit 25.

[0034] The mobile unit control unit 21 controls the mobile unit 10. For example, the mobile unit control unit 21 has a microprocessor and executes various processes on the mobile unit 10 according to the program. The mobile unit memory unit 16 is an information storage device that stores setting values ​​and programs that have been loaded into the mobile unit 10 in advance. The mobile unit memory unit 16 stores information about the movement path of the mobile unit 10, which will be described later.

[0035] The mobile body-side IF group 11, connected to the mobile body control unit 21, has a video input terminal 17, a power output terminal 18, a shooting signal output terminal 19, and a movement information input / output terminal 20. The video input terminal 17 is paired with the video output terminal 110 of the imaging device-side IF group 105, and these terminals are connected by a cable. The power output terminal 18 is paired with the power input terminal 120 of the imaging device-side IF group 105, and these terminals are connected by a cable. The shooting signal output terminal 19 is paired with the shooting signal input terminal 130 of the imaging device-side IF group 105, and these terminals are connected by a cable. The mobile body-side movement information input / output terminal 20 is paired with the movement information input / output terminal 140 of the imaging device-side IF group 105, and these terminals are connected by a cable. The movement information input / output terminal 20 is equivalent to the movement information input / output terminal 140. Battery 12 is a rechargeable battery that supplies power to each component of the mobile body 10 and can also store power through charging. Power from battery 12 is also supplied to the gimbal 50. Power from battery 12 is also supplied to the imaging device 100 via a cable from the power output terminal 18 to the power input terminal 120.

[0036] The drive unit 13 is a propulsion device that moves the mobile body 10 by rotating the fan blades. The drive unit 13 is controlled by the mobile body control unit 21. The position acquisition unit 14 acquires the position information of the mobile body 10 and outputs it to the mobile body control unit 21. The position acquisition unit 14 is a device that acquires position information, for example, based on GNSS (Global Navigation Satellite System). The mobile body control unit 21 acquires the position information of the mobile body 10 from the position acquisition unit 14, performs calculations, and calculates information on the moving speed and direction of movement of the mobile body 10. The position information also includes altitude information of the mobile body 10.

[0037] The wireless communication unit 15 is a wireless device that communicates wirelessly with the wireless communication unit 604 located on the remote control device 600 to send and receive various information and signals. The wireless communication unit 15 is controlled by the mobile control unit 21.

[0038] The mobile unit control panel 25 includes a power switch for the mobile unit 10, as well as operating elements such as buttons and dials. The user can instruct the mobile unit 10 to change various settings by operating the mobile unit control panel 25.

[0039] The remote control device 600 (Figure 4) includes a control unit 605, a wireless communication unit 604, a battery 601 for the remote control device, a display unit 602, and an operation unit 603. The control unit 605 of the remote control device 600 has, for example, a microprocessor and executes various processes in the remote control device 600 according to a program. The wireless communication unit 604 can transmit and receive various information and signals by wirelessly communicating with the wireless communication unit 15 of the mobile body 10. The battery 601 is a secondary battery that supplies power to each component of the remote control device 600 and can also store power by charging. The display unit 602 and the operation unit 603 are as described above. Although the remote control device 600 is exemplified as a controller that remotely controls the mobile body 10 according to the operator's instructions, it may also be a remote control device that transmits control information such as pre-programmed movement patterns to the mobile body 10 to move the mobile body 10.

[0040] Next, we will explain the exposure of the image sensor 150 with reference to Figure 5. Figure 5 is a timing chart for explaining the exposure of the image sensor 150. The horizontal axis is the time axis, and the vertical axis shows the readout rows of the image sensor 150 from the first row to the nth row, in order from top to bottom. The time difference between adjacent rows in the rolling shutter method is denoted as Δt. The length of the longer side of the rectangle corresponding to each readout row corresponds to the exposure time.

[0041] The image sensor 150 has multiple photoelectric conversion elements arranged in a matrix. When exposure begins, exposure starts for the pixels corresponding to the first row of pixels on the image sensor 150. Then, after a time elapsed of Δt, exposure starts for the pixels corresponding to the second row of pixels. The time difference between the second and third rows, and the time difference between the third and fourth rows, are all Δt. For example, if the final readout row is the nth row, the exposure time difference (denoted as t) between the first and nth rows is "t = n·Δt". Rolling shutter distortion occurs due to the exposure time difference t, which is known as curtain speed. Figure 5 illustrates a configuration in which the image sensor 150 is read out sequentially from the first row to the nth row, but a configuration in which the image sensor 150 is read out sequentially from the nth row to the first row is also possible. The type of rolling shutter distortion that occurs will be described later.

[0042] Referring to Figure 6, the process of the mobile body 10 moving while the imaging device 100 captures still images for surveying will be described. Figure 6 is a flowchart illustrating the flow of this process. In S100, the power switch of the mobile body 10 is operated and power is turned on. The mobile body control unit 21 executes the startup process for the mobile body 10. Power is supplied to the imaging device 100 via the power output terminal 18 and the power input terminal 120, and the imaging device 100 starts up. Then, the process proceeds to S101.

[0043] In S101, the control unit 605 of the remote control device 600 determines whether the power switch of the remote control device 600 has been operated. If it is determined that the power switch of the remote control device 600 has been operated and power has been turned on, the process proceeds to S102. If it is determined that power has not been turned on, the process proceeds to S115.

[0044] In S102, the control unit 605 of the remote control device 600 determines whether or not rolling shutter distortion correction for the live view image is set to ON, based on the operation instructions from the photographer to the operation unit 603. Hereinafter, the live view image will also be referred to as "LV image," and the rolling shutter distortion correction will also be referred to as "RS distortion correction." If it is determined that RS distortion correction is set to ON, the process proceeds to S103. If it is determined that RS distortion correction is not set to ON, the process proceeds to S104.

[0045] In S103, the control unit 605 of the remote control device 600 determines whether the high frame rate mode for LV video is set to ON based on the operation instructions from the photographer to the operation unit 603. The unit of frame rate is "frames per second" and is denoted as "fps". The high frame rate mode is a mode in which the fps value exceeds a predetermined threshold. If it is determined that the high frame rate mode is set to ON, the process proceeds to S105. If it is determined that the high frame rate mode is not set to ON, the process proceeds to S106. In S105, the control unit 605 of the remote control device 600 transmits a first signal via the wireless communication unit 604 to indicate that the high frame rate mode setting for LV video is ON. The wireless communication unit 15 of the mobile unit 10 receives the first signal and notifies the mobile unit control unit 21 of the setting ON information. The mobile unit control unit 21 stores the setting ON information in the mobile unit memory unit 16. Next, the process proceeds to S107.

[0046] In S106, the control unit 605 of the remote control device 600 transmits a second signal via the wireless communication unit 604 to indicate that the high frame rate mode setting for LV video is OFF. The wireless communication unit 15 of the mobile unit 10 receives the second signal and notifies the mobile unit control unit 21 of the OFF setting information. The mobile unit control unit 21 stores the OFF setting information in the mobile unit memory unit 16. Next, the process proceeds to S107.

[0047] In S104, the control unit 605 of the remote control device 600 transmits a third signal via the wireless communication unit 604 to indicate that the RS distortion correction setting for the LV video is OFF. The wireless communication unit 15 of the mobile unit 10 receives the third signal and notifies the mobile unit control unit 21 of the OFF setting information. The mobile unit control unit 21 stores the OFF setting information in the mobile unit memory unit 16. Next, the process proceeds to S107.

[0048] In S107, the control unit 605 of the remote control device 600 determines whether or not a movement start operation has been performed on the moving object 10 based on the operation instruction from the photographer on the operation unit 603. If it is determined that a movement start operation has been performed, the process proceeds to S108. If it is determined that a movement start operation has not been performed, the process proceeds to S115. Note that if the moving object 10 moves based on a pre-programmed movement pattern, a determination may also be made in S107 as to whether or not the movement start time has arrived.

[0049] In S108, the process of starting the movement of the mobile body 10 is executed. The control unit 605 of the remote control device 600 transmits a movement start signal via the wireless communication unit 604. The wireless communication unit 15 of the mobile body 10 receives the movement start signal and notifies the mobile body control unit 21. The mobile body control unit 21 starts the movement of the mobile body 10. The movement control of the mobile body 10 is automatically performed based on the movement path information recorded in advance in the mobile body memory unit 16. The movement path of the mobile body 10 will be described later with reference to Figure 7.

[0050] Following S108, in S109, the mobile body control unit 21 determines whether the mobile body 10 has reached the shooting start point based on the output result of the position acquisition unit 14. If it is determined that the mobile body 10 has reached the shooting start point, the process proceeds to S110. If it is determined that the mobile body 10 has not reached the shooting start point, the process returns to S108 and continues.

[0051] In S110, still image capture for surveying the survey area begins. The imaging device 100 comprehensively captures still images within the predetermined survey area. After all the processing shown in Figure 6 is completed, the captured still images are stitched together to generate a single image of the survey area. This image is the output of the survey acquired by drone mapping. The process proceeds from S110 to S111.

[0052] In S111, the mobile body control unit 21 determines whether the mobile body 10 has reached the end point of the shooting process based on the output result of the position acquisition unit 14. If it is determined that the mobile body 10 has reached the end point of the shooting process, the process proceeds to S112. If it is determined that the mobile body 10 has not reached the end point of the shooting process, the process returns to S110 and continues.

[0053] In S112, the mobile body control unit 21 executes a process to move the mobile body 10 toward the starting point, and then proceeds to process S113. In S113, the mobile body control unit 21 determines whether the mobile body 10 has reached the starting point based on the output result of the position acquisition unit 14. If it is determined that the mobile body 10 has reached the starting point, the process proceeds to S114. If it is determined that the mobile body 10 has not reached the starting point, the process returns to S112 and continues.

[0054] In S114, the mobile unit control unit 21 performs the process of landing the mobile unit 10 and stopping the drive unit 13. Next, in S115, the mobile unit control unit 21 determines whether or not the power to the mobile unit 10 has been cut off. If it is determined that the power to the mobile unit 10 has been cut off, the process proceeds to S116 and ends. If it is determined that the power is still on, the process returns to S101 and continues.

[0055] The movement path of the mobile body 10 will be explained with reference to Figure 7. Figure 7 is a schematic diagram showing an example of the movement path of the mobile body 10. The movement path line 700 is a line that shows the movement path of the mobile body 10. For convenience, it is shown as a plan view, but the actual movement of the flying mobile body 10 is three-dimensional. The movement start point 701, indicated by the × mark, is the point where the mobile body 10 begins to move. The user at the movement start point 701 will monitor the movement of the mobile body 10 using the remote control device 600.

[0056] The survey area 702, shown by a dashed rectangular frame in Figure 7, is the area where still images are taken for surveying purposes. The group of shooting points 703 within the survey area 702 is a set of shooting points where still images are taken by the imaging device 100. Each shooting point constituting the group of shooting points 703 is shown as a black dot. The shooting start point 704 indicates the position just before the mobile body 10 enters the survey area 702. The shooting end point 705 indicates the position just after the mobile body 10 leaves the survey area 702.

[0057] In Figure 7, after the mobile body 10 passes the shooting end point 705, it returns towards the movement start point 701. The movement path information includes information from when the mobile body 10 takes off from the movement start point 701 and returns to the movement start point 701 and lands there again. The process in S108 (Figure 6) corresponds to the process in which the mobile body 10 takes off from the movement start point 701 and moves to the shooting start point 704 while flying. In S110, the imaging device 100 comprehensively takes still images at each shooting point over the survey range 702. In S111, the mobile body control unit 21 determines whether the mobile body 10 has reached the shooting end point 705 based on the output result of the position acquisition unit 14. If it is determined that the mobile body 10 has reached the shooting end point 705, in S112 the mobile body control unit 21 performs the process of moving the mobile body 10 towards the movement start point 701. In S113, if it is determined that the mobile body 10 has reached the starting point 701 based on the output of the position acquisition unit 14, the mobile body control unit 21 performs the process of landing the mobile body 10 and stopping the drive unit 13 in S114.

[0058] Figure 8 is an external perspective view showing the mobile unit 10 taking still images for surveying purposes. The arrow 750 indicates the direction of movement of the mobile unit 10 for convenience. With the pitch movable part 53 rotated, the imaging device 100 takes images with the interchangeable lens 500 pointed downwards. In other words, the mobile unit 10 is in a state where it can photograph the ground.

[0059] To improve the efficiency of the surveying, the mobile body 10 moves in the direction indicated by the arrow 750 while the imaging device 100 takes pictures, which may cause rolling shutter distortion. This will be explained in detail using Figure 9. Figure 9 is a schematic diagram to explain the occurrence of rolling shutter distortion. Figure 9(A) shows the start of exposure. Figure 9(B) shows the end of exposure. Figure 9(C) shows a still image acquired when exposure has been performed from Figure 9(A) to Figure 9(B).

[0060] In Figures 9(A) and 9(C), the imaging range 751 is the range on the image sensor 150 where imaging is possible. The exposure positions 752 shown in Figures 9(A) and 9(B) are the positions on the image sensor 150 where exposure occurs at the timings corresponding to each figure. The subject is a convenient star-shaped object located within the measurement range 702, and the subject area 753 is shown. The double-headed arrow 754 shown in Figure 9(B) represents the amount of rolling shutter distortion.

[0061] At the start of exposure shown in Figure 9(A), the subject area 753 is contained within the imaging range 751. However, as shown in Figure 5, if an exposure time difference t occurs in the image sensor 150, at the end of exposure shown in Figure 9(B), the subject area 753 is no longer contained within the imaging range 751 because the moving object 10 is moving in the direction of movement (direction of arrow 750). In Figure 9(B), the subject area 753 extends below the image range 751. The amount of overhang corresponds to the rolling shutter distortion (double-headed arrow 754).

[0062] As shown in Figure 9(C), the output still image has a portion of the subject area 753 missing. The subject area 753 extends below the imaging range 751, and the captured area is reduced by the amount of rolling shutter distortion shown in Figure 9(B). This means that the number of shooting locations included in the group of shooting locations 703, as shown in Figure 7, must be increased. Also, as shown in Figure 9(C), the image of the subject area 753 is stretched vertically on the paper, and the aspect ratio changes (aspect ratio distortion). This is a significant disadvantage when taking images for surveying where accurate still image acquisition is necessary. In this embodiment, RS distortion correction processing is performed to address the disadvantages caused by rolling shutter distortion (reduction in the shooting range, aspect ratio distortion). This will be explained with reference to Figure 10.

[0063] Figure 10 is a timing chart illustrating the RS distortion correction process in this embodiment. Figure 10(A) shows the processing during LV video shooting when the high frame rate mode for LV video is set to OFF in S103 (Figure 6), or when still images are taken at the shooting locations of the shooting location group 703. Figure 10(B) shows the processing during LV video shooting when the high frame rate mode for LV video is set to ON in S103. The main unit executing the processing shown in Figure 10 is the control unit 180 of the imaging device 100.

[0064] The vertical axes in Figures 10(A) and 10(B) show the following timings. • Timing of the start of imaging in the imaging device 100 • Timing of receiving movement information from mobile device 10 • Timing of calculation of the amount of movement on the image sensor • Timing related to the exposure period of the image sensor 150 • Timing related to the movement period of the image sensor 150 in a predetermined direction (+ direction, - direction)

[0065] The horizontal axis in Figures 10(A) and 10(B) represents the time axis. In Figure 10(A), times t1 to t8 are shown on the time axis, and in Figure 10(B), times t11 to t18 are shown on the time axis. In each figure, the larger the number attached to the symbol "t", the later the time in time.

[0066] The periods in Figure 10(A) are as follows: • First period (t1~t2): This period indicates the time during which the mobile body 10 sends a shooting start signal to the imaging device 100 via the shooting signal output terminal 19 from the mobile body control unit 21. The shooting start signal is generated before the exposure period (t5~t6).

[0067] • Second period (t2~t3): This period indicates the time during which movement information is sent from the mobile body 10 to the imaging device 100 through the movement information input / output terminal 20. Movement information includes information such as the direction of movement of the mobile body 10, the speed of movement of the mobile body 10, and the altitude of the mobile body 10. For example, during the second period, the control unit 180 sends a request signal to the mobile body 10 requesting that it send movement information. Upon receiving the request signal, the mobile body control unit 21 processes the transmission of a movement information signal to the imaging device 100.

[0068] • Third period (t3~t4): This period indicates the time during which the control unit 180 calculates the amount of movement on the image sensor. The amount of movement on the image sensor is the amount of movement of the image sensor 150 by the image sensor drive device 220 in order to perform RS distortion correction. In addition to the movement information, the control unit 180 calculates the amount of movement on the image sensor based on the focal length of the imaging optical system in the interchangeable lens 500, which is stored in the memory unit 190. Specifically, the amount of movement on the image sensor is denoted as d (mm), the focal length related to the interchangeable lens 500 as f (mm), the movement speed of the moving body 10 as v (m / s), the exposure time as T (s), and the altitude of the moving body 10 as h (m). The amount of movement on the image sensor d (mm) can be expressed by the following formula. d = (f·v·T) / h ···(1)

[0069] • The fourth period (t4~t5): This period indicates the time during which the image sensor 150 is moved by the image sensor drive device 220. The control unit 180 calculates half of the amount of movement d on the image sensor calculated by equation (1), and calculates the target drive amount and drive direction (movement direction) of the image sensor 150 corresponding to this amount (d / 2). The first drive direction corresponding to the movement direction of the moving body 10 is defined as the - direction (minus direction), and the second drive direction corresponding to the opposite direction is defined as the + direction (plus direction). The fourth period is the time during which the image sensor 150 is moved in the - direction by the image sensor drive device 220 with respect to the movement direction of the moving body 10.

[0070] • The fifth period (t5~t6): This indicates the exposure period for the image sensor 150. The length of the exposure period (exposure time) is the sum of the exposure time difference t and the time corresponding to the shutter speed. For convenience, the shutter speed will be assumed to be sufficiently fast compared to the speed corresponding to the exposure time difference t.

[0071] In this embodiment, during the fifth period, drive control is performed by the image sensor drive device 220 to move the image sensor 150 by an amount of movement on the image sensor in the direction opposite to the direction of movement of the moving body 10. Specifically, with respect to the direction of movement of the moving body 10, the image sensor drive device 220 moves the image sensor 150 in the + direction. The process of moving the image sensor 150 in the + direction is the process of moving the image sensor 150 to the lower side of the paper by an amount equal to the amount of rolling shutter distortion (double-headed arrow 754) shown in Figure 9. This process can reduce the amount of rolling shutter distortion.

[0072] In the fifth period, the control unit 180 outputs a control command to the image sensor drive unit 220 by further superimposing the amount of movement for blur correction on the target drive amount based on the vibration detected by the blur detection unit 170. The image sensor drive unit 220 drives (moves) the image sensor 150 according to the control command, thereby correcting image blur caused by vibration.

[0073] • The sixth period (t6~t7): This period indicates the time during which the image sensor 150 is moved in the - direction by the image sensor drive unit 220 and returned to the center position of the movable range. The control unit 180 acquires a position detection signal from the image sensor position detection unit 230. Based on the position detection value, the control unit 180 detects how much the image sensor 150 has displaced relative to the center position of the movable range and controls the image sensor drive unit 220 to return the image sensor 150 to the center position.

[0074] • The seventh period (t7~t8): This period indicates when the next image will be taken. The same processing as the period from time t1 to time t7 will be performed.

[0075] The processing for still image capture at each of the shooting locations constituting the shooting location group 703 is completed during the period from time t1 to time t7. For example, when still image capture at the Nth shooting location is completed, it is expected that the process will move to still image capture at the N+1th shooting location, but it is assumed that the movement from the Nth shooting location to the N+1th shooting location takes about 1 second. In this case, it is not possible to immediately move to still image capture at the N+1th shooting location, and during the transition period of movement, capture for LV video is performed. If the high frame rate mode for LV video is set to OFF, the processing shown during the period from time t1 to time t7 is repeated in order to capture LV video.

[0076] The sum of the lengths of the fourth period (t4~t5) and the sixth period (t6~t7) is approximately the same as the length of the fifth period (t5~t6). In other words, the frame rate of the LV video will be slowed down by the sum of the lengths of the fourth and sixth periods, which may make it impossible to provide the smooth video required for LV video. Therefore, referring to Figure 10(B), we will explain the case where the high frame rate mode for LV video is set to ON in S103 shown in Figure 6. The processing performed with this setting makes it possible to provide smooth LV video.

[0077] In Figure 10(B), times t11 to t14 correspond to times t1 to t4 shown in Figure 10(A). Since the period from t11 to t14 is equivalent to the period from t1 to t4 shown in Figure 10(A), their explanation is omitted.

[0078] The period from t14 to t15 is the period during which the image sensor 150 is moved in the - direction relative to the direction of movement of the moving body 10. The ratio to the amount of movement on the image sensor is different from the fourth period (t4 to t5) shown in Figure 10(A). During the period from t14 to t15, the control unit 180 determines an amount that is 1 / 4 of the amount of movement on the image sensor d calculated by equation (1), and calculates the target drive amount of the image sensor 150 corresponding to that amount (d / 4).

[0079] The period from t15 to t16 is the exposure period for the image sensor 150. The ratio to the amount of movement on the image sensor is different from the fifth period (t5 to t6) shown in Figure 10(A). The exposure time in the period from t15 to t16 is the same as in the fifth period and the drive direction is positive, but the control unit 180 calculates half the amount of movement d on the image sensor calculated by equation (1) and calculates the target drive amount of the image sensor 150 corresponding to that amount (d / 2). In other words, in the period from t15 to t16, the reduction in rolling shutter distortion is about half compared to the fifth period.

[0080] The period from t16 to t17 is the time when the image sensor drive unit 220 moves the image sensor 150 in the negative direction to return the image sensor 150 to the center position of the movable range. Based on the detection value of the image sensor position detection unit 230, the control unit 180 detects how much the image sensor 150 has displaced relative to the center position of the movable range and performs the process of returning the image sensor 150 to the center position of the movable range using the image sensor drive unit 220. During the period from t15 to t16, the amount of movement on the image sensor is half the amount d, and during the period from t14 to t15, it is one-quarter the amount of movement on the image sensor d. Therefore, during the period from t16 to t17, it is one-quarter the amount of movement on the image sensor d.

[0081] The period from t17 to t18 is the period during which the image sensor 150 is moved in the - direction relative to the direction of movement of the moving body 10. The control unit 180 determines 1 / 4 of the amount of movement d on the image sensor calculated by equation (1), and calculates the target drive amount of the image sensor 150 corresponding to that amount (d / 4).

[0082] In this embodiment, processing equivalent to that performed in the period t11-t14 is performed during the period t16-t17. This reduces the overall processing time. Processing similar to that performed in the period t14-t17 is continuously executed in order to continue shooting to acquire LV video.

[0083] In the fifth period (t5-t6) shown in Figure 10(A), the image sensor 150 moves by an amount equivalent to the amount of movement on the image sensor, whereas in the period (t15-t16) shown in Figure 10(B), the image sensor 150 moves by an amount equivalent to half the amount of movement d on the image sensor. Consequently, the amount of movement of the image sensor 150 during the sum of the periods t14-t15 and t16-t17 is half the amount of movement of the image sensor 150 during the sum of the fourth period (t4-t5) and the sixth period (t6-t7) shown in Figure 10(A). In other words, the sum of the lengths of the periods t14-t15 and t16-t17 can be reduced to half the sum of the lengths of the fourth period (t4-t5) and the sixth period (t6-t7). Therefore, by shortening the period from t14 to t17, the frame rate of the LV video can be increased.

[0084] The process shown in Figure 10(B) reduces rolling shutter distortion by about half compared to the process shown in Figure 10(A), but it can still provide the smooth image required for live viewing. For example, let's assume that "exposure time (length of the period from t15 to t16) ≈ exposure time difference t" and "t = 22ms". Let's assume that the sum of the length of the period from t14 to t15 and the length of the period from t16 to t17 is 11ms. In this case, it is possible to shoot for live viewing at a frame rate of 30fps.

[0085] [Modified Embodiment] Next, a modified embodiment of this embodiment will be described. Note that the description of the modified embodiment is also applicable to the embodiment described later, to the extent that it does not cause control inconsistencies.

[0086] In Figure 10(B), the amount of movement on the image sensor during the period t15-t16 is set to half of the amount of movement d on the image sensor during the fifth period (t5-t6) shown in Figure 10(A), but this example is not the only one. In modified embodiments, any ratio is set such that the amount of movement on the image sensor during the period t15-t16 is smaller than the amount of movement on the image sensor during the fifth period. If the amount of movement on the image sensor is small due to the slow movement speed of the moving body 10, it is also possible to perform the processing during the period t14-t17 shown in Figure 10(B) within the period corresponding to the set frame rate. In that case, during the period t15-t16, the control unit 180 calculates a target drive amount corresponding to the amount of movement (d) on the image sensor and performs the processing to move the image sensor 150 using the image sensor drive device 220.

[0087] Furthermore, if the frame rate decreases due to a heavy processing load on the control unit 180, the control unit 180 in the modified embodiment performs the following processing. In the above embodiment, with regard to the calculation of the amount of movement on the image sensor during the period t16 to t17, which is performed for each shooting, the control unit 180 calculates the amount of movement on the image sensor only when predetermined conditions are met. The predetermined conditions are that the change in the movement speed or the change in altitude of the moving body 10 exceeds a threshold since the time when the control unit 180 last calculated the amount of movement on the image sensor. By executing the calculation process for the amount of movement on the image sensor only when the control unit 180 determines that the predetermined conditions are met, the frequency of calculations can be reduced. As a result, the processing load on the control unit 180 can be further reduced, and the decrease in the frame rate can be suppressed.

[0088] Furthermore, the embodiment is not limited to one in which the moving speed information, moving direction information, and altitude information of the moving body 10 are acquired by the moving body 10. In a modified embodiment, a position acquisition unit composed of GNSS is installed in the imaging device 100, so that the moving speed information, moving direction information, and altitude information can be acquired inside the imaging device 100. In the above embodiment, the calculation of the amount of movement on the image sensor is performed based on the altitude information of the moving body 10 as the value of the distance information of the subject, but the embodiment is not limited to this example. The imaging system of the modified embodiment is equipped with a distance measuring device that can directly measure the subject distance (distance between the imaging means and the subject), and the control unit 180 can calculate the amount of movement on the image sensor using the measured subject distance instead of the altitude information of the moving body 10.

[0089] In the above embodiment, for convenience, the RS distortion correction in the vertical direction of the paper shown in Figure 9 was described, but the invention is not limited to this example. The control unit 180 of the modified embodiment can perform RS distortion correction in any direction parallel to the imaging plane of the image sensor 150.

[0090] In the above embodiment, an example configuration was shown in which the image sensor 150 is moved by the image sensor drive device 220 in RS distortion correction. As a modified embodiment, there is a control configuration in which the control unit 180 moves the image blur correction lens (shift lens, etc.) that constitutes the imaging optical system by means of a drive mechanism. There is also a control configuration in which the movement of the image blur correction lens and the movement of the image sensor are used in combination. The imaging optical system is an optical system composed of multiple optical elements (lenses, apertures, etc.) in an interchangeable lens, or an optical system composed of multiple optical elements in the lens section of an imaging device.

[0091] Furthermore, the embodiment is not limited to one in which the control unit 180 and the mobile body control unit 21 each have their own processors. For example, there is a modified embodiment in which the mobile body control unit 21 is abolished and the functions of the mobile body control unit 21 are integrated into the control unit 180. Alternatively, there is a modified embodiment in which the control unit 180 is abolished and the functions of the control unit 180 are integrated into the mobile body control unit 21.

[0092] In the above embodiments, when capturing a still image, the control unit 180 controls the movement of the image sensor or image stabilization lens according to a first drive amount calculated by the control unit 180. When capturing a display image (LV image), the control unit 180 controls the movement of the image sensor or image stabilization lens according to a second drive amount calculated by the control unit 180. Alternatively, when capturing a display image at a first frame rate, the control unit 180 controls the movement of the image sensor or image stabilization lens according to a first drive amount calculated by the control unit 180. Furthermore, when capturing a display image at a second frame rate higher than the first frame rate, the control unit 180 controls the movement of the image sensor or image stabilization lens according to a second drive amount calculated by the control unit 180. By changing from the first drive amount to the second drive amount, it is possible to provide an imaging device that can acquire smooth video data required for LV images while suppressing the occurrence of rolling shutter distortion when capturing LV images.

[0093] [Second Embodiment] A second embodiment will be described with reference to Figure 11. In this embodiment, matters similar to those in the previous embodiment will be omitted from explanation, and the differences will be explained mainly. This method of omitting explanations will be the same in the embodiments described later.

[0094] In this embodiment, the process for further increasing the frame rate when shooting for LV video will be explained in detail using Figure 11. Figure 11 is a timing chart for explaining the RS distortion correction process in this embodiment. The time axis shows times t21 to t32. The period from t21 to t26 is equivalent to the period from t11 to t16 shown in Figure 10(B), so their explanation will be omitted.

[0095] The period from t26 to t27 corresponds to the period from t16 to t17 shown in Figure 10(B). During the period from t26 to t27, the image sensor 150 is moved in the - direction, which corresponds to the direction of movement of the moving body 10. The difference from the period from t16 to t17 is that the image sensor 150 is not moved to the center position of the movable range.

[0096] The period from t27 to t28 corresponds to the period from t17 to t18 shown in Figure 10(B). The control unit 180 calculates a target drive amount corresponding to an amount smaller than 1 / 4 of the movable amount d on the image sensor and controls the drive of the image sensor 150.

[0097] The amount of movement on the image sensor during the period t26-t28, as shown in Figure 11, is smaller than the sum of the amount of movement on the image sensor during the period t14-t15 and the amount of movement on the image sensor during the period t16-t17, as shown in Figure 10(B). The length of the period t26-t28 is shorter than the sum of the length of the period t14-t15 and the length of the period t16-t17. Therefore, the processing shown in Figure 11 makes it possible to further increase the frame rate.

[0098] During shooting to acquire LV video, the process from t25 to t28 is repeatedly executed. The amount of movement on the image sensor during the period from t26 to t28 is smaller than the amount of movement on the image sensor during the period from t25 to t26. In other words, if the process from t25 to t28 is repeatedly executed, the image sensor 150 approaches the edge of the movable range in the + direction, which corresponds to the direction opposite to the direction of movement of the moving body 10. The edge of the movable range corresponds to the limit position of the image sensor 150 as moved by the image sensor drive device 220.

[0099] Therefore, when the exposure period from t29 to t30 shown in Figure 11 has elapsed, the control unit 180 executes a process to move the image sensor 150 toward the center position of the movable range. During the period from t30 to t31, the image sensor 150 moves toward the center position of the movable range, similar to the period from t26 to t27. However, based on the output result of the image sensor position detection unit 230, the control unit 180 determines whether the detected position of the image sensor 150 has fallen below a certain distance (threshold) from the edge of its movable range. If this determination condition is met, the control unit 180 simultaneously performs a process to return the image sensor 150 to the center position of the movable range. During the period from t30 to t31 shown in Figure 11, it is determined that the determination condition is met, and the process to return the image sensor 150 to the center position of the movable range is performed. Therefore, since the period from t30 to t31 is longer than the period from t26 to t27, the shooting period including the period from t30 to t31 becomes longer. However, since the periods other than t30-t31 are short, the overall advantage is that the frame rate can be increased. The same processing as in the t25-t28 period is repeatedly performed during the subsequent shooting period for LV video.

[0100] According to this embodiment, it is possible to provide LV video at a higher frame rate compared to the first embodiment.

[0101] [Third Embodiment] A third embodiment will be described with reference to Figure 12. In the first embodiment, the exposure time difference t (Figure 5) was kept constant. In this embodiment, processing related to an image sensor in which the exposure time difference t can be changed, or an image sensor in which the exposure time difference t can be changed to 0, will be described.

[0102] Figure 12 is a timing chart illustrating the exposure of the image sensor 150 according to this embodiment. Referring to Figure 12(A), the exposure when the exposure time difference t is shortened will be explained. Referring to Figure 12(B), the exposure when no curtain speed time is required will be explained.

[0103] Figure 12(A) differs from Figure 5 in that the exposure time difference is halved. At the start of exposure, exposure begins for pixels corresponding to the first row of pixels on the image sensor 150. Then, after a time of Δt / 2, exposure begins for pixels corresponding to the second row of pixels. Assuming there are n rows of pixels, the exposure time difference across the first to n rows of pixels is denoted as t2. The time elapsed from the first to the nth row is "exposure time difference t2 = n·Δt / 2", which is half the time shown in Figure 5, "exposure time difference t = n·Δt". In other words, rolling shutter distortion can be reduced by half. Hereafter, this exposure method will be referred to as the high-speed curtain speed mode. Disadvantages of the high-speed curtain speed mode include a decrease in dynamic range and an increase in noise in dark places.

[0104] In Figure 12(B), the exposure time difference is zero or within the acceptable range (exposure time difference is below the threshold time). The exposure time difference t in Figure 5 and the exposure time difference t2 in Figure 12(A) do not occur. Hereafter, the mode using the exposure method called the global shutter method will be referred to as the global shutter mode. Disadvantages of the global shutter mode generally include a further reduction in dynamic range compared to the high-speed curtain mode and increased noise in dark places.

[0105] Furthermore, the exposure mode described in Figure 5 will be described below as the normal curtain speed mode. During the period from t3 to t4 shown in Figure 10(A) and the period from t13 to t14 shown in Figure 10(B), the control unit 180 calculates the amount of movement on the image sensor and controls the drive of the image sensor 150 with a target drive amount corresponding to the calculated amount of movement on the image sensor. At that time, if the control unit 180 determines that the position of the image sensor 150 exceeds the limit of the movable range, it performs a process to change the exposure mode of the image sensor 150 from the normal curtain speed mode to the high-speed curtain speed mode. Due to the mode change, the exposure time difference becomes t to t2, which is half the time. Accordingly, the amount of movement on the image sensor can be halved.

[0106] Furthermore, if the control unit 180 determines that the position of the image sensor 150 exceeds the limit of its movable range even after changing from the normal shutter speed mode to the high-speed shutter speed mode, it performs a process to change from the high-speed shutter speed mode to the global shutter mode. As shown in Figure 12(B), there is no exposure time difference in the global shutter mode, so rolling shutter distortion does not occur. Therefore, the process of moving the image sensor 150 by the image sensor drive device 220 during the period from t4 to t7 shown in Figure 10(A) and the period from t14 to t17 shown in Figure 10(B) does not need to be performed.

[0107] In normal shutter speed mode and high shutter speed mode, the amount of noise generated in dark conditions is smaller compared to global shutter mode, allowing for sufficiently fast exposure of the image sensor.

[0108] On the other hand, in global shutter mode, the amount of noise generated in dark places is large, so the shutter speed must be slowed down. In that case, image blur may occur in proportion to the decrease in shutter speed. To correct this image blur, the control unit 180 can perform a process to move the image sensor 150 using the image sensor drive device 220 during the period from t4 to t7 shown in Figure 10(A) and during the period from t14 to t17 shown in Figure 10(B).

[0109] In this embodiment, it is possible to provide even higher quality LV images by changing from normal shutter speed mode to high-speed shutter mode, or from high-speed shutter mode to global shutter mode. Furthermore, the rolling shutter and global shutter modes may be switchable according to user operation.

[0110] Although preferred embodiments and modified embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0111] Embodiments of this disclosure include the following configurations and methods: [Configuration 1] An acquisition means for acquiring movement information related to the movement of an imaging device, A driving means for moving the image sensor or the lens that performs image blur correction in a direction parallel to the imaging plane, The system includes a control means that controls the movement of the image sensor or lens by the drive means according to the movement information during a first shooting session to acquire a still image and a second shooting session to acquire a display image. When the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the drive means to move the image sensor or lens, and during the second shooting, the control means calculates a second drive amount and controls the drive means to move the image sensor or lens. An imaging device characterized by the following features. [Configuration 2] An acquisition means for acquiring movement information related to the movement of an imaging device, A driving means for moving the image sensor or the lens that performs image blur correction in a direction parallel to the imaging plane, The system includes control means that controls the movement of the image sensor or lens by the driving means according to the movement information during a first shooting to acquire a display image at a first frame rate, and during a second shooting to acquire a display image at a second frame rate higher than the first frame rate. When the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the drive means to move the image sensor or lens, and during the second shooting, the control means calculates a second drive amount and controls the drive means to move the image sensor or lens. An imaging device characterized by the following features. [Configuration 3] The second drive amount used to move the image sensor or lens during the exposure period of the image sensor during the second shooting is smaller than the first drive amount used to move the image sensor or lens during the exposure period of the image sensor during the first shooting. The imaging apparatus according to configuration 1 or 2, characterized by the above. [Structure 4] The control means controls the image sensor or lens to move in a first direction parallel to the imaging surface by the driving means before the image sensor is exposed, and to move the image sensor or lens in a second direction opposite to the first direction by the driving means during the exposure period of the image sensor. An imaging device according to any one of configurations 1 to 3 characterized by the above. [Composition 5] The control means controls the driving means to move the image sensor or lens in the first direction after the exposure period for the image sensor has elapsed. The imaging apparatus according to configuration 4, characterized by the features described above. [Composition 6] The amount of drive used by the driving means to move the image sensor or lens before exposure of the image sensor and after the exposure period has elapsed is smaller than the amount of drive used by the driving means to move the image sensor or lens during the exposure period of the image sensor. The imaging apparatus according to configuration 5, characterized in that it is a device. [Composition 7] The image sensor uses a rolling shutter system, which generates exposure time differences for each pixel row, It includes a detection means for detecting the position of the image sensor or lens, If the control means determines that the position of the image sensor or lens exceeds the limit of the movable range, it changes from a first mode in which the exposure time difference is a first time difference to a second mode in which the exposure time difference is smaller than the first time difference. An imaging apparatus according to any one of configurations 1 to 6 characterized by the above. [Structure 8] The system has detection means for detecting the position of the image sensor or lens, If the control means determines that the position of the image sensor or lens exceeds the limit of the movable range, it changes from a first mode in which there is a time difference in exposure for each row of pixels in the image sensor to a second mode in which there is no time difference in exposure for each row of pixels in the image sensor, or the time difference is less than or equal to a threshold time. An imaging apparatus according to any one of configurations 1 to 6 characterized by the above. [Composition 9] The system has detection means for detecting the position of the image sensor or lens, The control means, based on the position of the image sensor or lens detected by the detection means after the exposure period has elapsed, controls the drive means to move the image sensor or lens toward the center of the movable range. An imaging device according to any one of configurations 1 to 8. [Configuration 10] It can be mounted on a mobile device, The acquisition means acquires the movement information from the moving body, During the first and second imaging, the control means controls the movement of the image sensor or lens during the exposure period using the movement information and the drive amount, which corresponds to the amount of movement of the image sensor or lens calculated from the focal length of the imaging optical system that forms an image on the image sensor. An imaging device according to any one of configurations 1 to 9, characterized by the above. [Composition 11] When the control means performs shooting of the display image, it determines that the change in the movement speed of the moving body is greater than or equal to a threshold value from the time when the movement amount of the image sensor or lens was previously calculated, and then calculates the movement amount of the image sensor or lens. The imaging apparatus according to configuration 10, characterized by the above. [Composition 12] It can be mounted on a mobile device, The control means calculates the amount of movement of the image sensor or lens based on the focal length, the movement speed of the moving body, the exposure time, and the altitude of the moving body. An imaging apparatus according to any one of configurations 1 to 11, characterized by the above. [Composition 13] An imaging device as described in any one of configurations 1 to 12, Mobile and A connecting device for connecting the imaging device and the mobile body, The system includes a remote control device for remotely controlling the mobile body. An imaging system characterized by the following: [Method 1] A control method performed in an imaging device having a driving means for moving an image sensor or a lens for correcting image blur in a direction parallel to the imaging plane, An acquisition process to acquire movement information related to the movement of the imaging device, The system includes a control step in which, during a first shooting to acquire a still image and a second shooting to acquire a display image, the control means controls the movement of the image sensor or lens by the drive means according to the movement information, In the control process described above, when the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the movement of the image sensor or lens by the drive means, and during the second shooting, the control means calculates a second drive amount and controls the movement of the image sensor or lens by the drive means. A control method characterized by the following: [Method 2] A control method performed in an imaging device having a driving means for moving an image sensor or a lens for correcting image blur in a direction parallel to the imaging plane, An acquisition process to acquire movement information related to the movement of the imaging device, The system includes a control step in which, during a first shooting to acquire a display image at a first frame rate, and during a second shooting to acquire a display image at a second frame rate higher than the first frame rate, the control means controls the driving means to move the image sensor or lens according to the movement information. In the control process described above, when the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the movement of the image sensor or lens by the drive means, and during the second shooting, the control means calculates a second drive amount and controls the movement of the image sensor or lens by the drive means. A control method characterized by the following: [Explanation of symbols]

[0112] 1. Imaging System 10 Mobile Units 50 Gimbal 100 Imaging device 150 image sensors 220 Image sensor drive unit 600 Remote Control Devices

Claims

1. An acquisition means for acquiring movement information related to the movement of an imaging device, A driving means for moving the image sensor or the lens that performs image blur correction in a direction parallel to the imaging plane, The system includes a control means that controls the movement of the image sensor or lens by the drive means according to the movement information during a first shooting to acquire a still image and a second shooting to acquire a display image. When the imaging device is moving while taking a picture, during the first shot, the control means calculates a first drive amount and controls the drive means to move the image sensor or lens, and during the second shot, the control means calculates a second drive amount and controls the drive means to move the image sensor or lens. An imaging device characterized by the following features.

2. An acquisition means for acquiring movement information related to the movement of an imaging device, A driving means for moving the image sensor or the lens that performs image blur correction in a direction parallel to the imaging plane, The system includes a control means that controls the movement of the image sensor or lens by the driving means according to the movement information during a first shooting to acquire a display image at a first frame rate, and during a second shooting to acquire a display image at a second frame rate higher than the first frame rate. When the imaging device is moving while taking a picture, during the first shot, the control means calculates a first drive amount and controls the drive means to move the image sensor or lens, and during the second shot, the control means calculates a second drive amount and controls the drive means to move the image sensor or lens. An imaging device characterized by the following features.

3. The second drive amount used to move the image sensor or lens during the exposure period of the image sensor during the second shooting is smaller than the first drive amount used to move the image sensor or lens during the exposure period of the image sensor during the first shooting. The imaging apparatus according to claim 1 or 2.

4. The control means controls the image sensor or lens to move in a first direction parallel to the imaging surface by the driving means before the image sensor is exposed, and to move the image sensor or lens in a second direction opposite to the first direction by the driving means during the exposure period of the image sensor. The imaging apparatus according to claim 1 or 2.

5. The control means controls the driving means to move the image sensor or lens in the first direction after the exposure period for the image sensor has elapsed. The imaging apparatus according to feature 4.

6. The amount of drive used by the driving means to move the image sensor or lens before exposure of the image sensor and after the exposure period has elapsed is smaller than the amount of drive used by the driving means to move the image sensor or lens during the exposure period of the image sensor. The imaging apparatus according to feature 5.

7. The image sensor uses a rolling shutter system, which generates exposure time differences for each pixel row, It includes a detection means for detecting the position of the image sensor or lens, If the control means determines that the position of the image sensor or lens exceeds the limit of the movable range, it changes from a first mode in which the exposure time difference is a first time difference to a second mode in which the exposure time difference is smaller than the first time difference. The imaging apparatus according to feature 1.

8. The system has detection means for detecting the position of the image sensor or lens, If the control means determines that the position of the image sensor or lens exceeds the limit of the movable range, it changes from a first mode in which there is a time difference in exposure for each row of pixels in the image sensor to a second mode in which there is no time difference in exposure for each row of pixels in the image sensor, or the time difference is less than or equal to a threshold time. The imaging apparatus according to feature 1.

9. The system has detection means for detecting the position of the image sensor or lens, The control means, based on the position of the image sensor or lens detected by the detection means after the exposure period has elapsed, controls the drive means to move the image sensor or lens toward the center of the movable range. The imaging apparatus according to feature 1.

10. It can be mounted on a mobile device, The acquisition means acquires the movement information from the moving body, During the first and second imaging, the control means controls the movement of the image sensor or lens during the exposure period using the movement information and the drive amount, which corresponds to the amount of movement of the image sensor or lens calculated from the focal length of the imaging optical system that forms an image on the image sensor. The imaging apparatus according to feature 1.

11. When the control means performs shooting of the display image, it determines that the change in the movement speed of the moving body is greater than or equal to a threshold value from the time when the movement amount of the image sensor or lens was previously calculated, and then calculates the movement amount of the image sensor or lens. The imaging apparatus according to feature 10.

12. It can be mounted on a mobile device, The control means calculates the amount of movement of the image sensor or lens based on the focal length, the movement speed of the moving body, the exposure time, and the altitude of the moving body. The imaging apparatus according to claim 1 or 2.

13. The imaging device according to claim 1, Mobile and A connecting device for connecting the imaging device and the mobile body, The system includes a remote control device for remotely controlling the mobile body. An imaging system characterized by the following:

14. A control method performed in an imaging device having a driving means for moving an image sensor or a lens for correcting image blur in a direction parallel to the imaging plane, An acquisition process to acquire movement information related to the movement of the imaging device, The system includes a control step in which, during a first shooting to acquire a still image and a second shooting to acquire a display image, the control means controls the movement of the image sensor or lens by the drive means according to the movement information, In the control process described above, when the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the movement of the image sensor or lens by the drive means, and during the second shooting, the control means calculates a second drive amount and controls the movement of the image sensor or lens by the drive means. A control method characterized by the following:

15. A control method performed in an imaging device having a driving means for moving an image sensor or a lens for correcting image blur in a direction parallel to the imaging plane, An acquisition process to acquire movement information related to the movement of the imaging device, The system includes a control step in which, during a first shooting to acquire a display image at a first frame rate, and during a second shooting to acquire a display image at a second frame rate higher than the first frame rate, the control means controls the driving means to move the image sensor or lens according to the movement information. In the control process described above, when the imaging device is moving while taking a picture, during the first shooting, the control means calculates a first drive amount and controls the movement of the image sensor or lens by the drive means, and during the second shooting, the control means calculates a second drive amount and controls the movement of the image sensor or lens by the drive means. A control method characterized by the following:

Images