Imaging device and control method thereof

Abstract

To perform more appropriate distortion correction while improving the quality of a picked-up image.SOLUTION: An imaging apparatus has: an imaging unit that has an imaging optical system and an image pick-up device; attitude control means that moves the direction of the imaging unit in a pan direction and / or a tilt direction; shift control means that moves at least one of the imaging optical system and the image pick-up device in a plane parallel to an imaging surface; synchronization control means that, in driving one of the attitude control means and the shift control means, drives the other in synchronization with the drive of the one of the means so as to cancel a change in an imaging range of the imaging unit due to the drive of the one of the means; and determination means that sequentially determines the amount of distortion of a subject in a picked-up image obtained by the imaging unit. The synchronization control means stops the drive when the amount of distortion determined by the determination means becomes equal to or less than a predetermined threshold.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to imaging control, and more particularly to imaging control for reducing subject distortion. [Background technology]

[0002] When a subject is not facing the camera directly, such as when photographing a tall building from the ground, distortion occurs in the captured image. A known technique for solving this problem is to face the camera directly to the subject and to capture the image by shifting the image sensor (or optical system) in parallel with the imaging surface (hereafter referred to as "shift control").

[0003] Patent Document 1 discloses a technique for detecting the inclination of a subject relative to a camera using distance measurement data obtained based on imaging, and performing facing correction. It also discloses that facing correction is performed by rotation control of a lens unit and shift control of an imaging element. Patent Document 2 discloses a technique for detecting distortion (angle) of an edge of a subject relative to the vertical direction, and shifting the optical system so as to cancel out the detected distortion. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2003-185902 A [Patent Document 2] JP 2011-059283 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, the inclination of the subject is detected based on the difference between multiple distance measurement data acquired at different image height positions on the image sensor. Therefore, when the distance measurement accuracy is low, the distortion of the subject may not be appropriately corrected. In addition, in Patent Document 2, shift control of the optical system is performed to offset the distortion of the edge, but the fluctuation of the imaging range due to the shift control is not taken into consideration. Therefore, the position of the subject in the captured image may change during the period in which the shift control is performed, resulting in deterioration of the quality.

[0006] The present invention has been made in view of the above problems, and has an object to provide a technique that enables more appropriate distortion correction while improving the quality of a captured image. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, an image pickup apparatus according to the present invention has the following arrangement. an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; a synchronization control means for driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in the imaging range of the imaging unit caused by driving the other is cancelled out; a determination means for sequentially determining a distortion amount of a subject in an image captured by the imaging unit; A receiving means for receiving a user instruction for the shift control means; having The synchronization control means stops driving when the distortion amount determined by the determination means becomes equal to or less than a predetermined threshold value. death, the receiving means is configured to receive a drive stop instruction to stop driving the shift control means, When the reception means receives the drive stop instruction while the attitude control means and the shift control means are being driven, the synchronization control means adjusts one of the orientation of the imaging unit and the shift position of the imaging element based on the other at the timing when the drive stop instruction is received. . Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique for performing more appropriate distortion correction while improving the quality of a captured image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an imaging system. [Diagram 2] FIG. 4 is a diagram illustrating a subject distortion. [Diagram 3] FIG. 4 is a diagram illustrating shift control. [Figure 4] 11A and 11B are diagrams illustrating control for reducing subject distortion while maintaining the imaging range. [Diagram 5] 11A and 11B are diagrams for explaining calculation of drive amount for rotation control and shift control; [Figure 6] 11A and 11B are diagrams illustrating changes over time in a captured image during distortion correction. [Figure 7] FIG. 13 is a diagram illustrating an example of a user interface for distortion correction. [Figure 8] FIG. 4 is a diagram illustrating a control method in the first synchronous control. [Figure 9] 11 is a flowchart of a process in a first synchronization control. [Figure 10] FIG. 11 is a diagram illustrating a control method in the second synchronous control. [Figure 11] 13 is a flowchart of a process in a second synchronization control. [Figure 12] 13 is a flowchart of a process executed by an imaging device according to a modified example. [Figure 13] 11A and 11B are diagrams illustrating a distortion shape determination based on depth information. [Figure 14] 11A and 11B are diagrams illustrating a distortion shape determination based on edge detection. [Figure 15] 11A and 11B are diagrams illustrating a distortion shape determination when a plurality of subjects are present; [Figure 16] 11A and 11B are diagrams illustrating how to determine a correction target when a plurality of subjects are present. [Figure 17]FIG. 13 is a diagram showing the relationship between rotation control and shift control for each distortion shape. [Figure 18] 11A and 11B are diagrams illustrating calculation of a distortion amount based on depth information. [Figure 19] 11A and 11B are diagrams illustrating calculation of a distortion amount based on edge detection. [Figure 20] FIG. 2 is a diagram illustrating a hardware configuration of an imaging apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (First embodiment) As a first embodiment of an imaging device according to the present invention, an imaging system will be described below as an example.

[0012] <System configuration> 1 is a block diagram illustrating the configuration of an imaging system. The imaging system includes an imaging device and a monitoring device 111. The imaging device has a camera unit 100, which is an imaging section, an auto gain controller (AGC) 107, an analog-to-digital (AD) converter 108, a camera signal processing section 109, and a communication section 110. The imaging device further has a drive amount determination section 112, a shift control section 113, a rotation control section 114, a shift driving section 115, and a rotation driving section 116.

[0013] The camera unit 100 includes an imaging optical system, a band pass filter (BPF) 104, a color filter 105, and an image sensor 106. Here, the imaging optical system has a zoom lens 101 that moves in the optical axis direction to change the focal length, a focus lens 102 that moves in the optical axis direction to adjust the focus, and an aperture unit 103 that adjusts the amount of light. Note that the imaging optical system may be integrated with the imaging device, or may be detachable from the imaging device.

[0014] The light passing through the imaging optical system forms a subject image as an optical image on the imaging element 106 via the BPF 104 and the color filter 105. The BPF 104 may be removable from the optical path of the imaging optical system. The subject image is photoelectrically converted by the imaging element 106.

[0015] The analog electric signal (image pickup signal) output from the image pickup element 106 is gain-adjusted by the AGC 107, converted into a digital signal by the AD converter 108, and then input to a camera signal processing unit 109. The camera signal processing unit 109 performs various image processes on the digital image pickup signal to generate a video signal.

[0016] The video signal is output to a monitoring device 111 connected to the imaging device by wired or wireless communication via a communication unit 110. Furthermore, the monitoring device 111 receives an instruction from a user and outputs a control signal such as a command to each of a shift control unit 113 and a rotation control unit 114 via the communication unit 110.

[0017] The drive amount determination unit 112 determines the other drive amount based on either the rotation drive amount when controlling the rotation of the camera unit 100 or the shift drive amount when controlling the shift of the image sensor 106. The rotation control is an attitude control related to the rotational movement in the pan direction and / or tilt direction. At this time, the determined drive amount is the other drive amount that performs a correction operation to offset the change in the imaging range caused by one (rotation control or shift control). In other words, the other drive amount is determined so as to suppress the change in the position of the subject in the captured image. Details of the drive amount determination in the drive amount determination unit 112 will be described later with reference to Figs. 2 to 5.

[0018] The shift control unit 113 instructs the image sensor 106 to perform shift drive based on a drive instruction (shift drive amount) determined by the drive amount determination unit 112 or received via the communication unit 110. The rotation control unit 114 instructs the camera unit 100 to perform rotation drive based on the rotation drive amount determined by the drive amount determination unit 112 or instructed via the communication unit 110.

[0019] The shift driving unit 115 drives the image sensor 106 based on the shift driving instructed by the shift control unit 113. The rotation driving unit 116 drives the camera unit 100 based on the rotation driving instructed by the rotation control unit 114. Here, the shift control unit 113 and the shift driving unit 115 instruct and drive the image sensor 106 to perform shift driving, but may instruct and drive the optical system to perform shift driving.

[0020] <Shift control to compensate for subject distortion> 2 is a diagram for explaining object distortion. Here, a schematic diagram of a case where a building is photographed looking up from the ground and a captured image at that time are shown as an example. If the camera unit does not face the object 201 directly (i.e., if the object 201 (the wall of the building) and the optical axis 204 are not perpendicular), the building will be distorted into a trapezoid shape (object distortion) as shown in the captured image 200.

[0021] Fig. 3 is a diagram for explaining shift control. Here, a schematic diagram of shooting the same subject 201 (building) as in Fig. 2 by shift control while facing directly on it, and an example of the captured image at that time are shown. Specifically, the optical axis 204 is set perpendicular to the subject 201 (wall of the building), and the imaging plane 203 is shifted downward (relative to the optical system 202) so that the subject 201 falls within the imaging range. This shift control results in a state in which the subject distortion is compensated, as shown in the captured image 300.

[0022] However, in order to capture an image such as captured image 300 in which the subject distortion is compensated for, it is necessary to appropriately control the camera unit 100 and the image sensor 106, which requires information on the elevation angle 205 and the subject distance 206.

[0023] Fig. 4 is a diagram for explaining control for reducing subject distortion while maintaining the imaging range. Specifically, it shows a state in which, for the imaging state of Fig. 2, rotation control (rotation amount α) of the camera unit is performed so that the optical axis 204 approaches perpendicular to the subject 201 (wall of a building) while shift control (shift amount x) is performed to maintain the imaging range.

[0024] That is, when the camera unit 100 is rotated (tilted) by a minute amount of rotation α, the imaging range changes from the original imaging range 400 to imaging range 401 when the camera unit is rotated by α. For example, in an image captured by imaging range 401, the ground surface is captured for 1 / 3 from the bottom, and the top of the building is missing (cut off). To eliminate such a change in the imaging range (to maintain imaging range 400), it is sufficient to shift the imaging element 106 downward within the imaging surface by the shift amount x.

[0025] 5 is a diagram for explaining the calculation of the drive amount for rotation control and shift control. Specifically, the method for calculating the shift amount x of the image sensor 106 according to the rotation amount α of the camera unit 100 in order to maintain the imaging range as shown in FIG.

[0026] The triangle OAB shown in Fig. 5 is an enlarged view of the triangle OAB shown in Fig. 4. Here, α is the amount of rotation of the camera unit 100, α 0 is the vertical angle of view / 2, x is the shift amount of the image sensor 106, and x 0 is the sensor vertical size / 2, and l is the image-forming position. In this case, the shift amount x of the image sensor when the camera unit 100 is rotated by α is expressed by the following formula (1).

[0027]

number

[0028] Here, the relationship in equation (2) holds.

[0029]

number

[0030] From equations (1) and (2), when camera unit 100 is rotated by α, the shift amount x of the image sensor required to eliminate change in the imaging range can be expressed by equation (3) using the sensor vertical size and imaging position.

[0031]

number

[0032] That is, by performing rotation control and shift control using the relationship of formula (3), it is possible to reduce the distortion of the object. Also, it is possible to suppress the change in the imaging range during the correction control (rotation control and shift control) (i.e., to maintain the imaging range).

[0033] 6 is a diagram showing changes over time in a captured image during distortion correction, for example, changes in a captured image displayed on the display unit of monitoring device 111 from the start of correction control (rotation control and shift control) to the end of the correction control are shown.

[0034] At the start of imaging (before correction control begins), a distortion occurs in which the top of the building, which is the subject, is shorter than the bottom (hereinafter referred to as "top distortion"). For this state, shift control of the image sensor 106 synchronized with the rotation control of the camera unit 100 is started using the relationship in equation (3). This causes the distortion of the subject to be gradually corrected over time while maintaining the relative position of the subject in the captured image. This correction of the top distortion is hereinafter referred to as "top correction". Similarly, corrections of the bottom distortion, right distortion, and left distortion are referred to as bottom correction, right correction, and left correction, respectively.

[0035] 7 is a diagram illustrating a user interface (UI) for distortion correction. For example, this UI can be provided as a physical button or a GUI on a display unit in the monitoring device 111. The user decides which distortion correction (upper correction, lower correction, right correction, left correction) to perform according to the state of distortion of the subject in the captured image displayed on the display unit of the monitoring device 111, and presses the corresponding button on the UI. This causes a user instruction (drive instruction, drive start instruction, drive stop instruction) including information on the corresponding shift direction to be transmitted.

[0036] 6, "upper correction" is performed. When the user starts pressing the button, control of the camera unit 100 and the image sensor 106 is started based on the formula (3), and when the user stops pressing the button, the control is stopped.

[0037] The user can accurately correct the distortion of the subject by pressing the buttons while checking the captured image that changes over time, as shown in Figure 6. For example, if the "Upper Correction" button is pressed for too long, resulting in overcorrection, the user can press the "Lower Correction" button.

[0038] <Effects> The calculation of the shift amount expressed by the above-mentioned formula (3) does not require information such as the elevation angle of the imaging device or the subject distance. Specifically, the information required to calculate the shift amount by formula (3) is the rotation amount of the camera unit 100, the sensor vertical size, and the imaging position. This information is information known for each imaging device or information acquired from control information. In this way, the above-mentioned shift control is useful in that it is not affected by the detection accuracy of information such as the elevation angle and the subject distance.

[0039] 6, it is possible to prevent changes in the imaging range during correction control by synchronizing the rotation control of lens unit 100 and the shift control of image sensor 106. In other words, the subject position relative to the imaging range does not change during correction control, and only the distortion of the subject is corrected, resulting in good quality.

[0040] <Operation of the imaging device> The following will further explain the above-mentioned synchronous control between the rotation control of the camera unit 100 and the shift control of the image sensor 106. In particular, two different synchronous control methods will be explained.

[0041] <First synchronous control> 8 is a diagram for explaining a control method in the first synchronous control. The first synchronous control is a method for sequentially receiving rotation control instructions for one step (predetermined amount) of the camera unit 100, sequentially determining the shift amount of the image sensor 106, and performing synchronous control. That is, while precisely specifying the control target positions of the camera unit 100 and the image sensor 106, the drive of both is synchronized.

[0042] 8 shows the relationship between the amount of rotation of camera unit 100 under certain imaging conditions and the amount of shift of image sensor 106. Here, the amount of rotation of camera unit 100 per step is denoted as α', and the amount of shift x of image sensor 106 corresponding to the amount of rotation from a reference position is calculated, and control is performed based on the calculation result.

[0043] 9 is a flowchart of the process in the first synchronization control. The following operations are performed by each unit of the imaging device.

[0044] In step S900, the image capture device calculates the shift amount of the image sensor 106 based on the formula (3), x 0 Get (sensor vertical size / 2).

[0045] In step S901, the imaging device determines whether or not the distortion correction button (FIG. 7) has been pressed by a user operation. For example, this is determined based on whether or not a control signal transmitted from the monitoring device 111 and input via the communication unit 110 indicates that the distortion correction button has been pressed. If the button has been pressed, the process proceeds to S902, and if the button has not been pressed, the process continues with the determination.

[0046] Of the four types of buttons shown in FIG. 7, the one appropriate for the shape of distortion to be corrected is selectively pressed by the user. The drive direction (shift direction) of the camera unit 100 and the image sensor 106 is uniquely determined by the type of correction selected here. For example, when upper correction is pressed, rotation control is controlled in the tilt downward direction, and shift control is controlled in the downward direction. When lower correction is pressed, the controls are respectively opposite to the case of upper correction. When left correction is pressed, rotation control is controlled in the pan right direction, and shift control is controlled to the right direction. When right correction is pressed, the controls are respectively opposite to the case of left correction.

[0047] In step S902, the imaging device acquires the imaging position l. The imaging position l is the imaging distance between OB shown in FIG. 5, which indicates the imaging distance between the optical system 202 and the imaging surface 203. The imaging position l can be acquired from the focus lens position used to control the focus lens. It is necessary to acquire the imaging position l only once from when the distortion correction button is pressed by the user until the user stops pressing the button.

[0048] In step S903, the drive amount determination unit 112 sets a one-step rotation amount α' of the camera unit 100. The smaller the one-step rotation amount α' is, the greater the effect of maintaining the imaging range.

[0049] In step S904, drive amount determination unit 112 sets a rotation target position α of camera unit 100 in rotation control unit 114. The rotation target position α is set based on the one-step rotation amount α' set in S903. Using the rotation position of camera unit 100 at the start of distortion correction as a reference, the amount obtained by adding the one-step rotation amount α' each time the camera unit rotates one step is the rotation target position α of camera unit 100.

[0050] In step S905, the drive amount determination unit 112 determines the shift target position x of the image sensor 106 based on the rotation target position α of the camera unit 100 set in S904. This determination is made based on the formula (3), and the required x 0 , l, and α are the values ​​acquired or set in S900, S902, and S904, respectively.

[0051] In step S906, the drive amount determination unit 112 sets in the shift control unit 113 the shift target position x of the image sensor 106 calculated in S905.

[0052] In step S907, rotation control unit 114 and shift control unit 113 respectively control rotation drive unit 116 and shift drive unit 115 to drive camera unit 100 and image sensor 106. That is, they are driven based on the rotation target position α of camera unit 100 set in S904 and the shift target position x of image sensor 106 set in S906.

[0053] In step S908, the rotation control unit 114 and the shift control unit 113 each determine whether or not the control position of either the rotation drive of the camera unit 100 or the shift drive of the image sensor 106 has reached the mechanical drive end. If it is determined that either has reached the drive end, the distortion correction control ends. If neither has reached the drive end, proceed to S909.

[0054] In step S909, the imaging device determines whether pressing the distortion correction button has ended. For example, similar to S901, this is determined based on whether the control signal transmitted from the monitoring device 111 and input via the communication unit 110 indicates pressing of the distortion correction button. If it has ended, the distortion correction control is terminated, and if it has not been pressed, the process returns to S904 and the distortion correction control is continued.

[0055] If the distortion correction button is kept pressed, the processes of S904 to S908 are repeated to perform distortion correction while maintaining the relative position of the subject with respect to the imaging range. Here, the shift drive amount of the image sensor 106 is calculated in response to one-step rotation control of the camera unit 100, but it is also possible to set one-step shift drive amount of the image sensor 106 and calculate the rotation drive amount in response to the shift drive amount. Also, the shift control may be performed by controlling the optical system 203 instead of the image sensor 106.

[0056] As described above, in the first synchronous control, the drive of the camera unit 100 and the image sensor 106 can be synchronized while the control target positions of the two are precisely specified.

[0057] <Second synchronous control> 10 is a diagram for explaining a control method in the second synchronous control. In the second synchronous control, the drive speeds of the rotation control of the camera unit 100 and the shift control of the image sensor 106 are specified at the start of the control, thereby synchronizing the two controls.

[0058] 11 is a flowchart of the process in the second synchronization control. The following operations are performed by each unit of the imaging device. Note that steps S900 to S902, S907, and S909 are the same as those in the first synchronization control, and therefore descriptions thereof will be omitted.

[0059] In step S1100, the image pickup device calculates the shift movement amount x of the image pickup element 106. lim Get the shift movement amount x limis the distance from the current position of the image sensor 106 to the mechanical driving end.

[0060] In step S1101, the imaging device detects the rotational movement amount α of the camera unit 100. lim The rotational movement amount α lim is the angle from the current position of the camera unit 100 to the mechanical driving end.

[0061] In step S1102, the drive amount determination unit 112 determines the maximum shift drive amount x max Calculate the maximum shift drive amount x max is the rotational movement amount α of the camera unit 100 acquired in S1101. lim This is the shift drive amount of the image sensor 106 required to maintain the relative position of the subject with respect to the imaging range when the image sensor 106 is rotated by the maximum shift drive amount x max is calculated based on formula (3).

[0062] In step S1103, the drive amount determination unit 112 determines the shift movement amount x calculated in S1100. lim and the maximum shift drive amount calculated in S1102 x max Compare with x. lim >x max In this case, proceed to S1104. lim >x max In the case of x, when the camera unit 100 is controlled to the mechanical drive end, even if the image sensor 106 is shifted to maintain the relative position of the subject, the image sensor 106 does not strike the mechanical drive end. lim ≦x max In this case, proceed to S1106. lim ≦x max In the case of (a), when the camera unit 100 is controlled to the mechanical drive limit, the image sensor 106 hits the mechanical drive limit, and the relative position of the subject cannot be maintained.

[0063] In step S1104, the drive amount determination unit 112 determines the shift target position of the image sensor 106 as the current position x nowThe maximum shift drive amount x calculated in S1102 max The position is set to the position obtained by adding

[0064] In step S1105, the drive amount determination unit 112 determines the rotation target position of the camera unit 100 based on the current position α now The rotational movement amount α obtained in S1101 lim The position is set to the position obtained by adding

[0065] In step S1106, the drive amount determination unit 112 determines the maximum rotation drive amount α of the camera unit 100. max Calculate the maximum rotation drive amount α max is the shift movement amount x of the image sensor 106 acquired in S1100. lim This is the rotation drive amount of the camera unit 100 required to maintain the relative position of the subject with respect to the imaging range when shift control is performed by only the maximum rotation drive amount α max is calculated based on formula (3).

[0066] In step S1107, the drive amount determination unit 112 determines the shift target position of the image sensor 106 as the current position x now Shift movement amount obtained by S1100 x lim The position is set to the position obtained by adding

[0067] In step S1108, the drive amount determination unit 112 determines the rotation target position of the camera unit 100 as the current position α now The maximum rotation drive amount α calculated in S1106 max The position is set to the position obtained by adding

[0068] In step S1109, the drive amount determination unit 112 determines the rotation speed v r is set in the rotation control unit 114. Note that the rotation control speed v r The slower the , the more fine control the user has in indicating when distortion correction begins and ends.

[0069] In step S1110, the drive amount determination unit 112 determines the shift speed v s is set in the shift control unit 113. At this time, the shift speed v is set so that the camera unit 100 and the image sensor 106 reach the target position at the same time (i.e., the driving times are the same). s Set.

[0070] In step S1111, the rotation control unit 114 and the shift control unit 113 respectively determine whether the camera unit 100 and the image sensor 106 have reached their target positions. If both have reached their target positions, the distortion correction control ends. If at least one of them has not reached their target positions, the process proceeds to S909.

[0071] In step S1112, the imaging device performs a process of stopping distortion correction. The process of stopping the distortion correction is a process of shifting the imaging element 106 to a shift position where the imaging range is maintained with respect to the orientation of the camera unit 100 at the time when the distortion correction end instruction is received, and then ending the control.

[0072] If the linearity between the amount of rotation of camera unit 100 and the amount of shift of image sensor 106 is low, the relative position of the subject with respect to the imaging range may not be maintained completely at a point on the way to the target position. For this reason, when distortion correction is terminated before the control target position by a user operation, the imaging range is adjusted by the stop processing in S1112.

[0073] As described above, in the first synchronous control, the drive speeds of the camera unit 100 and the image sensor 106 are specified at the start of control, thereby making it possible to synchronize the drive of both.

[0074] Both the first and second synchronous controls described above can synchronize the camera unit 100 and the image sensor 106. However, the first synchronous control is superior to the second synchronous control in that it is more effective at maintaining the relative position of the subject during distortion correction control. On the other hand, the second synchronous control is superior to the first synchronous control in shooting conditions that require high-speed control because it requires fewer calculations and fewer drives (one time).

[0075] <Hardware configuration of the imaging device> Next, an example of the hardware configuration of an imaging device will be described with reference to the block diagram of Fig. 20. Note that the configuration shown in Fig. 20 is merely one example of a configuration that can be applied to an imaging device, and can be modified / altered as appropriate.

[0076] The CPU 211 executes processing using computer programs and data stored in the main storage device 212. As a result, the CPU 211 controls the operation of the entire imaging device, and executes or controls each of the processes described above as being performed by the imaging device. For example, the CPU 211 executes processing using computer programs and data stored in the main storage device 212, thereby realizing the functions of each of the functional units of the camera signal processing unit 109, the drive amount determination unit 112, the shift control unit 113, and the rotation control unit 114 shown in FIG. 1.

[0077] The main memory device 212 is a storage device such as a Random Access Memory (RAM). The main memory device 212 has areas for storing computer programs and data loaded from the auxiliary memory device 213, images captured by the imaging unit 215, and various data received from the monitoring device 111 via the communication unit 110. Furthermore, the main memory device 212 has a work area used when the CPU 211 executes various processes. In this way, the main memory device 212 can provide various areas as appropriate.

[0078] The auxiliary storage device 213 is a large-capacity information storage device such as a hard disk drive (HDD), a read only memory (ROM), or a solid state drive (SSD). The auxiliary storage device 213 stores an OS (operating system) and computer programs and data for causing the CPU 211 to execute or control each of the processes described above as being performed by the imaging device. The auxiliary storage device 213 also stores data received from the monitoring device 111 via the communication unit 110. The computer programs and data stored in the auxiliary storage device 213 are loaded into the main storage device 212 as appropriate under the control of the CPU 211, and become targets for processing by the CPU 211.

[0079] The driving unit 214 drives the camera unit 100 based on the imaging parameters received from the monitoring device 111. For example, it corresponds to a shift driving unit 115 that controls the shift of the imaging element 106, and a rotation driving unit 116 that controls the rotation of the camera unit 100. Note that the control target by the driving unit 214 is not limited to a specific target, and may be another target (for example, the position of the camera unit 100).

[0080] The camera unit 100 has an imaging element and an optical system, and forms an image of a subject on the imaging element with the intersection of the optical axis of the optical system and the imaging element as the imaging center. The imaging element may be a complementary metal-oxide semiconductor (CMOS), a charged coupled device (CCD), or the like. The communication unit 110 communicates data with the monitoring device 111.

[0081] As described above, according to the first embodiment, when shift control of the image sensor is performed, rotation control (tilt or pan) of the camera unit is performed in synchronization with the shift control of the image sensor. More specifically, the rotation control is performed in synchronization so as to cancel the change in the imaging range caused by the shift control. This synchronous control makes it possible to suppress the change in the imaging range during correction control, and it becomes possible to provide a high-quality image.

[0082] (Modification) In the modified example, an example will be described in which the imaging device determines the shape of distortion occurring in the subject based on the captured image, and performs rotation control of the camera unit 100 and shift control of the imaging element 106 so as to correct the shape of the distortion.

[0083] FIG. 12 is a flowchart of a process executed by an imaging device according to a modified example.

[0084] In step S1200, the imaging apparatus determines whether or not a distortion correction execution instruction has been received from the user. If it is determined that a distortion correction execution instruction has been received from the user, the process proceeds to S1201. On the other hand, if it is determined that no instruction has been received, the imaging apparatus continues to wait for a distortion correction instruction.

[0085] In step S1201, the imaging device determines whether or not a distorted subject (hereinafter, distorted subject) is present in the captured image. If it is determined that a distorted subject is present, the process proceeds to S1202, and if it is determined that a distorted subject is not present, the process proceeds to S1203. As a method for determining whether or not a distorted subject is present, a method based on depth information and a method based on edge detection will be described below, but the presence or absence of a distorted subject may be determined by other methods.

[0086] 13 is a diagram for explaining the determination of the distorted shape based on the depth information. In the method of determining the presence or absence of a distorted subject based on the depth information (such as a depth image in which each pixel has a depth value at that position), when the subject distance changes monotonically in a certain subject area, it is determined that the subject does not face the imaging device directly and is a distorted subject.

[0087] 14 is a diagram for explaining the determination of the distorted shape based on edge detection. In the method of determining the presence or absence of a distorted object based on edge detection, if the edge interval along the up-down direction (or left-right direction) of a pair of edges detected in a certain object region is not constant, it is determined that the object is distorted.

[0088] In step S1202, the imaging device sets the object determined to be a distorted object in S1201 as a distortion correction target. Depending on the scene, there may be multiple distorted objects in the captured image. In this case, it is preferable to set the object to be corrected by accepting a selection of the object to be corrected from the user.

[0089] Fig. 15 is a diagram for explaining the determination of the distorted shape when multiple subjects are present. Fig. 16 is a diagram for explaining the determination of the correction target when multiple subjects are present. Specifically, Fig. 16 shows a screen for accepting the selection of the correction target subject from the user. Fig. 16 shows a display screen in a form in which multiple polygonal regions obtained by dividing the object determined to be a distorted subject in S1201 are displayed superimposed on the captured image, and the selection of one polygonal region is accepted from the user.

[0090] In step S1203, the imaging apparatus notifies the user that a distorted subject (a subject to be corrected) is not present in the captured image.

[0091] In step S1204, the image capturing apparatus determines the type of distortion shape of the object to be corrected (upper distortion, lower distortion, right distortion, left distortion). In the case of determining the distortion shape, the determination can be based on depth information or edge detection.

[0092] For example, the depth information corresponding to the captured image 1300 in FIG. 13 indicates that the depth (subject distance) for the building region in the captured image 1300 changes from closer to farther from the bottom to the top of the image. Due to the law of perspective, the closer something is, the larger it appears, and the farther it is, the smaller it appears. Therefore, the distorted shape of the building in the captured image 1300 can be determined to be "upper distortion" (large distortion toward the bottom of the image and small distortion toward the top).

[0093] Similarly, the depth information corresponding to the captured image 1301 indicates that the depth (subject distance) for the building region of the captured image 1301 changes from near to far from the left edge to the right edge of the image. Therefore, the distorted shape of the building in the captured image 1301 can be determined to be a "right distortion" (large to the left of the image and small to the right). In this way, by utilizing the depth information, it is possible to determine the distorted shape based on the direction in which the depth of the subject (building) in the captured image changes.

[0094] Also, in the captured image 1400 of FIG. 14, it is shown that the interval between a pair of edges detected for a subject in the captured image narrows from the bottom edge to the top edge of the image. In this case, the distorted shape occurring in the subject can be determined to be "upper distortion". Furthermore, in the captured image 1401, it is shown that the interval between a pair of edges detected for a subject in the captured image narrows from the left edge to the right edge of the image. In this case, the distorted shape occurring in the subject can be determined to be "right distortion". In this way, it is possible to determine the distorted shape of the subject based on the direction in which the edge interval between a pair of edges detected for the subject changes.

[0095] In step S1205, the imaging apparatus sets the driving direction for rotation control of the camera unit 100 and shift control of the imaging element 106 that is suitable for each distortion shape, based on the distortion shape determined in S1204.

[0096] Fig. 17 is a diagram showing the relationship between rotation control and shift control for each distortion shape. As shown in Fig. 17, for each distortion shape, the driving direction of the camera unit 100 and the image sensor 106 for correcting it can be uniquely determined.

[0097] In step S1206, the imaging device performs distortion correction driving. Here, it is assumed that distortion correction driving (step driving) is performed in predetermined increments and S1206 to S1208 are repeated. Note that this process is similar to the process (S902 to S907) described above with reference to FIG. 9, so a description thereof will be omitted.

[0098] In step S1207, the image capturing apparatus calculates the current amount of distortion occurring in the subject to be corrected. The amount of distortion can be calculated based on depth information or edge detection.

[0099] FIG. 18 is a diagram for explaining the calculation of the amount of distortion based on the depth information. FIG. 18 exemplarily illustrates a change in the captured image during the distortion correction drive of S1206. The captured image 1800 shows that the difference in depth (object distance) is large between the upper and lower parts of the object to be corrected. In this case, the amount of distortion is calculated as a large value. The captured image 1801 obtained after one or more steps of distortion correction drive is performed has a smaller difference in depth between the upper and lower parts of the object to be corrected compared to the captured image 1800. Therefore, the amount of distortion is calculated as a small value compared to the captured image 1800. Furthermore, the captured image 1802 obtained after the distortion correction drive is performed has approximately the same depth between the upper and lower parts of the object to be corrected. Therefore, the amount of distortion is calculated as a value close to 0. When the amount of distortion is calculated based on the depth information in this way, it is possible to evaluate the depth difference between multiple regions (for example, the upper and lower parts) of the object to be corrected as the amount of distortion.

[0100] FIG. 19 is a diagram for explaining the calculation of the amount of distortion based on edge detection. FIG. 19 exemplarily illustrates a change in a captured image during the distortion correction drive of S1206. A captured image 1900 shows that the difference in edge interval between a pair of edges at the upper and lower parts of a subject to be corrected is large. In this case, the amount of distortion is calculated as a large value. A captured image 1901 obtained after one or more steps of distortion correction drive is performed has a smaller difference in edge interval between the upper and lower parts of a subject to be corrected compared to the captured image 1900. Therefore, the amount of distortion is calculated as a small value compared to the captured image 1900. Furthermore, a captured image 1902 obtained after the distortion correction drive is performed has approximately the same edge interval between the upper and lower parts of a subject to be corrected. Therefore, the amount of distortion is calculated as a value close to 0. When the amount of distortion is calculated based on edge detection in this way, it is possible to evaluate the difference in edge interval between multiple regions (for example, the upper and lower parts) of the subject to be corrected as the amount of distortion.

[0101] In step S1208, the imaging device determines whether the amount of distortion calculated in S1207 is equal to or less than a predetermined threshold. If the amount of distortion is not equal to or less than the predetermined threshold, the process returns to S1206 because distortion correction is insufficient. On the other hand, if the amount of distortion is equal to or less than the predetermined threshold, the process proceeds to S1209.

[0102] In step S1209, the imaging device determines whether the distortion shape determination (S1204) has been performed twice. This is because, depending on the subject, there may be cases where both vertical distortion (upper or lower distortion) and horizontal distortion (right or left distortion) exist in the captured image. In the correction based on a single distortion shape determination (S1204 to S1208), only the vertical or horizontal distortion is corrected, so it is determined whether the distortion shape determination has been performed twice.

[0103] As described above, the imaging device is capable of correcting distortion occurring in a subject of a captured image, based on depth information corresponding to the captured image and the result of edge detection on the captured image.

[0104] The disclosure of this specification includes the following imaging device, control method, and program. (Item 1) an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; a synchronization control means for driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in the imaging range of the imaging unit caused by driving the other is cancelled out; a determination means for sequentially determining a distortion amount of a subject in an image captured by the imaging unit; having The synchronization control means stops driving when the distortion amount determined by the determination means becomes equal to or less than a predetermined threshold value. 1. An imaging device comprising: (Item 2) The shift control means moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface so that distortion of a subject in a captured image obtained by the imaging section is compensated for. 2. The imaging device according to item 1, (Item 3) The synchronization control means determines a second drive amount of the one of the two based on a first drive amount of the other one of the two. 3. The imaging device according to item 1 or 2, (Item 4) The synchronization control means further determines the second driving amount based on a size of the image sensor and an image forming distance from the image pickup optical system to the image sensor. 4. The imaging device according to item 3, (Item 5) The shift control means further includes a receiving means for receiving a user instruction for the shift control means. 5. The imaging device according to any one of items 1 to 4, (Item 6) the receiving means is configured to sequentially receive a drive instruction to drive the shift control means by a predetermined amount, When the drive instruction is received by the reception means, the synchronization control means drives the shift control means by a predetermined amount based on the drive instruction, and sequentially determines and drives the attitude control means by a drive amount corresponding to the predetermined amount of drive. 6. The imaging device according to item 5, (Item 7) The receiving means is configured to receive a drive instruction to drive the shift control means, When the drive instruction is received by the reception means, the synchronization control means determines drive speeds of both the attitude control means and the shift control means and starts driving. 6. The imaging device according to item 5, (Item 8) When the drive instruction is received by the reception means, the synchronization control means determines the drive speed so that the drive times of the attitude control means and the shift control means are the same. 8. The imaging device according to item 7, (Item 9) the receiving means is configured to further receive a drive stop instruction to stop driving the shift control means, When the reception means receives the drive stop instruction while the attitude control means and the shift control means are being driven, the synchronization control means adjusts one of the orientation of the imaging unit and the shift position of the imaging element based on the other at the timing when the drive stop instruction is received. 9. The imaging device according to item 7 or 8, (Item 10) The drive instruction includes information on the shift direction of the image sensor to be shifted by the shift control means. 10. The imaging device according to any one of items 6 to 9, (Item 11) The shift control means further includes a receiving means for receiving a drive start instruction for repeatedly driving the shift control means by a predetermined amount, The shift control means starts driving when the driving start instruction is received by the receiving means. 5. The imaging device according to any one of items 1 to 4, (Item 12) The imaging device further includes a depth acquisition unit that acquires depth information corresponding to the captured image obtained by the imaging unit, The determining means determines the amount of distortion based on a difference in depth within a subject area included in the captured image. 12. The imaging device according to item 11, (Item 13) The imaging device further includes an edge detection unit that detects an edge of a subject included in an image captured by the imaging unit, The determining means determines the amount of distortion based on a difference in edge interval between a pair of edges detected for the subject by the edge detecting means. 12. The imaging device according to item 11, (Item 14) The imaging device further includes a determination unit for determining a distortion shape type of a subject in a captured image obtained by the imaging unit, The driving directions of the attitude control means and the shift control means are determined based on the distortion shape type determined by the determination means. 5. The imaging device according to any one of items 1 to 4, (Item 15) The imaging device further includes a depth acquisition unit that acquires depth information corresponding to the captured image obtained by the imaging unit, The determining means determines the distortion shape type based on a direction of change in depth within a subject area included in the captured image. 15. The imaging device according to item 14, (Item 16) The imaging device further includes an edge detection unit that detects an edge of a subject included in an image captured by the imaging unit, The determining means determines the distortion shape type based on a direction of change in an edge interval between a pair of edges detected for the subject by the edge detecting means. 15. The imaging device according to item 14, (Item 17) The image capturing device further includes a receiving unit that receives a selection of one of the objects to be determined by the determining unit when there are a plurality of objects having distortion in the captured image. 17. The imaging device according to any one of items 14 to 16, (Item 18) A control method for an imaging device, comprising: The imaging device includes: an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; having The control method includes: a determining step of sequentially determining a distortion amount of a subject in an image captured by the imaging unit; a synchronization control step of driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in an imaging range of the imaging unit caused by driving the other is cancelled out; Including, In the synchronization control step, when the distortion amount determined in the determination step becomes equal to or smaller than a predetermined threshold value, the driving of the attitude control means and the shift control means is stopped. A control method comprising: (Item 19) A program for causing a computer to execute a control method for an imaging device, The imaging device includes: an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; having The control method includes: a determining step of sequentially determining a distortion amount of a subject in an image captured by the imaging unit; a synchronization control step of driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in an imaging range of the imaging unit caused by driving the other is cancelled out; Including, In the synchronization control step, when the distortion amount determined in the determination step becomes equal to or smaller than a predetermined threshold value, the driving of the attitude control means and the shift control means is stopped. A program characterized by:

[0105] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0106] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0107] 100 camera unit; 101 zoom lens; 102 focus lens; 103 aperture unit; 104 band pass filter; 105 color filter; 106 imaging element; 107 AGC; 108 AD converter; 109 camera signal processing unit; 110 communication unit; 111 monitoring device; 112 drive amount determination unit; 113 shift control unit; 114 rotation control unit; 115 shift drive unit; 116 rotation drive unit

Claims

1. an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; a synchronization control means for driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in the imaging range of the imaging unit caused by driving the other is cancelled out; a determination means for sequentially determining a distortion amount of a subject in an image captured by the imaging unit; A receiving means for receiving a user instruction for the shift control means; having The synchronization control means stops driving when the distortion amount determined by the determination means becomes equal to or less than a predetermined threshold value, The receiving means is configured to receive a drive stop instruction to stop driving of the shift control means, When the reception means receives the drive stop instruction while the attitude control means and the shift control means are being driven, the synchronization control means adjusts one of the orientation of the imaging unit and the shift position of the imaging element based on the other at the timing when the drive stop instruction is received.

1. An imaging device comprising:

2. The shift control means moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface so that distortion of a subject in a captured image obtained by the imaging section is compensated for.

2. The imaging device according to claim 1 .

3. The synchronization control means determines a second drive amount of the one of the two based on a first drive amount of the other one of the two.

3. The imaging device according to claim 2.

4. The synchronization control means further determines the second driving amount based on a size of the image sensor and an image forming distance from the image sensing optical system to the image sensor.

4. The imaging device according to claim 3.

5. the receiving means is configured to sequentially receive a drive instruction to drive the shift control means by a predetermined amount, When the drive instruction is received by the reception means, the synchronization control means drives the shift control means by a predetermined amount based on the drive instruction, and sequentially determines and drives the attitude control means by a drive amount corresponding to the predetermined amount of drive.

2. The imaging device according to claim 1 .

6. The receiving means is configured to receive a drive instruction to drive the shift control means, When the drive instruction is received by the reception means, the synchronization control means determines drive speeds of both the attitude control means and the shift control means and starts driving.

2. The imaging device according to claim 1 .

7. When the drive instruction is received by the reception means, the synchronization control means determines the drive speed so that the drive times of the attitude control means and the shift control means are the same.

7. The imaging device according to claim 6.

8. The drive instruction includes information on the shift direction of the image sensor to be shifted by the shift control means.

6. The imaging device according to claim 5.

9. The shift control means further includes a receiving means for receiving a drive start instruction for repeatedly driving the shift control means by a predetermined amount, The shift control means starts driving when the driving start instruction is received by the receiving means.

2. The imaging device according to claim 1 .

10. The imaging device further includes a depth acquisition unit that acquires depth information corresponding to the captured image obtained by the imaging unit, The determining means determines the amount of distortion based on a difference in depth within a subject area included in the captured image.

10. The imaging device according to claim 9.

11. The imaging device further includes an edge detection unit that detects an edge of a subject included in an image captured by the imaging unit, The determining means determines the amount of distortion based on a difference in edge interval between a pair of edges detected for the subject by the edge detecting means.

10. The imaging device according to claim 9.

12. The imaging device further includes a determination unit for determining a distortion shape type of a subject in a captured image obtained by the imaging unit, The driving directions of the attitude control means and the shift control means are determined based on the distortion shape type determined by the determination means.

2. The imaging device according to claim 1 .

13. The imaging device further includes a depth acquisition unit that acquires depth information corresponding to the captured image obtained by the imaging unit, The determining means determines the distortion shape type based on a direction of change in depth within a subject area included in the captured image.

13. The imaging device according to claim 12.

14. The imaging device further includes an edge detection unit that detects an edge of a subject included in an image captured by the imaging unit, The determining means determines the distortion shape type based on a direction of change in an edge interval between a pair of edges detected for the subject by the edge detecting means.

13. The imaging device according to claim 12.

15. The image capturing device further includes a receiving unit for receiving a selection of one of the objects to be determined by the determining unit when there are a plurality of objects having distortion in the captured image.

13. The imaging device according to claim 12.

16. A control method for an imaging device, comprising: The imaging device includes: an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; A receiving means for receiving a user instruction for the shift control means; having The control method includes: a determining step of sequentially determining a distortion amount of a subject in an image captured by the imaging unit; a synchronization control step of driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in an imaging range of the imaging unit caused by driving the other is cancelled out; Including, In the synchronization control step, when the distortion amount determined in the determination step becomes equal to or smaller than a predetermined threshold value, the driving of the attitude control means and the shift control means is stopped, In the synchronization control step, when a drive stop instruction to stop driving of the shift control means is received by the reception means while the attitude control means and the shift control means are being driven, one of the orientation of the imaging unit and the shift position of the imaging element at a timing when the drive stop instruction is received is adjusted based on the other. A control method comprising:

17. A program for causing a computer to execute a control method for an imaging device, The imaging device includes: an imaging unit having an imaging optical system and an imaging element; An attitude control means for moving the orientation of the imaging unit in a pan direction and / or a tilt direction; a shift control unit that moves at least one of the imaging optical system and the imaging element in a plane parallel to an imaging surface; A receiving means for receiving a user instruction for the shift control means; having The control method includes: a determining step of sequentially determining a distortion amount of a subject in an image captured by the imaging unit; a synchronization control step of driving one of the attitude control means and the shift control means in synchronization with driving of the other so that a change in an imaging range of the imaging unit caused by driving the other is cancelled out; Including, In the synchronization control step, when the distortion amount determined in the determination step becomes equal to or smaller than a predetermined threshold value, the driving of the attitude control means and the shift control means is stopped, In the synchronization control step, when a drive stop instruction to stop driving of the shift control means is received by the reception means while the attitude control means and the shift control means are being driven, one of the orientation of the imaging unit and the shift position of the imaging element at a timing when the drive stop instruction is received is adjusted based on the other. A program characterized by:

Images