Imaging device, method, program, and storage medium

The imaging device uses motion vectors to adjust the zoom and stabilize images by ensuring the shake detection area remains within the field of view, enhancing image stabilization performance.

JP7580955B2Active Publication Date: 2024-11-12CANON KK
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2020115057
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-07-02
Publication Date
2024-11-12
Estimated Expiration
2040-07-02

AI Technical Summary

Technical Problem

Existing image stabilization techniques struggle to effectively utilize motion vectors for precise control of image stabilization, leading to suboptimal performance in detecting and correcting image shake.

Method used

An imaging device with a zoom driving mechanism, shake detection, and correction system that adjusts the angle of view to ensure the shake detection area remains within the field of view, using motion vectors to maintain image stabilization performance across varying angles.

Benefits of technology

The solution ensures effective image stabilization by maintaining the shake detection area within the field of view, reducing shake in captured images, and minimizing restrictions on zoom operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580955000003
    Figure 0007580955000003
  • Figure 0007580955000004
    Figure 0007580955000004
  • Figure 0007580955000005
    Figure 0007580955000005
Patent Text Reader

Abstract

To perform excellent antivibration control using motion vectors.SOLUTION: An imaging apparatus includes: image capturing means configured to capture an image formed by an imaging optical system; zoom driving means configured to change an angle-of-view of the imaging optical system; extraction means configured to extract an area in which a motion vector can be detected in the image captured by the imaging means; shake detection means configured to detect an amount of shake of the image on the basis of the motion vector in the area extracted by the extraction means; deflection correcting means configured to correct shake of the image on the basis of the amount of shake detected by the shake detection means; and control means configured to control the zoom driving means to prevent the area extracted by the extraction means from deviating from a field of view of the imaging optical system.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, a method, a program, and a storage medium. [Background technology]

[0002] Conventionally, there has been known a technique for controlling optical image stabilization so that a subject falls within a detection area (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-106947 A Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to perform good vibration reduction control using a motion vector. [Means for solving the problem]

[0005] An imaging device according to one aspect of the present invention includes an imaging optical system. The image of the subject is captured through an imaging means for capturing an image; a zoom driving means for changing an angle of view of the imaging optical system; By Imaging Will be image Based on the above, an image captured by the imaging means Detect motion vectors in do Area decision do decision Means and the said decision By means decision The above-mentioned area Detected from Based on motion vectors 、 The imaging means By Imaging The picture A shake detection means for detecting an amount of shake of an image, and an image captured by the imaging means based on the amount of shake detected by the shake detection means. picture A shake correction means for correcting image shake; decision By means decision The area is the imaging optical system. Angle of View and a control means for controlling the zoom driving means so as not to deviate from the zoom position. do. [Brief description of the drawings]

[0006] [Figure 1] 1 is a functional block diagram of a network camera according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing the mechanism of the network camera according to the first embodiment. [Diagram 3] 5A to 5C are diagrams illustrating the movement of a camera head during anti-shake control in the first embodiment. [Figure 4] 4 is a flowchart illustrating an image stabilization control process according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of the relationship between a shake detection area in a captured image and a specified angle of view in the first embodiment; [Figure 6] 4 is a flowchart illustrating a zoom control process in the first embodiment. [Figure 7] 5 is a flowchart illustrating a zoom control process using digital zoom in the first embodiment. [Figure 8] 13A and 13B are diagrams illustrating an example of the relationship between a shake detection area in a captured image and a specified angle of view in the second embodiment. [Figure 9] 13A to 13C are diagrams illustrating an example of the relationship between a shake detection area in a captured image and a specified angle of view in the third embodiment. [Figure 10] FIG. 13 is a functional block diagram of a network camera according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] (First embodiment) 1 is a functional block diagram of a network camera according to this embodiment. A network camera 1000 is connected to a client device (information processing device) (not shown) via a network 3000 so as to be able to communicate with each other.

[0008] The network camera 1000 includes an imaging unit 1001 , an image processing unit 1002 , a system control unit 1003 , a shake detection unit 1004 , a zoom driving unit 1005 , a pan driving unit 1006 , a tilt driving unit 1007 , a pan / tilt / zoom control unit 1008 , and a communication unit 1009 .

[0009] The configuration and functions of each part of the network camera 1000 will be described with reference to FIG.

[0010] The imaging unit 1001 is composed of a lens (image pickup optical system), an image pickup element, and a control group thereof, and captures an image of a subject and converts it into an electrical signal.

[0011] The image processing unit 1002 performs predetermined development processing and compression encoding processing on the signals captured and photoelectrically converted by the imaging unit 1001 to generate image data and transmits it to the system control unit 1003. The image processing unit 1002 also performs processing to analyze the captured image and detect image shake using a motion vector. Information on the detected shake is transmitted to the pan-tilt-zoom control unit 1008 and used for vibration reduction control, which will be described later.

[0012] The system control unit 1003 distributes the generated image data to the client device via the communication unit 1009. The communication unit 1009 also receives (accepts) a camera control command sent from the client device and transmits it to the system control unit 1003. It also transmits a response to the camera control command to the client device. The system control unit 1003 analyzes the transmitted camera control command and performs processing according to the command. For example, it instructs the image processing unit 1002 to set image quality and instructs the pan / tilt / zoom control unit 1008 to perform pan, tilt, and zoom operations. The image processing unit 1002 performs image processing based on the transmitted instructions. The pan / tilt / zoom control unit 1008 also controls the zoom driving unit 1005, pan driving unit 1006, and tilt driving unit 1007 based on the transmitted instructions.

[0013] The shake detection unit 1004 is configured with a gyro sensor or the like, and detects the angular velocity in the pan direction and tilt direction of the imaging unit 1001. The detected angular velocity information is transmitted to a pan / tilt control unit 1007 and used for vibration isolation control, which will be described later.

[0014] The zoom driving unit 1005 is composed of a mechanical driving system that performs zooming operation, a motor as its driving source, an encoder that detects the angle of the driving unit, etc. The pan driving unit 1006 is composed of a mechanical driving system that performs panning operation, a motor as its driving source, a sensor that detects the angle of the driving unit, etc. The tilt driving unit 1006 is composed of a mechanical driving system that performs tilting operation, a motor as its driving source, a sensor that detects the angle of the driving unit, etc. The operations of the zoom driving unit 1005, the pan driving unit 1006, and the tilt driving unit 1007 are controlled by a pan / tilt / zoom control unit 1008.

[0015] Fig. 2 is a diagram showing the mechanism of the network camera capable of pan-tilt operation according to this embodiment. Fig. 2(a) shows a view from above, and Fig. 2(b) shows a view from the side. In Fig. 2, 1101 is a bottom case, 1102 is a turntable, 1103 is a camera head support, and 1104 is a camera head.

[0016] The mechanical mechanism of the pan-tilt drive unit will be described with reference to Fig. 2. In Fig. 2, the pan drive unit is composed of a bottom case 1101 and a turntable 1102, and the turntable 1102 rotates in the horizontal direction. The pan drive unit of the network camera of this embodiment can rotate left and right from -175 degrees to +175 degrees.

[0017] The tilt drive unit is composed of a camera head support 1103 and a camera head 1104 provided on a turntable 1102, and the camera head 1104 rotates vertically. The tilt drive unit of the network camera of this embodiment can rotate from -45° diagonally downward to 90° directly upward, with the horizontal direction being 0°.

[0018] In this way, the network camera of this embodiment can change the shooting direction by rotating the camera head horizontally and vertically. Also, by using this pan-tilt driving unit to drive the camera head so as to cancel the shake detected by the shake detection unit, it is configured to shoot an image with reduced shake.

[0019] Fig. 3 is a diagram showing the movement of the camera head during PT vibration isolation control. In Fig. 3, 1104A indicates the camera head when there is no shake, 1104B indicates the camera head when it is tilted forward due to shake, 1104C indicates the camera head when it is tilted backward due to shake, and 2001 indicates the shooting direction.

[0020] 3, when there is no shake, the camera head faces horizontally as shown by 1104A, and captures the image in the shooting direction shown by 1201. If the camera tilts backward from this state due to shaking of the installation surface as shown by 1104C, the tilt angle is controlled downward to maintain the shooting direction shown by 1201. Also, if the camera tilts forward due to shaking of the installation surface as shown by 1104B, the tilt angle is controlled upward to maintain the shooting direction shown by 1201. In this way, with PT vibration reduction control, the tilt of the camera head due to shaking of the installation surface is detected, and the PT drive unit is controlled to maintain the shooting direction, thereby capturing an image with reduced shake.

[0021] 4 is a flowchart illustrating the PT vibration reduction control process S1000. When the camera starts shooting, the pan-tilt-zoom control unit 1008 proceeds to the process of S1000.

[0022] In step S1001 , the pan / tilt / zoom control unit 1008 acquires the angular velocity in the pan direction and the angular velocity in the tilt direction from the shake detection unit 1004 .

[0023] In step S1002, the pan / tilt / zoom control unit 1008 calculates a pan angle value by integrating the acquired angular velocity in the pan direction, and also calculates a tilt angle value by time-integrating the acquired angular velocity in the tilt direction.

[0024] In step S1003, the image processing unit 1002 calculates a motion vector. The motion vector is obtained by using, for example, block matching. In block matching, the entire image is divided into blocks of, for example, 16 pixels x 16 pixels. For each divided block, the block is slid in the previous frame to search for the most matching part. To determine the match, the absolute value of the brightness difference for each pixel in the block is taken, and the sum is used as an evaluation value, and the part where this evaluation value is the smallest is searched for. The amount of movement of the target block is obtained as a motion vector from the position of the most matching part in the previous frame and the position in the current frame. Other blocks are searched in the same way, and motion vectors are obtained for all blocks. The motion vectors of each block calculated in this way are averaged to obtain the motion vector for the entire image. The pan-tilt-zoom control unit 1008 obtains the motion vector obtained by the image processing unit 1002.

[0025] In step S1004, the pan / tilt / zoom control unit 1008 resolves the motion vector into a pan (horizontal) direction and a tilt (vertical) direction, converts them into angle values, and calculates pan angle values ​​and tilt angle values.

[0026] In step S1005, the pan / tilt / zoom control unit 1008 calculates a correction angle (correction amount) in the pan direction from the calculated pan angle value. It also calculates a correction angle (correction amount) in the tilt direction from the calculated tilt angle value. The correction angle is calculated so as to cancel the angle caused by the shake. For example, when the tilt angle value is 0.1 degrees, the correction angle is set to -0.1 degrees. For the shake detected by the gyro sensor and the shake detected by the motion vector, effective information is used according to the characteristics. For example, the shake detection method of the motion vector is mainly used for low frequency shake, while the shake detection method of the gyro sensor is mainly used for high frequency shake.

[0027] In step S1006, the pan-tilt-zoom control unit 1008 issues drive instructions to the pan driving unit 1006 and the tilt driving unit 1007 in accordance with the calculated correction angles for the pan direction and tilt direction, and ends the process.

[0028] As described above, in the PT vibration reduction control, the pan / tilt drive unit is controlled so as to cancel the shake detected in each of the pan and tilt directions, thereby reducing the shake.

[0029] Fig. 5 is a diagram showing an example of an image captured when the camera of this embodiment is installed on a ship and captures an image in the direction of the sea. In Fig. 5, 2001 indicates the imaging angle of view at the wide-angle end, 2002 indicates the shake detection area, 2003A and 2003B indicate the PTZ designated angle of view (imaging range), and 2004 indicates the shake detectable angle of view.

[0030] As shown in Figure 5, when the image is taken at the wide-angle end angle of view 2001, an island is included in the upper part of the image. These areas shown in 2002 are fixed and include subjects with feature points such as edges, making it possible to detect shake using motion vectors. Other areas, such as the sky and sea, have few feature points and therefore shake cannot be detected correctly. At the wide-angle end angle of view 2001, the shake detectable area shown in 2002 is used to detect shake using motion vectors as explained above, and anti-shake control is performed according to the detected shake.

[0031] When the angle of view is changed from this state by PTZ operation, for example, when the angle of view is changed to the angle shown by 2003A, the motion vector detectable area 2002 is included in the shooting angle of view, so shake can be detected and anti-shake control can be performed effectively. On the other hand, when the angle of view is changed to the angle shown by 2003B, the motion vector detectable area 2002 is not included in the shooting angle of view, so shake cannot be detected correctly. As a result, anti-shake control cannot be performed effectively and an image with shake is obtained. Therefore, instead of optically zooming to the specified angle of view of 2003B, the optical zoom is limited to the angle of view shown by 2004 which includes the motion vector detectable area 2002 on the wider angle side than the angle of view, so that shake can be detected based on the motion vector. This maintains a state in which anti-shake control can be performed effectively, and an image with reduced shake can be shot.

[0032] 6 is a flowchart illustrating the zoom control process S2000. When the camera receives a PTZ operation instruction, the pan-tilt-zoom control unit 1008 proceeds to the process of S2000.

[0033] In step S2001, the pan / tilt / zoom control unit 1008 extracts a shake detection area. In extracting the shake detection area, an area including a fixed subject having feature points such as edges from which a motion vector can be correctly detected is extracted. For example, edge detection is used to detect a point in the image where the brightness suddenly changes, and the outline of the object in the captured image is extracted. The area surrounded by the outline is regarded as the shake detection area. For example, the area indicated by 2002 in FIG. 5 is extracted as the shake detection area.

[0034] In step S2002, the pan / tilt / zoom control unit 1008 determines whether or not the shake detection area is included in the specified angle of view. If it is included, the process proceeds to S2003, and if it is not included, the process proceeds to S2004.

[0035] In step S2003, the pan-tilt-zoom control unit 1008 performs optical zoom up to the designated angle of view.

[0036] In step S2004, the pan-tilt-zoom control unit 1008 calculates the shake detectable angle of view. In calculating the shake detectable angle of view, the specified angle of view is expanded to the wide-angle side, and the angle of view that includes the shake detection area extracted in step 2001 is calculated. For example, the angle of view at the wide-angle end is assumed to be 40 degrees horizontally and 30 degrees vertically, and is expressed by coordinates of -20 degrees to +20 degrees horizontally and -15 degrees to +15 degrees vertically with the center at 0 degrees. In this case, the shake detection area is assumed to be -17 degrees to +5 degrees horizontally and 8 degrees to 9.5 degrees vertically. Then, assume that a PTZ operation of pan +10 degrees, tilt +2 degrees, and zoom with a horizontal angle of view of 16 degrees is instructed. When the angle of view when the PTZ operation is performed is expressed on the coordinate system, it is in the range of +2 degrees to +18 degrees horizontally and -4 degrees to +8 degrees vertically, which is a range that does not include the shake detection area. It is expanded from this state to a range that includes the shake detection area.

[0037] The difference between the specified angle of view range of 8 degrees vertically and the shake detection area range of 9.5 degrees vertically is: 9.5-8=1.5 degrees If the vertical angle of view is expanded by 1.5 degrees up and down, it becomes 15 degrees from -5.5 degrees to +9.5 degrees. At that time, the vertical angle of view is 15×4 / 3=20 degrees The field angle ranges from 0 degrees to +20 degrees horizontally and from -5.5 degrees to +9.5 degrees vertically, and is the field angle that includes the shake detection area. In this way, the field angle where shake detection is possible is determined to be 20 degrees.

[0038] As an example, the angle of view is expanded to a wide angle with the specified angle of view as the center, but this is not limiting. For example, by shifting the center of the specified angle of view toward the shake detection area, the range in which the angle of view is expanded to include the shake detection area can be narrowed, and the use restrictions on the optical zoom can be further reduced.

[0039] In step S2005, the pan-tilt-zoom control unit 1008 performs optical zoom up to a shake-detectable angle of view, and ends the process.

[0040] As described above, in image stabilization control using image stabilization detection based on motion vectors, an area in a captured image where a motion vector can be detected is extracted, and an optical zoom value is calculated so that the extracted area does not deviate from the angle of view (field of view). Then, by performing zoom control using this optical zoom value, it is possible to perform image angle control that takes image stabilization performance into consideration at any angle of view.

[0041] In this embodiment, the method of limiting the zoom operation to include the shake detection area has been described, but the present invention is not limited to this. For example, digital zoom may be performed from the angle of view of the limited optical zoom to the specified angle of view. This allows the specified angle of view to be captured while maintaining the image stabilization performance, and reduces the limitations on the angle of view designation.

[0042] 7 is a flowchart illustrating a zoom control process S3000 using digital zoom. When the camera receives a PTZ operation instruction, the pan-tilt-zoom control unit 1008 proceeds to the process of S3000.

[0043] In step S3001, the pan-tilt-zoom control unit 1008 extracts a shake detection area in the same manner as in the method described above.

[0044] In step S3002, the pan-tilt-zoom control unit 1008 determines whether the shake detection angle of view is included in the specified angle of view. If it is included, the process proceeds to S3003, and if it is not included, the process proceeds to S3004.

[0045] In step S3003, the pan-tilt-zoom control unit 1008 performs optical zoom up to the designated angle of view.

[0046] In step S3004, the pan-tilt-zoom control unit 1008 calculates the shake-detectable angle of view in the same manner as described above.

[0047] In step S3005, the pan-tilt-zoom control unit 1008 calculates the digital zoom magnification. For example, in the example described above, when the optical zoom is limited to a shake detectable angle of view of 20 degrees with respect to a specified angle of view of 16 degrees, 20 / 16=1.25 Therefore, the digital zoom magnification is set to 1.25x.

[0048] In step S3006, the pan-tilt-zoom control unit 1008 performs optical zoom up to an angle at which a shake detection image can be captured.

[0049] In step S3007, the pan-tilt-zoom control unit 1008 performs digital zoom up to the digital zoom magnification, and ends the process.

[0050] In addition, in the present embodiment, a method for limiting zooming to include a shake detection area based on a motion vector has been described, but the present invention is not limited to this. For example, depending on the situation, there may be cases where optical zooming is desired even if some shake is tolerated. In this regard, a control method for limiting optical zooming by prioritizing anti-shake control and a control method for prioritizing optical zooming by allowing shake may be provided, and the control methods may be made switchable.

[0051] In addition, in the present embodiment, a method of restricting zoom operation to include a shake detection area based on a motion vector has been described, but the present invention is not limited to this. For example, a plurality of shake detection means may be provided, including a shake detection means based on a motion vector and a shake detection means other than the motion vector, such as a gyro sensor. In an angle of view where shake detection based on a motion vector is effective, the shake detected by the shake detection means based on a motion vector is mainly used. In an angle of view where shake detection based on a motion vector is not effective, the shake prevention control can be performed by mainly using the shake detected by the shake detection means other than the motion vector.

[0052] In addition, in this embodiment, the PT anti-shake control means that performs shake correction using a pan-tilt drive unit is shown as the anti-shake control means, but is not limited to this. For example, an optical anti-shake control means (correction optical system) that optically corrects shake, or an electronic anti-shake control means that corrects shake by image clipping may be used.

[0053] Second Embodiment Next, a second embodiment of the present invention will be described with reference to Fig. 8, mainly in the case where the magnitude of the residual shake due to vibration isolation control is taken into consideration. The same components as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment, and detailed descriptions thereof will be omitted, and the description will focus on the differences from the first embodiment. This method of omitting descriptions will be the same in the other embodiments described later.

[0054] Fig. 8 is a diagram showing an example of an image captured when the camera of this embodiment is installed on a ship and captures an image in the direction of the sea with different residual shake due to vibration isolation control. In Fig. 8, 2001 indicates the imaging angle of view at the wide-angle end, 2002 indicates the shake detection area, 2003 indicates the PTZ operation angle of view, 2004A indicates the angle of view where shake detection is possible when the residual shake is large, and 2004B indicates the angle of view where shake detection is possible when the residual shake is small.

[0055] 8, when the angle of view 2003 is specified, instead of optically zooming to the specified angle of view 2003 as described above in the first embodiment, the optical zoom is limited to an angle of view indicated by 2004A that includes a shake detectable area 2002 on the wider angle side than the angle of view 2003. This makes it possible to detect shake using a motion vector, and maintains a state in which vibration reduction control can be performed effectively.

[0056] In PT anti-shake control, if the shake has a magnitude or frequency that exceeds the responsiveness of the PT drive unit, the shake cannot be tracked and residual shake occurs. If the shake detectable angle of view is set in a state where the shake detection area is included just inside the edge of the angle of view, the shake detection area will fall outside the angle of view in a situation where residual shake occurs during anti-shake control. Therefore, when setting the shake detectable angle of view, it is necessary to provide a margin so that the shake detection area does not fall outside the angle of view even if residual shake occurs. However, if the margin is set too large, the limitation on the range of use of the optical zoom will be expanded. Therefore, the shake detection area is changed depending on the size of the residual shake, and when the residual shake is small, the shake detectable area is kept small. This makes it possible to appropriately set the shake detectable angle of view and reduce unnecessary limitations on the range of use of the optical zoom.

[0057] The processing of this embodiment will be described mainly with respect to different processing, while omitting similar processing, using the flowchart of Fig. 6. When the camera receives a PTZ operation instruction, the pan-tilt-zoom control unit 1008 proceeds to processing of S2000.

[0058] Steps S2001 to S2003 are performed in the same manner as described above.

[0059] In step S2004, the pan / tilt / zoom control unit 1008 calculates the shake detectable angle. Here, the shake detectable angle is set to 20 degrees, as in the example described above. A margin is added to this, taking into account the remaining shake angle. For example, if the shake angle in the tilt direction is 1.5 degrees, and the tilt angle that can respond to that shake frequency is 1 degree, then the remaining shake angle is 1.5-1=0.5 The shake detectable angle is calculated by adding a margin of 0.5 degrees to 20 degrees, giving 20.5 degrees. For example, if the responsive tilt angle under different conditions is 1.2 degrees, the shake remaining angle is 1.5-1.2=0.3 The detectable shake angle is calculated by adding a margin of 0.3 degrees to 20 degrees, resulting in 20.3 degrees.

[0060] The subsequent processing is performed in the same manner as described above.

[0061] As described above, in image stabilization control using image shake detection based on motion vectors, an area in a captured image where a motion vector can be detected is extracted, and an optical zoom value that does not deviate from the angle of view is calculated according to the magnitude of the remaining shake. By controlling the zoom using this optical zoom value, image angle control that takes image stabilization performance into account at any angle of view can be performed, and limitations on the range of use of the optical zoom can be reduced.

[0062] (Third embodiment) Next, a third embodiment of the present invention will be described, mainly focusing on a case where there are multiple shake detection regions, with reference to Fig. 9. The same components as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment, and detailed descriptions thereof will be omitted, and differences from the first embodiment will be mainly described.

[0063] Fig. 9 is a diagram showing an example of a captured image in which there are multiple shake detection areas when the camera of this embodiment is installed on a ship and captures an image in the direction of the sea. In Fig. 9, 2001 indicates the imaging angle of view at the wide-angle end. Also, 2002A and 2002B indicate shake detection areas. Also, 2003C and 2003D indicate PTZ designated angles of view. Also, 2004C indicates the angle of view at which shake detection is possible when the angle of view of 2003C is designated. Also, 2004D indicates the angle of view at which shake detection is possible when 2003D is designated.

[0064] As shown in Figure 9, when the image is shot at the wide-angle end angle of view 2001, two island shadows are included on the upper right and left sides of the image. The areas indicated by 2002A and 2002B include subjects that are fixed and have characteristic points such as edges, and shake can be detected using motion vectors. When the image is shot at the wide-angle end angle of view 2001, the shake detectable areas indicated by 2002A and 2002B are used to detect shake using motion vectors as explained above, and anti-shake control is performed according to the detected shake.

[0065] In this way, when there are multiple shake detection areas, if the shake detectable angle of view is set, selecting the shake detection area that is closer to the specified angle of view allows the optical zoom to be used up to the angle of view closer to the specified angle of view. For example, when the angle of view 2003C is specified, the shake detectable angle of view is set to 2004C so as to include the shake detection area of ​​2002A, while when the angle of view 2003D is specified, the shake detectable angle of view is set to 2004D so as to include the shake detection area of ​​2002B. This reduces the limitations on the range of use of the optical zoom.

[0066] The processing of this embodiment will be described mainly with respect to different processing, while omitting similar processing, using the flowchart of Fig. 6. When zooming of the camera is designated, the pan-tilt-zoom control unit 1008 proceeds to processing of S2000.

[0067] In step S2001, the pan / tilt / zoom control unit 1008 extracts a shake detection region using the same method as described above, and in this example, two regions 2002A and 2002B shown in FIG.

[0068] Thereafter, steps S2002 and S2003 are carried out in the same manner as described above.

[0069] In step S2004, the pan-tilt-zoom control unit 1008 calculates the shake detectable angle of view, compares the two extracted areas 2002A and 2002B with the specified angle of view, and selects the closest one as the shake detection area.

[0070] For example, let's say the angle of view at the wide-angle end is 40 degrees horizontally and 30 degrees vertically, with the center at 0 degrees and expressed as coordinates of -20 degrees to +20 degrees horizontally and -15 degrees to +15 degrees vertically. Within that, one shake detection area is -17 degrees to -7 degrees horizontally and 8 degrees to 9.5 degrees vertically, and another shake detection area is 10 degrees to 15 degrees horizontally and 8 degrees to 9.5 degrees vertically. Let's also say that PTZ operation is instructed to pan +10 degrees, tilt +2 degrees, and zoom with a horizontal angle of view of 16 degrees. When the angle of view during PTZ operation is expressed on coordinates, it is in the range of +2 degrees to +18 degrees horizontally and -4 degrees to +8 degrees vertically, which does not include the shake detection areas.

[0071] Here, the center of gravity coordinates of the specified angle of view are Horizontal (18-2) / 2+2=10 degrees Vertical (8-(-4)) / 2-4=2 degrees The coordinates of the center of gravity of the shake detection area 2002A are Horizontal (-7-(-17)) / 2-17=-12 degrees Vertical (9.5-8) / 2+8=8.75 degrees The coordinates of the center of gravity of another shake detection area 2002B are: Horizontal (15-10) / 2+10=12.5 degrees Vertical (9.5-8) / 2+8=8.75 degrees The distance between the center of gravity of the specified angle of view and the center of gravity of the shake detection area 2002A is

[0072]

number

[0073] On the other hand, the distance between the center of gravity of the specified angle of view and the center of gravity of the shake detection area 2002B is

[0074]

number

[0075] The distance to the determined designated angle of view is compared, and since the designated angle of view is closer to the shake detection area 2002B, the shake detection area 2002B is selected.

[0076] The shake detectable angle of view is calculated for the designated angle of view in the same manner as described above so as to include the selected shake detection area 2002B. The subsequent processing is the same as that described above.

[0077] As described above, in image stabilization control using image stabilization detection based on motion vectors, multiple regions in a captured image where motion vectors can be detected are extracted. Then, one of the multiple extracted regions closest to the specified image angle is selected based on the specified image angle. Then, an optical zoom value is determined such that the selected extracted region does not deviate from the image angle. By controlling the zoom using this optical zoom value, image angle control that takes image stabilization performance into account at any image angle can be performed, and limitations on the range of use of the optical zoom can be reduced.

[0078] (Fourth embodiment) Next, a fourth embodiment of the present invention will be described, mainly with reference to Fig. 10, with respect to a configuration in which shake detection and pan / tilt / zoom are performed using separate lens barrels. By using the same reference numerals as in the first embodiment for components similar to those in the first embodiment, detailed descriptions of those components will be omitted, and differences from the first embodiment will be mainly described.

[0079] 10 is a functional block diagram of a network camera according to this embodiment. A network camera 1000 is connected to a client device (information processing device) (not shown) via a network 3000 so as to be able to communicate with each other.

[0080] The configuration and functions of each part of the network camera 1000 will be described with reference to FIG.

[0081] The network camera 1000 includes a first lens barrel unit denoted by 1010 and a second lens barrel unit denoted by 2003. The first lens barrel unit 1010 includes a first imaging unit 1001, a first image processing unit 1002, a shake detection unit 1004, a zoom driving unit 1005, a pan driving unit 1006, a tilt driving unit 1007, and a pan / tilt / zoom control unit 1008, as in the first embodiment. The second lens barrel unit 4003 includes a second imaging unit 4001 and a second image processing unit 4002. In addition to the first lens barrel unit 1010 and the second lens barrel unit 4003, the network camera 1000 includes a system control unit 1003 and a communication unit 1009.

[0082] As described in the first embodiment, the first lens barrel unit is capable of changing the angle of view by pan-tilt-zoom and performing anti-shake control by the pan-tilt driver. However, in this embodiment, the first image processor 1002 does not perform image shake detection processing using a motion vector.

[0083] The second imaging unit 4001 is composed of a lens, an imaging element, and a control group thereof, and performs imaging of a subject and conversion into an electrical signal. The second imaging unit also has a lens capable of capturing a wide-angle image. The second image processing unit 4002 performs a predetermined development process and compression encoding process on the signal captured and photoelectrically converted by the second imaging unit 4001, generates image data, and transmits it to the system control unit 1003. The second image processing unit 4002 also performs a process of analyzing the image captured by the second imaging unit 4001 and detecting image shake using a motion vector. Information on the detected shake is transmitted to the pan-tilt-zoom control unit 1008 and used for vibration control, which will be described later.

[0084] The system control unit 1003 distributes the image data generated by the first image processing unit 1002 and the image data generated by the second image processing unit 4002 to the client device via the communication unit 1009.

[0085] Furthermore, the communication unit 1009 receives a camera control command for the first lens barrel unit 1010 and a camera control command for the second lens barrel unit 4003 transmitted from the client device, and transmits them to the system control unit 1003. It also transmits a response to the camera control command to the client device. The system control unit 1003 analyzes the transmitted camera control command and performs processing according to the command. For example, it instructs the first image processing unit 1002 or the second image processing unit 4002 to set image quality. The first image processing unit 1002 and the second image processing unit 4002 perform image processing based on the transmitted instruction.

[0086] In this way, the imaging device in this embodiment has two lens barrels: a first lens barrel that can change the angle of view by pan-tilt zoom and can perform vibration isolation control by a pan-tilt driver, and a second lens barrel that can capture a wide angle different from the first lens barrel. Then, a shake detection process is performed using a motion vector for a wide-angle captured image captured by the second lens barrel, and vibration isolation control is performed by the first lens barrel using the detected shake. This makes it possible to eliminate limitations on the range of use of optical zoom.

[0087] The processing of this embodiment will be described mainly with respect to different processing, while omitting similar processing, using the flowchart of Fig. 4. When the camera starts shooting, the pan-tilt-zoom control unit 1008 proceeds to processing of S1000.

[0088] Steps S1001 to S1002 are performed in the same manner as described above.

[0089] In step S1003, the second image processing unit 4002 obtains a motion vector in the same manner as described above. The pan / tilt / zoom control unit 1008 acquires the motion vector obtained by the second image processing unit 4002.

[0090] Steps S1004 to S1006 are performed in the same manner as described above.

[0091] In this way, the pan / tilt drive unit is controlled so as to cancel the shake detected in each of the pan and tilt directions, thereby reducing the shake.

[0092] As described above, a shake detection process based on a motion vector is performed on a wide-angle captured image captured by the second lens barrel. Then, the shake detected by the second lens barrel is used to perform vibration reduction control in the first lens barrel. This makes it possible to maintain vibration reduction performance at any angle of view and eliminate limitations on the range of use of the optical zoom.

[0093] In this embodiment, the imaging device has two lens barrels, a first lens barrel capable of changing the angle of view by pan-tilt-zoom and performing anti-shake control by a pan-tilt driver, and a second lens barrel separate from the first lens barrel, but the configuration is not limited to this. For example, the second imaging device may always perform shake detection processing based on a motion vector for a wide-angle captured image, transmit the detected shake to the first imaging device, and perform anti-shake control in the first imaging device using the transmitted shake.

[0094] (Other embodiments) A program (software) that realizes all or part of the control in the present invention and the functions of the above-mentioned embodiments may be supplied to an imaging device or an information processing device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the imaging device or the information processing device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

[0095] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

Claims

1. an imaging means for capturing an image of a subject via an imaging optical system; a zoom driving means for changing an angle of view of the imaging optical system; a determination means for determining an area in which a motion vector is to be detected in the image captured by the imaging means, based on the image captured by the imaging means; a vibration detection unit that detects a vibration amount of an image captured by the imaging unit based on a motion vector detected from the area determined by the determination unit; a shake correction unit that corrects the shake of an image captured by the imaging unit based on the amount of shake detected by the shake detection unit; a control means for controlling the zoom driving means so that the area determined by the determination means does not deviate from the angle of view of the imaging optical system; An imaging device comprising:

2. 2. The imaging device according to claim 1, a pan driving means for driving at least the imaging optical system in a pan direction; a tilt driving means for driving at least the imaging optical system in a tilt direction; and an imaging device characterized in that the shake correction means corrects shake of an image captured by the imaging means by driving the imaging optical system using at least one of the pan driving means and the tilt driving means in accordance with the amount of shake detected by the shake detection means.

3. 3. The imaging device according to claim 1, the imaging optical system has a shake correction optical system, The image pickup apparatus according to claim 1, wherein the image pickup means is configured to pick up an image captured by the image pickup means by controlling the drive of the image pickup means and the image pickup means in response to the amount of image shake detected by the image pickup means.

4. 4. The imaging device according to claim 1, an image capturing apparatus, characterized in that the image blur correction means corrects blur in the image captured by the image capturing means by cutting out a range from the image captured by the image capturing means according to the amount of shake detected by the shake detection means.

5. 5. The imaging device according to claim 1, an imaging device characterized in that the control means controls the imaging means to perform digital zoom so that when the zoom driving means performs optical zoom, the area determined by the determination means does not deviate from the angle of view of the imaging optical system.

6. 6. The imaging device according to claim 1, the control means controls the zoom drive means based on a difference between an amount of shake of an image captured by the imaging means detected by the shake detection means and an amount of correction performed by the shake correction means.

7. 7. The imaging device according to claim 1, The imaging device further includes a receiving unit for receiving a designation of an imaging range, The control means controls the zoom drive means so that an area among the plurality of areas determined by the determination means that is close to the imaging range specified by the reception means is included in the angle of view of the imaging optical system.

8. 1. A method for controlling an imaging device, comprising: an imaging step of capturing an image of a subject via an imaging optical system; a determining step of determining an area in which a motion vector is to be detected in the image captured by the imaging step, based on the image captured by the imaging step; a detection step of detecting an amount of shake of an image captured by the imaging step based on a motion vector detected from the area determined by the determination step; a correction step of correcting a shake of an image captured by the imaging step based on the amount of shake detected by the shake detection step; a control step of controlling a zoom driving means for changing an angle of view of the imaging optical system so that the area determined in the determination step does not deviate from the angle of view of the imaging optical system; The method according to claim 1, further comprising:

9. A program for causing a computer to execute a method for controlling an imaging device, the method comprising: an imaging step of capturing an image of a subject via an imaging optical system; a determining step of determining an area in which a motion vector is to be detected in the image captured by the imaging step, based on the image captured by the imaging step; a detection step of detecting an amount of shake of an image captured by the imaging step based on a motion vector detected from the area determined by the determination step; a correction step of correcting a shake of an image captured by the imaging step based on the amount of shake detected by the shake detection step; a control step of controlling a zoom driving means for changing an angle of view of the imaging optical system so that the area determined in the determination step does not deviate from the angle of view of the imaging optical system; A program comprising:

10. A computer-readable storage medium storing the program according to claim 9.

Citation Information

Patent Citations

  • Image blur detector and electronic camera

    JP2007019743A

  • Method of correcting distortion of taken image and device of correcting distortion of taken image, and imaging apparatus

    JP2007215114A

  • Moving picture processing apparatus, moving picture photographing apparatus and moving picture photographing program

    JP2008048152A

  • Imaging device and control method therefor

    JP2013009039A

  • Image shake correction device, imaging device, control method, and program

    JP2017106947A