IMAGE STABILIZATION DEVICE AND CONTROL METHOD THERE, IMAGE ACCTIMATOR AND STORAGE MEDIUM

The image stabilization device addresses distortion issues from ultra-wide-angle lenses by weighting motion vectors based on perspective size, enhancing stabilization accuracy and performance.

DE102018101035B4Inactive Publication Date: 2024-12-05CANON KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
DE102018101035
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2017-01-20
Filing Date
2018-01-18
Publication Date
2024-12-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Conventional image stabilization methods fail to correct distortion caused by ultra-wide-angle lenses, leading to inaccurate motion vectors and unstable image stabilization performance.

Method used

An image stabilization device that calculates and adjusts image blur correction amounts based on perspective size, using motion vectors weighted according to perspective changes, integrating lens and camera shake detection to enhance stabilization accuracy.

Benefits of technology

Stabilizes images effectively even with ultra-wide-angle lenses by reducing the influence of perspective distortion, improving the accuracy and stability of image stabilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Image stabilization device with a procurement facility for obtaining information about a lens, a motion vector detection device for detecting a motion vector from a multitude of images captured by an image capture device for capturing an object image and a calculating device for calculating an image blur correction amount for correcting image blur based on the motion vector detected by the motion vector detection device and a wobble of an image recording device detected by a wobble detection device for detecting the wobble of the image recording device, wherein the calculating device for changing weights with which the motion vector and the wobble of the image-taking device are acted upon is set up to calculate the amount of image blur correction based on information obtained by the procurement device regarding a perspective of the lens, and wherein The information regarding the perspective of the lens includes a perspective determination value calculated from a ratio of a first image acquisition distance and a second image acquisition distance, or from a difference in the number of image elements between the first and second image acquisition distances, wherein the first and second image acquisition distances are each obtained by projecting a first angle and a second angle onto an image acquisition plane, wherein the first and second angles are each obtained by uniformly dividing an image angle calculated from a focal length of the lens and a size of the image acquisition plane, wherein the first angle is closer to an optical axis of the lens than the second angle.
Need to check novelty before this filing date? Find Prior Art

Description

BACKGROUND OF THE INVENTION Area of ​​the invention

[0001] The present invention relates to methods for correcting blurriness in an image captured by an image-taking device. Description of the related technique

[0002] There are known methods for detecting, for example, camera shake and correcting the resulting image blur. Among these methods is optical image stabilization, which involves controlling an optical lens in response to detected shake, thereby eliminating the shake and correcting the image blur. Another method is electronic image stabilization, which involves extracting a section from successive images captured by an image-taking device and correcting the resulting blur through a projection transformation in response to detected shake.There is also an image capture device image stabilization, which is a more recent method of mechanically moving an image capture device in response to a detected shake in such a way that the shake is eliminated, thereby correcting image blur.

[0003] Methods for detecting wobble include a method for detecting wobble of an image-taking device itself using an angular velocity sensor or an accelerometer, and a motion vector method for calculating and predicting wobble of an image-taking device using a motion vector calculated from successive images taken by an image-taking device.

[0004] It is now known that lens-induced distortion aberration distorts the periphery of an image and also affects a motion vector captured from the image, thereby degrading the accuracy of the motion vector. Japanese patent disclosure JP 2012-103741 A, for example, discloses a method for efficiently implementing a variety of different image correction processes to avoid the effects of distortion.

[0005] Recently, ultra-wide-angle lenses have been increasingly used, primarily as interchangeable lenses, which create a large perspective that cannot be corrected by the aforementioned distortion correction method. Such a large perspective significantly impacts motion vector capture and is also known to affect the stability and performance of image stabilization.

[0006] Some ultra-wide-angle lenses that can be attached to interchangeable-lens camera systems are beginning to cover a wider field of view than ever before. Ultra-wide-angle lenses can also be used with fixed-lens cameras. When such a lens is used, perspective distortion occurs depending on the position where incident light forms an image. However, the conventional method disclosed in Japanese patent publication JP 2012-103741 A cannot correct this perspective-induced distortion, resulting in a deterioration in the accuracy of a motion vector. This leads to a problem where image stabilization becomes unstable.

[0007] US 2014 / 0354834A1 discloses an optical device capable of reducing eccentric aberration and simultaneously reducing image blur. A first image stabilization unit optically corrects image blur based on the amount of camera shake detected by a camera shake detection unit. A second image stabilization unit also optically corrects image blur based on the amount of camera shake detected by the camera shake detection unit. The second image stabilization unit has a different image blur correction effect than the first image stabilization unit. The optical device includes a mode in which image stabilization is performed using the first image stabilization unit without using the second image stabilization unit.

[0008] US 2013 / 0163084A1 discloses an image stabilization device comprising an image stabilization unit that compensates for image blur by moving a compensation element, a first shake detection unit that detects an angular velocity of the shake, a second shake detection unit that detects shakes by another method, an orientation detection unit that detects a tilt state of an optical device, a computing unit that calculates a compensation value from signals of the first and second shake detection units, and an output unit that compensates the output of the first shake detection unit using the compensation value, wherein the computing unit performs a weighting of the compensation value of a drive axis of an image stabilization mechanism based on a correlation between the first signal and the second signal.

[0009] US 2016 / 0196639 A1 discloses a device for stabilizing an image and comprises: an image analysis unit that analyzes the properties of each block to calculate a weight for each block; a significant block extraction unit that extracts from the blocks a block with a weight greater than a threshold as a significant block; a feature point extraction unit that extracts a feature point from the significant block by extracting pixels of the significant block and comparing color and / or brightness between the pixels and neighboring pixels; a local motion vector calculation unit that calculates local motion vectors of blocks corresponding to the feature point by referencing a previous frame of the frame and estimating a motion of the at least one block;and a global motion vector processing unit that calculates a global motion vector based on the calculated at least one local motion vector. SUMMARY OF THE INVENTION

[0010] The present invention addresses the aforementioned problem. An image stabilization device is provided which can reduce the deterioration in the accuracy of image stabilization even when using a lens that causes a large perspective distortion.

[0011] According to the invention, image stabilization devices, an image acquisition device, a method for controlling an image stabilization device and computer-readable storage media as specified in the accompanying patent claims are provided.

[0012] Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 shows a block diagram of a configuration of an image acquisition system that is common to embodiments of the invention. Fig. Figure 2 shows a representation of a configuration of a first embodiment of a section for performing image stabilization. Fig. Figure 3 shows a representation describing a procedure for calculating a perspective size. Fig. Figure 4 shows a block diagram of a configuration of a signal processing section according to the first embodiment. Fig. Figure 5 shows a flowchart of a process in an image stabilization process according to the first embodiment. Fig. Figure 6 shows a flowchart of a perspective determination process. The Fig. Figures 7A to 7F show representations of a variance of motion vectors and a perspective determination value. Fig. Figure 8 shows a representation of a process for generating a weighting histogram according to a second embodiment. Fig. Figure 9 shows a representation of a relationship between an image acquisition device plane, a motion vector area, and a perspective size. The Fig. 10A and Fig. Figure 10B shows illustrations describing a procedure for eliminating the influence of a perspective through weighting. Fig. Figure 11 shows a block diagram of a configuration of a signal processing section according to the second embodiment. Fig. Figure 12 shows a block diagram of a configuration of a signal processing section according to the second embodiment. Fig. Figure 13 shows a flowchart of the process of generating a weighting histogram. Fig. Figure 14 shows a representation of a relationship between an image acquisition device plane and a perspective size. The Fig. Figures 15A to 15C show representations of a relationship between a motion vector detection direction and a perspective size. DESCRIPTION OF THE EXAMPLES OF EXECUTION

[0013] Exemplary embodiments of the invention are described in more detail below with reference to the accompanying drawings. First, the features common to the exemplary embodiments are described.

[0014] Fig. Figure 1 shows a block diagram of a configuration of an image acquisition system 100 common to the embodiments of the invention. The image acquisition system 100 is a digital camera with interchangeable lenses, primarily for capturing still and moving images. The invention is not limited to digital cameras and can be applied to various image acquisition systems. For example, the invention is not limited to digital cameras with interchangeable lenses and is also applicable to digital cameras with fixed lenses.

[0015] The image acquisition system 100 from Fig. 1 comprises a lens assembly 150 and a camera body 160. The lens assembly 150 is attached to the camera body 160 during use. The lens assembly 150 includes a zoom unit 101 with a zoom lens that can change the magnification. A zoom control unit 102 controls the zoom unit 101. An aperture unit 103 functions as an aperture. An aperture control unit 104 controls the aperture unit 103. An image stabilization unit 105 includes an image stabilization lens, such as a displacement lens, which moves, for example, in a direction different from the optical axis. The image stabilization unit 105, which constitutes a first image stabilization device, is actuated and controlled by an optical image stabilization control unit 106. A focusing unit 107 includes a focusing lens that is focused to produce an image of the object.A focus control unit 108 controls the focus unit 107.

[0016] A lens control unit 109 is used by a user to operate the lens device 150. A lens shake detection unit 110 detects the extent of any shake of the lens device 150 and outputs a detection signal to a lens system control unit 111. The lens system control unit 111, which contains a central processing unit (CPU), controls the entire lens device 150, that is, it performs central control of the drive units and correction control units of the lens device 150. The lens system control unit 111 communicates with a camera system control unit 124 of the camera main body 160 via a lens communication control unit 112.

[0017] Next, the camera main body 160 is described. The camera main body 160 contains a shutter unit 113. A shutter control unit 114 controls the shutter unit 113. An image acquisition unit 115, which contains an image acquisition device, performs a photoelectric conversion of an object image generated by lenses and outputs an electrical signal. An image acquisition signal processing unit 116 performs a process of converting the electrical signal output from the image acquisition device 115 into a video signal. A video signal processing unit 117 processes the video signal output from the image acquisition signal processing unit 116. The process changes depending on the intended use of the video signal. For example, the video signal processing unit 117 changes the extraction position of the video signal depending on the correction level of an electronic image stabilization control unit 123.The electronic image stabilization control unit 123, which constitutes a second image stabilization device, controls image stabilization by image extraction. The second image stabilization is not limited to electronic image stabilization and can, for example, be image stabilization achieved by controlling a mechanical movement of the image acquisition device or image stabilization achieved by driving and controlling a movable optical element in the camera main body.

[0018] A display unit 118 displays an image as required, based on a signal output from the video signal processing unit 117. A storage unit 119 stores various data, such as video information. A power supply unit 120 supplies power to the entire camera system as needed. A camera control unit 121, used by the user to operate the camera system, outputs an operating signal to the camera system control unit 124. A camera shake detection unit 122 detects the extent of camera shake and outputs a detection signal to the camera system control unit 124. The camera system control unit 124, which contains a CPU, performs central control of the entire camera system. The camera system control unit 124 communicates with the lens communication control unit 112 of the lens device 150 via a camera communication control unit 125.The lens communication control unit 122 and the camera communication control unit 125 communicate with each other, in particular, when the lens device 150 is attached to the camera main body 160 and electrically connected to it.

[0019] The operation of the image acquisition system configured in this way will be outlined next. The lens control unit 109 and the camera control unit 121 each contain an image stabilization switch that can be used to turn image stabilization on and off. When the user operates the image stabilization switch to turn on image stabilization, the lens system control unit 111 or the camera system control unit 124 instructs the optical image stabilization control unit 106 or the electronic image stabilization control unit 123 to perform an image stabilization operation. Until an instruction to turn off image stabilization is issued, the image stabilization control unit thus activated controls the image stabilization.

[0020] The camera control unit 121 includes an image stabilization mode switch used to select a first or second image stabilization mode. In the first mode, only optical image stabilization (first image stabilization) is performed. In the second mode, optical and electronic image stabilization (second image stabilization) are used in conjunction for image stabilization. When the first mode is selected, the readout position of the image acquisition unit 115 is fixed, and the readout area is increased, thus allowing the angular size of a captured image to be enlarged. When the second mode is selected, more significant image blur can be addressed by changing the extraction position depending on the degree of image blur correction, although the video signal extraction area of ​​the video signal processing unit 117 is reduced.

[0021] The camera control unit 121 includes a shutter release button configured to activate a first switch SW1 and a second switch SW2 in that order, depending on how far the button is pressed. When the user presses the shutter release button halfway, the first switch SW1 is activated. When the user presses the shutter release button fully, the second switch SW2 is activated. When the first switch SW1 is activated, the focus control unit 108 controls the focus unit 107 to adjust the focus, and the aperture control unit 104 controls the aperture unit 103 to adjust the exposure to a suitable value. When the second switch SW2 is activated, image data is stored in the storage unit 119, which is obtained by the image acquisition unit 115 performing a photoelectric conversion of an object image.

[0022] The camera control unit 121 also includes a motion recording switch. The camera main body 160 starts recording a motion picture after the motion recording switch is pressed and stops recording if the motion recording switch is pressed again by the user during recording. If the user operates the shutter release button to turn on the first switch SW1 and the second switch SW2 while recording a motion picture, a process of acquiring and recording a still image is performed during the motion picture recording. The camera control unit 121 also includes a playback mode selector switch, which is used to select a playback mode. When the playback mode is selected by operating the playback mode selector switch, the camera stops any image stabilization process. <Erstes Ausführungsbeispiel>

[0023] A first embodiment of the invention will now be described with reference to Fig. 2 describes a process of changing the gain for image stabilization using a motion vector based on perspective information. Here, "perspective" refers to the phenomenon that an object at a shorter physical distance appears larger, and an object at a greater physical distance appears smaller, depending on the focal length of the lens. Fig. 2 are, incidentally, sections of Fig. 1, which are not involved in image stabilization, are not shown. Furthermore, the zoom unit 101, the aperture unit 103, the image stabilization unit 105 and the focusing unit 101 are not shown. Fig. 1 shown in total as optical lens 201.

[0024] The focal length of the optical lens 201 is changed by the zoom control unit 102. A perspective determination value based on the changed focal length information, or the focal length information itself, is sent by the lens communication control unit 112 to the camera communication control unit 125. The perspective determination value or focal length information received by the camera communication control unit 125 is transmitted to a perspective determination processing unit 211. Meanwhile, light passing through the optical lens 201 is imaged onto the image acquisition device 205, and the resulting digital image data obtained by photoelectric conversion is sent to an image processing unit 206. The digital image data sent to the image processing unit 206 is separated into a main image to be recorded and a signal to be sent to a motion vector detection unit 208.A motion vector is captured based on two temporally successive frames of digital image data (from a multitude of image frames) that are sent to the motion vector capture unit 208. A global vector computation unit 209 calculates a global vector from a multitude of captured motion vectors. The calculated global vector is converted by a feedback computation unit 210 into a correction magnitude that enables image stabilization, and the correction magnitude is transmitted to the perspective determination processing unit 211. If the perspective determination processing unit 211 determines that strong perspective is present, it modifies and reduces an image blur correction magnitude based on the motion vector.

[0025] Meanwhile, camera shake detected by the camera shake detection unit 122 is converted into a shake correction value by a high-pass filter (HPF) 213 and a low-pass filter (LPF) 214. The shake correction value obtained by this conversion is then added to the image blur correction value based on the motion vector obtained via the perspective determination processing unit 211. The resulting correction value is then transferred to a projection transformation parameter generation unit 215, which converts the correction value into a projection transformation parameter. Finally, the main image (moving image), separated by the image processing unit 206, undergoes a projection transformation performed by a projection transformation unit using the projection transformation parameter, thus correcting any image blur.

[0026] Now, a method for calculating a perspective size with reference to Fig. 3 described. Fig. Figure 3 shows a representation of a relationship between an image angle (θ1 + θ2 + θ3) within which an image can be captured by an image recording device, an object position Q and a position q at which an object is mapped onto an image recording plane. Fig. Figure 3 also shows angles (θ1-θ3) obtained by uniformly subdividing the field of view for perspective calculation, and the number of image elements (y1-y3) on the image acquisition plane corresponding to the respective angles. A calculation method for estimating a perspective using these values ​​is now described. The relationships between θ1, θ2, and θ3, obtained by uniformly subdividing a field of view within which an image can be captured, and the corresponding number of image elements y1, y2, and y3 on an image acquisition device are shown as follows: y1=f×tan(θ1) y2=f×tan(θ2) y3=f×tan(θ3)

[0027] Next, expressions for using these values ​​to calculate ratios P(1-2), P(1-3) and P(2-3) of image heights of perspective sizes and relationships PD(1-2), PD(1-3) and PD(2-3) are shown, which represent exact image element difference sizes between image heights due to perspective: P(1−2)=y1 / y2 P(1−3)=y1 / y3 P(2−3)=y2 / y3 PD(1−2)=|y1−y2| PD(1−3)=|y1−y3| PD(2−3)=|y2−y3|

[0028] Since the value of P(1-2), P(1-3), or P(2-3) is further away from one, the perspective size between image heights is determined to be larger. The exact image element difference sizes between image heights can be calculated as PD(1-2), PD(1-3), and PD(2-3), and their effects can be determined. The effect of a perspective size is specifically pre-calculated and then taken into account when image stabilization control or vector weighting is performed.

[0029] Signal processing according to this embodiment is described with reference to Fig. 4. The signal processing sequence includes signal processing for the detection of a camera shake signal from a signal of the camera shake detection unit 122 of the camera main body 160 and a process for detecting a motion vector from an image captured by the image acquisition device 205, using the motion vector detection unit 208. The camera communication control unit 125 is also provided, which receives perspective information from the lens device 150. Camera shake information obtained from the camera shake detection unit 122, which contains a gyroscope, an accelerometer, etc., is integrated after low-frequency and high-frequency components have been removed by the HPF 213 and the LPF 214, respectively. Subsequently, only the camera shake frequency band to be corrected is converted into an image blur correction value.

[0030] From successive frames of digital image data acquired by the image acquisition device 205, the motion vector acquisition unit 208 acquires a motion vector (local motion vector) in each of a plurality of subdivided areas of an image. A plurality of acquired motion vectors (local motion vectors) and the perspective information acquired by the lens device 150 are used by a perspective weighting unit 408 contained in the global vector calculation unit 209 to assign weights to the motion vectors depending on a perspective size. Using the weighted motion vectors, a histogram is generated. From the generated motion vector histogram, the global vector calculation unit 209 acquires a peak value and calculates a global vector in an image.The global vector is then subjected to filter processing by the feedback calculation unit 210 and added to the aforementioned image blur correction amount, which is calculated based on the signal from the camera shake detection unit 122. The resulting image blur correction amount is converted into a pixel value by a pixel conversion unit 411, and a contact termination process is performed by a final processing unit 412. Finally, the image blur correction amount is set in an electronic image blur correction adjustment unit 413, and image stabilization is performed.

[0031] Fig. Figure 5 shows a flowchart of an image stabilization process according to this embodiment. During the start of the image stabilization process, a shake magnitude detected by the camera shake detection unit 122 is filtered to calculate a camera shake magnitude in step S501. In step S502, perspective information is then obtained for the lens device 150. In step S503, the motion vector detection unit 208 then detects a multitude of motion vectors from successive images captured by the image acquisition device 205. In step S504, the perspective weighting unit 408 applies weights to the motion vectors according to the perspective information and generates a motion vector histogram in which the axes represent the magnitude of a motion vector and the number of detected motion vectors.

[0032] In step S505, the global vector processing unit 209 searches for a peak value in the motion vector histogram generated in step S504 and sets a found peak value as a global vector in an image. In step S506, the feedback processing unit 210 performs filtering on the calculated global vector and calculates a motion vector-based blur correction value. In step S507, the perspective determination processing unit 211 multiplies the motion vector-based blur correction value by a gain that depends on a perspective size. In step S508, the camera shake value calculated from the signal of the camera shake detection unit 122 is combined with the blur correction value calculated from the motion vectors. In step S509, the pixel conversion unit 411 converts the combined blur correction value into a pixel value for the image acquisition unit 205.In step S510, the final processing unit trims the image blur correction amplitude after pixel conversion to an upper limit below which correction is permitted. In step S511, the electronic image blur correction amplitude setting unit 413 finally sets the trimmed image blur correction amplitude for an electronic stabilization process, and electronic image stabilization is performed. If it was determined in step S512 that image stabilization is enabled, this process is repeated, and image stabilization is performed continuously for a moving image.

[0033] Fig. Figure 6 shows a flowchart of a process in a perspective determination process at step S507 in Fig. 5. In the perspective determination process, a variance value θ (vec) of the recorded motion vectors is calculated at the beginning in step S601 using the expressions (10) and (11). Vec(average)=1n∑i=1nVec(i) θ(vec)=1n∑i=1n(Vec(i)−Vec(average))2

[0034] Next, a gain for motion vector-based image stabilization is calculated based on the variance value θ(vec) of the motion vectors. In step S602, the feedback processing unit 210 calculates a gain for a motion vector-based blur correction level based on the value of a perspective parameter estimated from information obtained by the lens device 150. In step S603, the gain calculated from the variance value θ(vec) of the motion vectors is multiplied by a gain corresponding to the perspective parameter to determine a final gain for the motion vector-based blur correction level.

[0035] The Fig. Figures 7A to 7F show illustrations describing the procedure for calculating a gain for a perspective-based blur correction measure based on the perspective and the procedure for calculating a gain for a blur correction measure based on the variance value of motion vectors, which are described above with reference to Fig. 6 were described. If the lens has a small range of change in perspective size, the motion vector histogram, in which the axes represent the size of a motion vector and the number of captured motion vectors, has low variance and exhibits a significant peak value class 701, as described in Fig. Figure 7A shows that if, on the other hand, the lens has a large range of perspective size changes, the variance range of motion vectors is greater, and a peak value class 702 is closer to other vector classes, as shown in Figure 7A. Fig. 7B is shown.

[0036] If the camera shakes, a phenomenon occurs in this situation where an area with a perspective size different from that of an area containing an object is shifted, and therefore a correct peak value class 703, indicated by a dotted line, is shifted to a hatched class 704, as described in Fig. This phenomenon is shown in Figure 7C. This phenomenon is particularly significant on an ultra-wide-angle lens, which has a large perspective size, affecting the stability of the image stabilization. This phenomenon has a particularly negative impact on the performance of motion vector-based image stabilization, such as the feedback of motion vectors to optical image stabilization, identification of an offset magnitude of the camera shake detection unit 122, etc.

[0037] In this embodiment, taking into account the above, an average value 705 of motion vectors and a variance value θ706 are calculated from their average value, as described in Fig. Figure 7D is shown. A gain for a motion vector-based correction magnitude is calculated based on the variance value θ, as shown in Figure 7D. Fig. Figure 7E shows how the influence of perspective is controlled. Furthermore, a gain for the motion vector-based correction magnitude is calculated based on a perspective determination value, which is derived from a value obtained by the lens device 150, as shown in Figure 7E. Fig. Figure 7F shows how the influence of perspective is controlled. By multiplying these two gains, the adverse influence of perspective is isolated with high accuracy, and the stability of the image stabilization performance is therefore improved. Furthermore, the adverse influence on the offset identification of the camera shake detection unit 122 is eliminated.

[0038] Even when a lens is used that produces a large perspective size, this embodiment adjusts the gain for motion-vector-based image stabilization depending on the perspective size. When the perspective size is large, the gain for motion-vector-based image stabilization is reduced. Consequently, stable image stabilization can be achieved even when using a lens that produces a large perspective size. <Zweites Ausführungsbeispiel>

[0039] A perspective weighting process according to a second embodiment of the invention is now described. The second embodiment differs from the first embodiment in the process of generating a perspective weighting histogram in step S504. Fig. 5. Fig. Figure 8 shows a representation of a process for generating a perspective weighting histogram according to the second embodiment.

[0040] Initially, in step S801, a weighting is performed in each motion vector detection area. A weight changes depending on a difference in the size of a perspective dimension, which is calculated from a value obtained by the lens device 150. Then, in step S802, a motion vector histogram is generated using weighted motion vectors, in which the axes represent the size of a motion vector and the number of detected motion vectors. The influence of a motion vector in an area with a large change in perspective dimension can be reduced.

[0041] Fig. Figure 9 shows a diagram describing the relative order of perspective magnitudes, the arrangement of motion vectors, and the weights applied to these vectors. In an area where light incident through the optical system of the object device 150 is imaged onto the image-capturing device 205, variations in perspective magnitude increase, particularly when moving from the optical center to the periphery. An area 901 with larger changes in perspective magnitude is surrounded by a solid line, and an area 902 with smaller changes in perspective magnitude is represented by a hatched ellipse. Motion vectors captured in motion vector capture frames 903 located in the area with larger changes in perspective magnitude are unstable for movement due to the large changes in perspective magnitude.In contrast, motion vectors captured in motion vector capture frames 904, which are provided in the area with small changes in perspective size, are stable for movement.

[0042] The Fig. 10A and Fig. Figure 10B shows diagrams describing a weighting as a function of changing the perspective size and its result. Fig. 10A, motion vectors are subjected to weights depending on a change in perspective size. Fig. Figure 10A shows an input / output function where the horizontal axis represents a change in perspective size and the vertical axis represents a weight α. The weight α takes on a continuous value from one (maximum) to zero and varies between zero and one depending on a change in perspective size. By reflecting this weighting in the number of captured motion vectors in a motion vector histogram, as in Fig. As shown in Figure 10B, while maintaining a peak value class 1002 unchanged, other classes are reduced; that is, class segments 1003, indicated by a dotted line, are removed from classes in an area with larger changes in perspective size. As a result, the influence of unstable motion vectors can be reduced in an area with larger changes in perspective size.

[0043] The aforementioned feature of this embodiment is not only effective in improving the stability of motion vector-based image stabilization, but also in providing feedback of a motion vector in Fig. 11 and identification of an offset of a wobble detection device in Fig. 12, which will now be described.

[0044] A in Fig. Figure 11 shows a signal processing flow that includes signal processing for the detection of a camera shake signal from a signal of the camera shake detection unit 122 of the camera main body 160 and a process for detecting a motion vector from an image captured by the image acquisition device 205 using the motion vector detection unit 208. Furthermore, the camera communication control unit 125 is provided, which receives perspective information from the lens device 250. Additionally, the optical image stabilization control unit 106 is provided, which corrects image blurring by optically actuating a displacement lens. Motion vector feedback for correcting low-frequency shake, which is difficult for a gyroscope, etc., to detect, is provided by feeding back a motion vector detected by a camera via the lens communication control unit 112.Camera shake information, obtained by the camera shake detection unit 112, which contains a gyroscope, an accelerometer, etc., is integrated after low-frequency and high-frequency components have been removed by the HPF 213 and the LPF 214, respectively. Only the camera shake frequency band to be corrected is then converted into an image blur correction value.

[0045] The motion vector acquisition unit 208 captures a multitude of motion vectors from successive frames of digital image data acquired by the image acquisition device 205. The captured motion vectors and the perspective information obtained from the lens device 150 are used by the perspective weighting unit 408, contained in the global vector calculation unit 209, to assign weights to the motion vectors depending on a perspective size. A histogram is generated using the weighted motion vectors. From the generated motion vector histogram, the global vector calculation unit 209 extracts a peak value and calculates a global vector in an image.The global vector is transmitted from the camera main body 160 to the optical image stabilization control unit 106 in the lens device 150, thereby enabling the application of the image stabilization effect of a motion vector, which is particularly effective in a low-frequency range, for optical image stabilization. The global vector is then subjected to filtering processing by the feedback calculation unit 210 and added to the aforementioned image blur correction amount, which is calculated based on a signal from the camera shake detection unit 122. The resulting image blur correction amount is converted into a pixel value by the pixel conversion unit 411, and a contact end process is performed by the final processing unit 412. Finally, the image blur correction amount is set in the electronic image blur correction adjustment unit 413, and image stabilization is performed.In this method, the perspective weighting process is also effective and contributes to an improvement in the stability of optical image stabilization.

[0046] A in Fig. The signal processing flow shown in Figure 12 includes signal processing for the detection of a camera shake signal from a signal of the camera shake detection unit 122 of the camera main body 160 and a process for detecting a motion vector from an image captured by the image acquisition device 205 using the motion vector detection unit 208. The camera communication control unit 125 is provided, which receives perspective information from the lens device 150. Furthermore, an offset identification processing unit 1514 is provided, which estimates global motion vectors sequentially to estimate an offset contained in the output of the camera shake detection unit 122 and to eliminate the offset of the camera shake detection unit 122.

[0047] The motion vector detection unit 208 detects a multitude of motion vectors from successive frames of digital image data acquired by the image acquisition device 205. The detected motion vectors and the perspective information obtained from the lens device 150 are used by the perspective weighting unit 408 to assign weights to the motion vectors depending on a perspective size. A histogram is generated using the weighted motion vectors. From the generated motion vector histogram, the global vector calculation unit 209 detects a peak value and calculates a global vector in an image. Camera shake information obtained from the camera shake detection unit 122, such as a gyroscope, is compared with global vectors in temporally sequential images, thus eliminating any offset for the gyroscope.The camera shake information, from which the offset has been removed, is integrated after the low-frequency and high-frequency components have been removed by the HPF 213 and LPF 214, respectively. Then, only the camera shake frequency band to be corrected is converted into an image blur correction value. The global vector is subjected to filtering by the feedback calculation unit 210 and added to the previously calculated image blur correction value. The resulting image blur correction value is converted into a pixel value by the pixel conversion unit 411, and a contact end process is performed by the final processing unit 412. Finally, the image blur correction value is set in the electronic image blur correction value setting unit 413, and image stabilization is performed.

[0048] In this method, the perspective weighting process is also effective and contributes to an improvement in the stability of the optical image stabilization.

[0049] Even when a lens is used that produces a large perspective size, according to this embodiment, motion vectors are subjected to weights that depend on the perspective size. Specifically, when the perspective size is large, the weight applied to a motion vector is reduced. Consequently, the influence of a motion vector detected in an area with a large perspective size can be reduced, and thus stable image stabilization performance can be achieved. <Drittes Ausführungsbeispiel>

[0050] A perspective weighting process according to a third embodiment of the invention is now described. The third embodiment differs from the first embodiment in the process of generating a perspective weighting histogram in step S504. Fig. 5. Fig. Figure 13 shows a representation of a process for generating a perspective weighting histogram according to the third embodiment.

[0051] Initially, a motion vector captured in each motion vector capture frame is assigned a weight if there is a significant difference in the change in perspective size obtained from a lens before and after a movement. The weight varies depending on the direction and position of the captured motion vector. The weighted motion vectors are used to generate a motion vector histogram, where the axes represent the size of a motion vector and the number of captured motion vectors. This process reduces the influence of unstable motion vectors output from areas where the change in perspective size within a motion vector capture frame is substantial.

[0052] Now, the relative order of the change variables of the perspective size in a motion vector capture frame, the arrangement of motion vectors, and weights applied to the motion vectors are considered with reference to Fig. 14 described. In an area where light incident through the optical system of the lens device 150 is imaged onto the image-taking device 205, the change in perspective size increases, particularly from the optical center to the periphery. An area 1201 where there are larger changes in perspective size is surrounded by a solid line, and an area 1202 where there are smaller changes in perspective size is represented by a hatched ellipse. Of the motion vectors captured in motion vector capture frames 1203 located in the area with larger changes in perspective size, one motion vector is unstable in a direction where the difference in perspective size before and after the movement is large.

[0053] Based on this, a weighting is determined depending on a change in perspective size within a recording framework, and its result is calculated with reference to the Fig. Described in sections 15A to 15C.

[0054] With reference to Fig. Section 15A describes a perspective weighting direction. A perspective magnitude increases from circle 1302 to circle 1303. Circles 1301 and 1302 are concentric circles whose center is an optical axis 1301. Therefore, there is a greater perspective change for a motion vector 1307 that is captured in the direction of a straight line 1304 extending radially from the optical axis. In contrast, there is a smaller perspective change for a motion vector 1308 that is captured in the tangential direction 1306 of the concentric circle. The perspective change can be calculated as the length and position (image acquisition distance) of a projection of a captured motion vector onto a straight line extending radially from the center, i.e., the optical axis. The perspective change thus calculated can be used in Fig. 15B can be used as input. Fig. In 15B, each motion vector is assigned a weight. The weight changes depending on a change in perspective size in areas before and after a movement within a capture frame. Fig.Figure 15B shows an input / output function form where the horizontal axis represents a change in perspective size before and after a movement in a capture frame, and the vertical axis represents a weight α. The weight α takes a continuous value from one (maximum) to zero and varies between zero and one depending on the change in perspective size. The weight α is set to zero for a movement vector that exhibits a large change in perspective size before and after a movement in a capture frame. Consequently, a class segment 1310 captured in an area where there is a large change in perspective size before and after a movement in a capture frame indicated by a dotted line is removed from a peak value class.As a result, it is possible to prevent the peak value class section from shifting to a class section 1311 (indicated by hatching) in an adjacent class. Therefore, the influence of an unstable motion vector can be reduced.

[0055] Even when a lens is used that produces a large perspective size, according to this embodiment, motion vectors are weighted as a function of changes in perspective size. Consequently, the influence of a motion vector calculated from a representative point (feature point) captured in an area with varying perspective sizes can be eliminated. Therefore, stable image stabilization performance can be achieved. Further examples of implementation

[0056] Embodiments of the invention can also be implemented by a computer of a system or device which reads and executes computer-executable instructions (for example, one or more programs) recorded on a storage medium (which can also be described as a 'non-volatile computer-readable storage medium') for carrying out the functions of the embodiments described above, and / or which contains one or more circuits (for example, an application-specific integrated circuit (ASIC)) for carrying out the functions of the embodiments described above.and by a method carried out by the computer of the system or device, for example by reading and executing the computer-executable instructions from the storage medium to perform the functions of the embodiments described above and / or controlling the circuit or circuits to perform the functions of the embodiments described above. The computer may have one or more processors (for example, a central processing unit, CPU), microprocessing unit,The MPU (Multi-Process Unit) may include a network of separate computers or separate processors for reading and executing the computer's executable instructions. The computer's executable instructions may be provided to the computer, for example, by a network or the storage medium. The storage medium may include, for example, a hard disk and / or random-access memory (RAM) and / or read-only memory (ROM), and / or distributed computing system memory and / or an optical disc (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™) and / or a flash memory device and / or a memory card, and the like.

[0057] Although the invention has been described with reference to exemplary embodiments, it is evident that the invention is not limited to the disclosed embodiments. The scope of protection of the following claims is to be interpreted in the broadest possible way to encompass corresponding modifications and equivalent structures and functions.

Claims

[1] Image stabilization device with a procurement facility for obtaining information about a lens, a motion vector detecting device for detecting a motion vector from a plurality of images recorded by an image recording device for recording an object image and a calculation device for calculating an image blur correction amount for correcting an image blur on the basis of the motion vector detected by the motion vector detection device and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculation device for changing weights applied to the motion vector and the shaking of the image pickup device is arranged to calculate the image blur correction amount on the basis of information acquired by the acquisition device regarding a perspective of the lens, and wherein the information regarding the perspective of the lens includes a perspective determination value calculated from a ratio of a first image pickup distance and a second image pickup distance or a difference in the number of picture elements between the first and second image pickup distances, wherein the first and second image pickup distances are each obtained by projecting a first angle and a second angle onto an image pickup plane, wherein the first and second angles are each obtained by equally dividing an image angle calculated from a focal length of the lens and a size of the image pickup plane, wherein the first angle is closer to an optical axis of the lens than the second angle. [2] An image stabilizing device according to claim 1, wherein when a change in perspective of the lens is greater than a predetermined value, the calculating means is arranged to reduce the weight applied to the motion vector to calculate the image blur correction amount. [3] An image stabilizing apparatus according to claim 1, further comprising image stabilizing means for correcting image blur based on the image blur correction amount calculated by said calculating means. [4] Image stabilization device with a procurement facility for obtaining information about a lens, a motion vector detecting device for detecting a motion vector from a plurality of images recorded by an image recording device for recording an object image and a calculation device for calculating an image blur correction amount for correcting an image blur on the basis of the motion vector detected by the motion vector detection device and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculation device for changing weights applied to the motion vector and the shake of the image pickup device is arranged to calculate the amount of image blur correction on the basis of information regarding a perspective of the lens acquired by the acquisition device, wherein the motion vector detecting means is arranged to divide an image captured by the image capturing means into a plurality of regions and calculate a single global vector as the motion vector using local motion vectors obtained in the respective divided regions, and wherein the calculation device is arranged to apply a weight to a local motion vector as a function of a size of a projection of the local motion vector onto a straight line extending radially from an optical axis as the center point. [5] An image stabilizing apparatus according to claim 4, wherein said motion vector detecting means is arranged to apply a weight depending on the perspective of the lens to each of the local motion vectors obtained in the respective divided areas to calculate the global vector. [6] An image stabilizing apparatus according to claim 5, wherein said motion vector detecting means is arranged to apply a weight to each of the local motion vectors obtained in the respective divided areas depending on the perspective of the lens, and thereafter generate a histogram of the local motion vectors obtained in the respective divided areas and calculate the global vector based on the histogram. [7] Image stabilization device according to claim 6, wherein the calculation device is arranged to change the weights applied to the motion vectors depending on variance values ​​of the motion vectors in the histogram. [8] An image stabilizing apparatus according to claim 4, wherein, in the calculation of the global vector by the motion vector detecting means, the weights applied to the local motion vectors obtained in the divided regions in a periphery of the image are smaller than the weights applied to the local motion vectors obtained in the divided regions near a center of the image. [9] Image recording device with the image stabilization device according to one of claims 1 to 8, and the image recording device for recording the object image. [10] Method for controlling an image stabilization device, with obtaining information about a lens, Detecting a motion vector from a plurality of images captured by an image recording device for capturing an object image and Calculating an image blur correction amount for correcting image blur based on the motion vector detected in the motion vector detection and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculation includes changing weights applied to the motion vector and the shake of the image pickup device to calculate the image blur correction amount based on information regarding a perspective of the lens obtained during the acquisition, and wherein the information regarding the perspective of the lens includes a perspective determination value calculated from a ratio of a first image pickup distance and a second image pickup distance or a difference in the number of picture elements between the first and second image pickup distances, wherein the first and second image pickup distances are each obtained by projecting a first angle and a second angle onto an image pickup plane, wherein the first and second angles are each obtained by equally dividing an image angle calculated from a focal length of the lens and a size of the image pickup plane, wherein the first angle is closer to an optical axis of the lens than the second angle. [11] Method for controlling an image stabilization device, with obtaining information about a lens, Detecting a motion vector from a plurality of images captured by an image recording device for capturing an object image and Calculating an image blur correction amount for correcting image blur based on the motion vector detected in the motion vector detection and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculating includes changing weights applied to the motion vector and the shake of the image pickup device to calculate the image blur correction amount on the basis of information acquired in the acquisition regarding a perspective of the lens, wherein the detecting includes dividing an image captured by the image pickup device into a plurality of regions and calculating a single global vector as the motion vector using local motion vectors obtained in the respective divided regions, and wherein the calculating includes applying a weight to a local motion vector depending on a size of a projection of the local motion vector onto a straight line extending radially from an optical axis as a center point. [12] A computer-readable storage medium storing a program that causes a computer to perform steps of a method for controlling an image stabilization device, the method comprising Obtaining information about a lens, Detecting a motion vector from a plurality of images captured by an image recording device for capturing an object image and Calculating an image blur correction amount for correcting image blur based on the motion vector detected in the motion vector detection and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculating includes changing weights applied to the motion vector and the shake of the image pickup device to calculate the image blur correction amount based on information regarding a perspective of the lens acquired in the obtaining, and wherein the information regarding the perspective of the lens includes a perspective determination value calculated from a ratio of a first image pickup distance and a second image pickup distance or a difference in the number of picture elements between the first and second image pickup distances, wherein the first and second image pickup distances are each obtained by projecting a first angle and a second angle onto an image pickup plane, wherein the first and second angles are each obtained by equally dividing an image angle calculated from a focal length of the lens and a size of the image pickup plane, wherein the first angle is closer to an optical axis of the lens than the second angle. [13] A computer-readable storage medium storing a program that causes a computer to perform steps of a method for controlling an image stabilization device, the method comprising Obtaining information about a lens, Detecting a motion vector from a plurality of images captured by an image recording device for capturing an object image and Calculating an image blur correction amount for correcting image blur based on the motion vector detected in the motion vector detection and a shake of an image pickup device detected by a shake detection device for detecting the shake of the image pickup device, wherein the calculating includes changing weights applied to the motion vector and the shake of the image pickup device to calculate the image blur correction amount based on information regarding a perspective of the lens acquired in the acquisition, wherein the detecting includes dividing an image captured by the image pickup device into a plurality of regions and calculating a single global vector as the motion vector using local motion vectors obtained in the respective divided regions, and wherein the calculating includes applying a weight to a local motion vector depending on a size of a projection of the local motion vector onto a straight line extending radially from an optical axis as a center point.

Citation Information

Patent Citations

  • Imaging apparatus, image processing apparatus, image processing method and program

    JP2012103741A

  • Image stabilization apparatus, control method thereof, optical apparatus and image capturing apparatus

    US20130163084A1

  • Optical device, control method therefor, and storage medium storing control program therefor

    US20140354834A1

  • Apparatus and method for image stabilization using image blur correction

    US20160196639A1