Image processing device, image processing method, program, and storage medium

Abstract

To provide an image processing apparatus that performs an appropriate image shake correction operation and subject tracking operation according to zoom magnification.SOLUTION: An image processing apparatus comprises: an electronic enlargement unit that enlarges and outputs a predetermined area from a picked-up image; a motion vector detection unit that detects motion vectors from a plurality of picked-up images; a first representative vector calculation unit that increases weight of a first motion vector detected in the predetermined area compared with weight of a second motion vector detected in an area other than the predetermined area to calculate a first representative motion vector; a second representative vector calculation unit that increases the weight of the second motion vector with respect to the weight of the first motion vector compared with that of the first representative vector calculation unit to calculate a second representative motion vector; and a third representative vector calculation unit that calculates a third representative motion vector on the basis of the first representative motion vector and the second representative motion vector. The third representative vector calculation unit changes a third representative vector value according to whether the magnification in the electronic enlargement unit is equal to or larger than the value of a threshold.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device having an image blur correction function. [Background technology]

[0002] There are two types of image stabilization methods for cameras: one is optical image stabilization, which moves a correction lens installed in part of the lens group in a direction different from the optical axis based on the shake applied to the camera, and the other is electronic image stabilization, which reduces positional deviation between image frames by changing the readout position of the image sensor based on the shake of the camera and displaying and recording the output image.

[0003] Methods for detecting camera shake include using sensors whose output changes in response to the shake, such as angular velocity sensors and acceleration sensors, and detecting motion vectors from the captured image and using these as shake information.

[0004] Cameras generally also have a zoom function, which can be achieved by either optical zoom, which uses a group of lenses that move along the optical axis, or electronic zoom, which changes the readout area (size) of the image sensor to enlarge or reduce the image. With electronic zoom, the number of pixels contained in the readout area decreases as the magnification increases, which is a drawback in that the resolution drops, but unlike optical zoom, high magnification zoom can be achieved without changing the lens size.

[0005] Here, this applies not only to electronic zoom but also to optical zoom, but as the zoom ratio increases, even a slight shake of the camera causes a large change in the shooting angle of view, making it difficult for the photographer to continue capturing the subject.

[0006] Many technologies have been proposed for cameras to automatically track subjects. For example, Patent Document 1 proposes a method for tracking subjects by setting a motion vector detection area so that a moving subject can be easily detected, and then making an optical image stabilization mechanism track the detected motion vector. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2017-098829 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the method disclosed in Patent Document 1, the motion vector detection area is set depending on the shooting mode. For example, in the panning shooting mode, the motion vector detection area for subject tracking is always set, and in the image stabilization mode, the motion vector detection area for image stabilization is always set. In other words, it is not assumed that the motion vector detection area will be dynamically switched in the same shooting mode.

[0009] Taking the aforementioned electronic zoom as an example, when the zoom magnification becomes high, it becomes difficult for the photographer to capture the subject, and in that case it is preferable for the camera to switch to automatically track the subject. On the other hand, when the zoom magnification is low, the photographer can capture the subject by operating the camera himself, so having the camera automatically track the subject would actually interfere with the photographer's intended camerawork. Therefore, in this case, it is preferable to detect the motion vector from the background area that does not include the main subject and perform image blur correction.

[0010] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an image processing device that can perform appropriate image blur correction operation and subject tracking operation according to the zoom magnification. [Means for solving the problem]

[0011] An image processing device according to the present invention includes electronic enlargement means for enlarging and outputting a predetermined area from a captured image, motion vector detection means for detecting motion vectors from a plurality of captured images, first representative vector calculation means for calculating a first representative motion vector by weighting a first motion vector detected in the predetermined area greater than a weighting a second motion vector detected in an area outside the predetermined area, second representative vector calculation means for calculating a second representative motion vector by weighting the second motion vector greater than that of the first representative vector calculation means, and and a third representative motion vector calculation means for calculating a third representative motion vector based on the first representative motion vector and the second representative motion vector, wherein when an enlargement ratio in the electronic enlargement means is equal to or greater than a threshold, the third representative motion vector calculation means determines the third representative motion vector by making a weight of the first representative motion vector greater than a weight of the second representative motion vector, and when an enlargement ratio in the electronic enlargement means is less than the threshold, the third representative motion vector calculation means determines the third representative motion vector by making a weight of the second representative motion vector greater than a weight of the first representative motion vector. Effect of the Invention

[0012] According to the present invention, it is possible to provide an image processing device that can perform an appropriate image blur correction operation and an appropriate subject tracking operation according to a zoom magnification. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the configuration of a video camera that is a first embodiment of an image processing device according to the present invention. [Diagram 2] 5A to 5C are views for explaining a method of calculating a representative motion vector at low zoom ratio in the first embodiment. [Diagram 3] 5A to 5C are diagrams for explaining a method of calculating a representative motion vector in high-magnification zoom according to the first embodiment. [Figure 4]5 is a flowchart illustrating an example of processing by a representative vector switching control unit in the first embodiment. [Diagram 5] 10 is a flowchart illustrating another example of the process of the representative vector switching control unit in the first embodiment. [Figure 6] 5 is a diagram for explaining a representative vector switching threshold value in the first embodiment. [Figure 7] FIG. 11 is a block diagram showing the configuration of a video camera according to a second embodiment. [Figure 8] FIG. 11 is a block diagram showing another configuration of the video camera according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] <First embodiment> Fig. 1 is a block diagram showing the configuration of a video camera which is a first embodiment of an image processing device according to the present invention. An example of the configuration and operation of the video camera 100 in Fig. 1 will be specifically described below. Note that, although this embodiment describes an example in which the present invention is applied to a video camera, the present invention is also applicable to a still camera having a video shooting function.

[0016] The imaging lens 101 is an imaging optical system composed of a group of lenses. The imaging element 102 is an element that converts an optical image into an electric signal, and is composed of a CCD, a CMOS element, or the like. The subject image formed by the imaging lens 101 is converted into an analog image signal by the imaging element 102, and this analog image signal is output as a captured image signal to a signal processing unit 103. The signal processing unit 103 converts the input captured image signal into a digital signal by A / D conversion, and then performs predetermined signal processing.

[0017] The signal processing unit 103 generates a video signal (image signal) conforming to, for example, the NTSC format from the signal obtained by the imaging element 102 , and supplies the generated signal to an image memory 104 and a motion vector detection unit 110 .

[0018] The image transformation unit 105 realizes the functions of electronic zoom and electronic image stabilization by changing the read range and read position (changing the angle of view) when reading out an image signal from the image memory 104. For example, the electronic zoom function is a function for displaying and recording an electronically enlarged image by reducing the range in which an image is read out from the image memory. The electronic image stabilization function is a function for correcting misalignment of images between frames by changing the read position of an image from the image memory 104 in a direction that cancels out the misalignment between images caused by camera shake. The video data output from the image transformation unit 105 is supplied to the recording control unit 106.

[0019] When an instruction to record a video signal is given by an operation unit (not shown) used to instruct the start and end of recording, the recording control unit 106 records the video signal supplied from the image transformation unit 105 onto a recording medium 107. The recording medium 107 is, for example, an information recording medium such as a semiconductor memory or a magnetic recording medium such as a hard disk.

[0020] The display control unit 108 causes the display device 109 to display the video signal supplied from the image transformation unit 105. The display control unit 108 drives the display device 109, and the display device 109 displays an image using a liquid crystal display element (LCD) or the like.

[0021] The motion vector detection unit 110 detects a motion vector of an image based on a luminance signal included in the current video signal generated by the signal processing unit 103 and a luminance signal included in the video signal of the previous field (or the previous frame) stored in the image memory 104. The motion vector detection method may be, for example, a block matching method that has been proposed in the past.

[0022] The block matching method is a method of dividing a captured image into regions called blocks, and detecting similar parts of the captured image, for example, one frame before and the current captured image, on a block-by-block basis. Within an arbitrary range in the captured image one frame before, the point with the highest correlation value with an arbitrary block in the current captured image is determined as the similar block position. The amount of displacement between the arbitrary block position in the current captured image and the similar block position in the captured image one frame before is calculated, and motion information between the captured image frames, i.e., the motion vector, is detected.

[0023] The first representative vector calculation unit 111 and the second representative vector calculation unit 112 calculate an overall motion vector from a plurality of block-unit motion vectors output from the motion vector detection unit 110. Here, the first representative vector calculation unit 111 acquires information on the electronic enlargement area from the electronic enlargement information acquisition unit 113, and calculates a representative motion vector (first representative motion vector) based on the motion vector detected in the electronic enlargement area. However, if a valid motion vector cannot be obtained in the electronic enlargement area, a motion vector outside the electronic enlargement area may be used. In addition, any method may be used as long as the motion vector within the electronic enlargement area is used preferentially, such as weighting the motion vector within the electronic enlargement area more heavily than the motion vector outside the electronic enlargement area. Note that in the present invention and this specification, calculating a third value by weighting one more heavily than the other includes calculating a third value using only one without using the other. In other words, calculation of a representative motion vector by the first representative vector calculation unit 111 based on a motion vector detected within the electronic enlargement area rather than based on a motion vector detected outside the electronic enlargement area is also considered to be included in calculating a representative motion vector by weighting the motion vector within the electronic enlargement area more heavily than the motion vector outside the electronic enlargement area.

[0024] The second representative vector calculation unit 112 calculates a representative motion vector from the motion vectors detected in the entire imaging area including outside the electronic enlargement area. Therefore, compared with the first representative vector calculation unit 111, the weight of the motion vector outside the electronic enlargement area is increased relative to the motion vector inside the electronic enlargement area to calculate a representative motion vector (second representative motion vector). For example, the first representative vector calculation unit 111 calculates a representative motion vector by weighting the motion vector inside the electronic enlargement area to 1 and the motion vector outside the electronic enlargement area to 0. On the other hand, the second representative vector calculation unit 112 calculates a representative motion vector by weighting the motion vector inside and outside the electronic enlargement area to 1. The second representative vector calculation unit 112 aims to calculate the motion vector of the background area. For this purpose, known techniques, such as a method of setting a high priority for the motion vector of the peripheral area assuming that the main subject is photographed near the center, a method of determining and excluding a moving subject, and a method of identifying a fixed subject by subject determination, may be used.

[0025] In addition, as a method for calculating the representative vector in this embodiment, for example, a histogram (frequency distribution) showing the frequency (frequency) of the magnitude (class) of motion vectors on a block-by-block basis is generated, and the motion vector with the highest frequency is determined as the representative vector.

[0026] The electronic enlargement information acquisition unit 113 acquires information related to electronic enlargement performed by the electronic zoom function, and outputs the information to each processing unit. The information related to electronic enlargement includes the electronic enlargement magnification, the horizontal and vertical resolutions of the electronic enlargement area, and the location to be electronically enlarged, i.e., address information of the electronic enlargement area on the image memory 104. The electronic enlargement magnification can also be calculated from the horizontal and vertical resolutions of the electronic enlargement area. Therefore, in the following description, the statement that the electronic enlargement magnification is acquired from the electronic enlargement information acquisition unit 113 also includes the case where the horizontal and vertical resolutions of the electronic enlargement area are acquired instead of the electronic enlargement magnification itself. In other words, the information to be acquired may be at least one of the electronic enlargement magnification or the horizontal and vertical resolutions of the electronic enlargement area.

[0027] The representative vector switching control unit 114 determines the representative vector to be input to the image deformation amount calculation unit 115 from among the representative vectors determined by the first representative vector calculation unit 111 and the second representative vector calculation unit 112. The details of the representative vector switching control in the representative vector switching control unit 114 will be described later, but the representative vector is generally performed as follows. That is, when a subject tracking function is realized by changing the read position from the image memory 104, the representative vector of the first representative vector calculation unit 111 is adopted, and when an electronic image stabilization function is realized, the representative vector of the second representative vector calculation unit 112 is adopted. This switching is controlled based on the electronic enlargement magnification obtained from the electronic enlargement information acquisition unit 113. When the electronic enlargement magnification is equal to or greater than a threshold value, the value of the first representative vector calculation unit 111 is output, and when the electronic enlargement magnification is less than the threshold value, the value of the second representative vector calculation unit 112 is output.

[0028] Image deformation amount calculation unit 115 determines the size of the area to be read from image memory 104 based on the electronic enlargement magnification obtained from electronic enlargement information acquisition unit 113, and also determines the position to read from image memory 104 based on the representative vector obtained from representative vector switching control unit 114. Image deformation unit 105 changes the read position and read area size of the image from image memory 104 based on the output from image deformation amount calculation unit 115. An electronic zoom function is realized by changing the read area size, and a subject tracking function and electronic image stabilization function are realized by changing the read position.

[0029] Control unit 120 is a control unit that controls the entire video camera 100, and is connected to each of the above-mentioned components of video camera 100. Control unit 120 controls each part of video camera 100 by executing a program stored in memory 121.

[0030] Next, the photographing angle of view, subject image, motion vector, and representative vector in the cases of low and high magnification of electronic zoom will be described with reference to FIGS.

[0031] The image shown in Fig. 2 is an image captured under conditions where the electronic zoom magnification is smaller than that of the image shown in Fig. 3. The representative vector switching control unit 114 determines whether the electronic enlargement magnification is equal to or greater than a threshold value or less than the threshold value, and it is assumed that the image in Fig. 2 has an electronic enlargement magnification less than the threshold value, and the image in Fig. 3 has an electronic enlargement magnification equal to or greater than the threshold value.

[0032] The image captured at time t0 is the image shown in Figure 2(A), where the frame line indicates the entire captured area and the dotted frame indicates the electronic enlargement area. Also, the image shown in Figure 2(B) is an enlarged view of the electronic enlargement area in Figure 2(A). Similarly, the image captured at time t1 is the image shown in Figure 2(C), and the image shown in Figure 2(D) is an enlarged view of the electronic enlargement area. Figure 2(E) shows motion vectors detected between images captured at times t0 and t1, and Figure 2(F) shows only the motion vectors of the electronic enlargement area.

[0033] FIG. 2(G) shows a histogram generated from the motion vectors in FIG. 2(E), and FIG. 2(H) shows a histogram generated from only the motion vectors in the electronically enlarged area in FIG. 2(F). The horizontal axis of the graph indicates the class, i.e., the magnitude of the motion vector, and the vertical axis indicates the frequency, i.e., the frequency of detection of the motion vector. As described above, the motion vector is detected by a block matching method, but the arrangement of each block is not shown here and will not be described. In addition, since the camera 100 is moving due to the photographer's adjustment of the composition, such as subject tracking, the position of the fixed subject in the image changes, and as a result, the value of motion vector group 2 does not become zero. For areas without a motion vector, it is assumed that the background has no features and the motion vector cannot be detected. In other words, the representative vector is selected from either motion vector group 1 or motion vector group 2.

[0034] In FIG. 2, the motion vectors detected based on the moving subject are designated as motion vector group 1, and the motion vectors detected based on the fixed subject (background) are designated as motion vector group 2. In the entire imaging area, the fixed subject is larger than the moving subject, and therefore, in FIG. 2(G), the detection frequency of motion vector group 2 is higher than the detection frequency of motion vector group 1. However, when only the electronic enlargement area is extracted, the moving subject occupies a larger area than the fixed subject, and therefore, in FIG. 2(H), the detection frequency of motion vector group 1 is higher than the detection frequency of motion vector group 2. Therefore, when the representative vector is determined based on the histogram shown in FIG. 2(G), the motion vector group 2 is selected, and when the representative vector is determined based on the histogram shown in FIG. 2(H), the motion vector group 1 is selected. As shown in FIG. 2(F) and FIG. 2(H), the process of extracting the motion vectors in the electronic enlargement area and calculating the most frequent motion vector as the representative vector is the process performed by the first representative vector calculation unit 111. 2(E) and 2(G), the process of extracting motion vectors of the entire imaging area and calculating the most frequently occurring motion vector as a representative vector is performed by the second representative vector calculation unit 112. The process performed by the second representative vector calculation unit 112 is a known representative vector calculation process that does not depend on electronic zoom.

[0035] As described above, for the image in Fig. 2, since the electronic magnification is less than the threshold, the histogram shown in Fig. 2(G) is adopted so that a representative vector is calculated from the motion vectors of the entire imaging area. That is, the motion vector calculated from the background area, such as the motion vector group 2, is supplied to the image deformation amount calculation unit 115 as the representative vector. As a result, the read position of the image from the image memory 104 is controlled so as to cancel the movement amount of the background area, and an electronic image stabilization function is realized.

[0036] Next, the calculation of the representative vector when the electronic zoom magnification is equal to or greater than the threshold value will be described with reference to FIG.

[0037] The image captured at time t0 is the image shown in Fig. 3(A), and the image in which only the electronically enlarged area is displayed is the image shown in Fig. 3(B). The image captured at time t1 is the image shown in Fig. 3(C), and the image in which only the electronically enlarged area is displayed is the image shown in Fig. 3(D). Since the video camera 100 has only one image transformation unit 105, it is not possible to perform multiple electronic enlargements on one image, but for ease of explanation, it is assumed that images with different electronic enlargement magnifications as shown in Figs. 2 and 3 can be obtained at time t0.

[0038] FIG. 3(E) shows the motion vectors of the entire imaging area calculated from the images of FIG. 3(A) and FIG. 3(C), and FIG. 3(F) shows only the motion vectors calculated from the electronically enlarged area. As in FIG. 2, the motion vectors based on the moving object are designated as motion vector group 1, and the motion vectors based on the fixed object are designated as motion vector group 2. FIG. 3(G) shows a histogram generated from the motion vectors of the entire imaging area, and FIG. 3(H) shows a histogram generated from the motion vectors of the electronically enlarged area. When the most frequently occurring motion vector is selected from the histogram of FIG. 3(G), motion vector group 2 is selected as the representative vector, and when the most frequently occurring motion vector is selected from the histogram of FIG. 3(H), motion vector group 1 is selected as the representative vector.

[0039] Since the electronic enlargement magnification of the image in Fig. 3 is equal to or greater than the threshold, the histogram shown in Fig. 3(H) is adopted so that a representative vector is calculated from the motion vectors in the enlarged area. That is, the motion vector calculated from the subject area, such as the motion vector group 1, is supplied to image deformation amount calculation unit 115 as a representative vector. This makes it possible to change the read position from image memory 104 so as to cancel the amount of movement of the subject area, thereby realizing a subject tracking function.

[0040] Next, the processing of the representative vector switching control unit 114 will be described with reference to the flowcharts of Figs. 4 and 5. First, simple switching control is shown in Fig. 4. The operations of the flowcharts shown in Figs. 4 and 5 are realized by the control unit 120 executing a program stored in the memory 121. The operations of the flowcharts shown in Figs. 4 and 5 are executed every time a motion vector detection process is executed. For example, in an NTSC imaging device, imaging processing is executed at intervals of 1 / 60 seconds, and motion vector detection processing and the flowcharts shown in Figs. 4 and 5 are executed in accordance with this. In other words, the processing cycle of the operations shown in Figs. 4 and 5 is 1 / 60 seconds.

[0041] In FIG. 4, in step S100, the control unit 120 acquires the currently set electronic enlargement magnification from the electronic enlargement information acquisition unit 113.

[0042] In step S101, the control unit 120 determines whether or not the electronic enlargement magnification is equal to or greater than a threshold value. If it is equal to or greater than the threshold value, the process proceeds to step S102, and if it is less than the threshold value, the process proceeds to step S103.

[0043] In step S 102 , the representative vector switching control unit 114 selects the vector value calculated by the first representative vector detection unit 111 , and outputs it to the image deformation amount calculation unit 115 .

[0044] In step S103, the representative vector switching control unit 114 selects the vector value calculated by the second representative vector detection unit 112 and outputs it to the image deformation amount calculation unit 115. This makes it possible to switch between subject tracking control and image blur correction control based on the magnitude of the electronic enlargement magnification. Note that the threshold value used in step S101 may be changed based on the magnitude of the optical magnification, the shooting mode, a user setting, or the like.

[0045] The above is a method for simply switching motion vectors based on electronic magnification, but this switching method has the following problem: When the electronic magnification approaches or exceeds a threshold value, the function that has been operating as electronic image stabilization suddenly changes to a subject tracking function, which may cause the angle of view to suddenly change in accordance with the movement of the subject, causing a sense of discomfort to the photographer.

[0046] In order to solve this problem, in Fig. 5, a response characteristic is imparted to the output switching by the representative vector switching control unit 114. As already explained, the process of the flowchart in Fig. 5 is also executed at a processing cycle linked to the imaging cycle.

[0047] In step S200, the control unit 120 acquires the currently set electronic enlargement magnification from the electronic enlargement information acquisition unit 113.

[0048] In step S201, the control unit 120 determines whether the electronic enlargement magnification has been changed from the value acquired in the previous process. If it has been changed, the process proceeds to step S202, and if it has not been changed, the process proceeds to step S203.

[0049] In step S202, the control unit 120 calculates the addition rate change amount α. The formula for calculating the addition rate change amount α can be expressed as follows, where α is the difference between the electronic enlargement magnification and the electronic enlargement magnification at the time of the previous process, that is, the electronic enlargement change amount multiplied by a predetermined coefficient β.

[0050] α = (electronic magnification ratio - previous electronic magnification ratio) × β The amount of change in the addition rate α is a value that determines the responsiveness of switching from the electronic image stabilization operation to the subject tracking operation, that is, how long it takes from when the representative vector switching control unit 114 starts changing the value output until it completely switches. The smaller the amount of change in the addition rate α, the longer it takes to switch the output. For example, when the electronic enlargement magnification is continuously changed by continuously pressing an operating member such as a zoom key, the amount of change in the electronic enlargement has a gradual continuity. Therefore, as in the above formula, by using this characteristic to calculate the amount of change in the addition rate α, it is possible to give a responsive characteristic to the output switching of the representative vector switching control unit 114. Conversely, when the electronic enlargement is switched ON / OFF every time the button is pressed, the amount of change in the electronic enlargement becomes a steeply large value, and the amount of change in the addition rate α also becomes a steeply large value. As a result, the output of the representative vector switching control unit 114 is also switched immediately.

[0051] The coefficient β in the above formula is a parameter for determining how quickly to switch from the electronic image stabilization operation to the subject tracking operation in accordance with the electronic zoom operation, and is set to an arbitrary value determined in advance in consideration of operability. The smaller the coefficient β, the longer the time required to complete the switch.

[0052] In step S203, the control unit 120 sets the addition rate change amount α to use the value from the previous process. This is processing for continuing switching control until switching of the output of the representative vector switching control unit 114 is completed, even if the electronic zoom operation is interrupted immediately after the electronic enlargement magnification exceeds the threshold, for example.

[0053] In step S204, the control unit 120 compares the electronic enlargement magnification with a threshold value, and if the electronic enlargement magnification is equal to or greater than the threshold value, the process proceeds to step S205, and if the electronic enlargement magnification is less than the threshold value, the process proceeds to step S206.

[0054] In step S205, the control unit 120 determines whether the addition rate k is less than 1, and if it is less than 1, executes step S206. The addition rate k is a value obtained by adding or subtracting the addition rate change amount α each time the process of FIG. 5 is repeated, and the range of values ​​that the addition rate k can take is 0≦k≦1. In step S206, the control unit 120 sets the new addition rate k to the value obtained by adding the addition rate k of the previous process to the addition rate change amount α. The determination process of step S205 is performed so as not to set the addition rate k to a value greater than 1.

[0055] In step S207, the control unit 120 determines whether the addition rate k is greater than 0, and if it is greater than 0, executes step S208. In step S208, a value obtained by subtracting the addition rate change amount α from the addition rate k of the previous process is set as a new addition rate k. Note that the determination process in step S207 is performed so as not to set k to a value smaller than 0.

[0056] In step S209, the control unit 120 calculates the average value of the output value of the first representative vector detection unit 111 and the output value of the second representative vector detection unit 112 according to the value of the addition ratio k, and sets the average value as the output value of the representative vector output control unit 114. This can be expressed by the following equation.

[0057] Output value=(output of the first representative vector detection unit 111)×k+ (output of the second representative vector detection unit 112)×(1−k) For example, when the electronic magnification is changed from a small state to a large state and exceeds a threshold, the output value is switched from the output value of the second representative vector calculation unit 112 to the output value of the first representative vector calculation unit 111 according to the electronic zoom speed. In other words, the ratio at which the output value of the first representative vector calculation unit 111 is used gradually increases by adding the addition ratio change amount α to the value of the addition ratio k for each cycle of FIG. 5, and finally, it is switched to the output value of the first representative vector calculation unit 111. Even if the electronic zoom operation is stopped during the switching, the switching control is continued until the output value of the first representative vector calculation unit 111 is completely switched to by the process of step S203. When the switching is completed, the value of k becomes 1, and k is limited to 1 as an upper limit by the judgment of step S205.

[0058] Next, the control unit 120 backs up the current electronic enlargement magnification in step S210, backs up the current addition rate change amount α in step S211, and backs up the current addition rate k in step S212, in preparation for the next process.

[0059] The above process reduces the abrupt change in the angle of view caused by switching between electronic image stabilization and object tracking functions when the electronic magnification exceeds a threshold value due to electronic zoom operation. Also, when the electronic magnification function can be switched ON / OFF, the above switching control can be completed instantly, and switching between electronic image stabilization and object tracking can be performed without delay.

[0060] Next, the threshold value of the electronic magnification ratio at which the electronic image stabilization operation and the subject tracking operation are switched over will be explained using Fig. 6. The higher the electronic magnification ratio, the greater the change in the angle of view caused by the photographer's hand shake, making it more difficult to capture the subject within the screen. This will be explained using Fig. 6.

[0061] As shown in Fig. 6(A), the horizontal width (width in the horizontal direction) of the electronic magnification area is defined as X. Fig. 6(B) is a diagram showing the relationship between the lens and the image sensor. The horizontal width X converted into a physical quantity on the image sensor is defined as d, and the focal length is defined as f. Both f and d have the same unit, for example, millimeters. Moreover, the value converted into an angle of the electronic magnification area d is defined as deg(X). Here, deg(X) can be expressed by the formula arctan(d / f).

[0062] Fig. 6(C) is a diagram showing the relationship between the camera shake angle θ when the photographer is shooting from a fixed point and the above-mentioned deg(X). Fixed point shooting here refers to shooting performed by holding the camera in one's hand without using a tripod or the like. When shooting a fixed subject, this is a method of shooting by holding the camera firmly so as not to change the shooting angle of view as much as possible, and is a shooting method that minimizes camera shake when shooting handheld.

[0063] As the electronic magnification increases, deg(X) decreases. This is because, as shown in Figure 6(B), as the electronic magnification area becomes smaller, the value of arctan(d / f) in the above formula decreases. Here, the electronic magnification at which deg(X) is smaller than θ is defined as MagTh. If the horizontal width of the entire imaging area is Xorg, MagTh can be expressed as Xorg / X. When the electronic magnification reaches MagTh, the photographer's own camera shake will cause a change in the angle of view that is greater than the horizontal width of the electronic magnification area. In other words, even if you intend to take a picture without moving the camera, for example, a subject that was on the right edge of the screen will move to the left edge of the screen due to camera shake.

[0064] When the angle of view movement caused by camera shake becomes large, it becomes difficult for the photographer to capture the subject within the screen. Even if the effects of camera shake can be avoided by the image stabilization function of the camera, in a situation where the screen moves by the horizontal width of the electronic enlargement area due to camera shake, it is still difficult for the photographer to keep the subject within the shooting screen by operating the camera himself. On the other hand, in the case of a small enlargement magnification range where the photographer can track the subject by operating the camera himself, it is preferable that the camera does not hinder the photographer's operation as much as possible. Therefore, when the electronic enlargement magnification is less than the threshold, the camera stops tracking the subject and performs electronic image stabilization to correct only the image shake that is unnecessary for the photographer. On the other hand, when the electronic enlargement magnification exceeds MagTh, the function is switched to subject tracking instead of image stabilization, making it easier for the photographer to capture the subject.

[0065] The camera shake angle θ during fixed-point shooting by the photographer may be determined by measuring the camera shake of the photographer in the past, and may be determined by a predetermined camera shake amount.

[0066] In addition, in the above description, the horizontal width of the electronic magnification area has been described, but the same applies to the vertical width.

[0067] As described above, in the first embodiment of the present invention, the motion vector used to change the read position from the captured image is selected from the motion vector calculated from the electronic enlargement area and the motion vector calculated from the entire imaging area according to the electronic enlargement magnification. This makes it possible to track the subject using the camera function when the electronic enlargement magnification becomes such that it is difficult for the photographer to track the subject by operating the camera himself, enabling good photography.

[0068] <Second embodiment> Next, a second embodiment of the present invention will be described. The video camera of the first embodiment only has an electronic image stabilization function. Video camera 200 of this embodiment has an optical image stabilization function in addition to the electronic image stabilization function. As described above, electronic image stabilization is a process that depends on the imaging cycle, but optical image stabilization is performed in a shorter cycle than electronic image stabilization, so it is possible to reduce high-frequency image blur components that could not be corrected by electronic image stabilization.

[0069] 7 is a block diagram showing the configuration of a video camera 200 which is a second embodiment of an image processing device of the present invention. In the explanation of video camera 200, blocks similar to those of video camera 100 explained in the first embodiment are given the same reference numerals and explanations thereof will be omitted.

[0070] In this embodiment, the shake applied to the video camera 200 is detected by the shake detection sensor 201. The shake detection sensor 201 is, for example, an angular velocity sensor that detects the shake of the video camera 200 as an angular velocity. If the shake detection sensor 201 is an analog element whose output voltage changes according to the amount of shake, the signal is converted into digital data via an A / D converter and then output to the shake detection sensor reference value calculation unit 202 and the subtractor 203.

[0071] The shake detection sensor reference value calculation unit 202 calculates a reference value of the shake detection sensor 201. The reference value of the shake detection sensor 201 is the output value of the shake detection sensor 201 when no shake occurs in the video camera 200. The method of calculating the reference value will be described in detail later.

[0072] The subtractor 203 subtracts the reference value calculated by the shake detection sensor reference value calculation unit 202 from the output of the shake detection sensor 201. By this process, the value output from the subtractor 203 becomes highly accurate shake information from which unnecessary reference value components have been removed.

[0073] Lens driving amount calculation section 204 calculates a drive target position of blur correction lens 216 based on the shake amount of video camera 200 output from subtractor 203. The processing by lens driving amount calculation section 204 includes, for example, an integration process for calculating an angle from an angular velocity, a correction process for the drive target position of blur correction lens 216 according to the focal length and the subject distance, and a process for converting the corrected drive target position into a unit of position information output from A / D converter 213.

[0074] A subtractor 205 calculates the difference between the output of the lens driving amount calculation unit 204 and the output of an A / D converter 213 that digitizes the output of the position detection sensor 212. Here, the output of the subtractor 205 is referred to as a deviation amount. A control filter 206 performs signal processing such as amplification and phase compensation on the deviation amount. A pulse width modulation unit 207 modulates the output of the control filter 206 into a waveform (i.e., a PWM waveform) that changes the duty ratio of a pulse wave and outputs the modulated output. A motor drive unit 208 is a circuit that applies a drive signal to a motor 209. Specifically, for example, the motor drive unit 208 is an H-bridge circuit, and the motor 209 is a voice coil type motor. A voltage is applied from the H-bridge circuit to the terminals of the voice coil type motor by the output of the pulse width modulation unit 207, and the amount of current flowing through the coil changes, thereby generating a drive force that moves the blur correction lens 216. It is shown that terminal A210 and terminal A214 are electrically connected, and similarly, terminal B211 and terminal B215 are electrically connected.

[0075] The position detection sensor 212 is composed of a magnet and a Hall sensor provided opposite it, and when the motion compensation lens 216 moves, the magnetic flux acting on the Hall sensor changes, causing a change in the output voltage. A differential amplifier circuit that amplifies the output of the Hall sensor may also be used. In this way, the position detection sensor 212 includes the circuit provided before the A / D converter 213.

[0076] The output of the position detection sensor 212 is supplied to the subtractor 205 and the shake detection sensor reference value calculation unit 202 via an A / D converter 213. The imaging lens 217 is an imaging optical system composed of a plurality of lenses including a motion compensation lens 216. The motion compensation lens 216 is, for example, a shift lens, and is an optical system that deflects the optical axis by moving on a plane perpendicular to the optical axis. In this way, when the current position has not reached the target position of the motion compensation lens 216, the amount of the shortfall is calculated as a deviation amount, and by driving the motion compensation lens 216, a feedback control system can be configured that constantly corrects the deviation amount.

[0077] In the second representative vector calculation unit 218, the process of calculating the representative vector is the same as that in the first embodiment, but the shake detection sensor reference value calculation unit 202 is added to the output target. Therefore, a different reference number is given to distinguish it from the second representative vector calculation unit 112.

[0078] The following describes the processing executed by the shake detection sensor reference value calculation unit 202. Note that since calculation of the reference value of the shake detection sensor is a known technique, only an outline will be described.

[0079] The output signal of the shake detection sensor 201 varies depending on environmental conditions such as the outside temperature when the applied vibration is zero. That is, the output of the shake detection sensor 201 is a signal in which the output fluctuation when the vibration is zero is further superimposed on the signal representing the shake amount of the video camera 200. Therefore, the shake detection sensor reference value calculation unit 202 calculates the output when the vibration applied to the shake detection sensor 201 is zero, that is, the reference value, and subtracts this from the original signal. This makes it possible to calculate an accurate shake amount. In this embodiment, a method using a motion vector is adopted as a method for calculating the reference value. If the output when the vibration is zero fluctuates, this reference value fluctuation is superimposed on the shake detection sensor 201. Therefore, even if the image blur correction is performed using the output signal of the shake detection sensor 201 as it is, accurate image blur correction cannot be performed, and a correction residue occurs. Since this correction residue appears as a blur amount in the image as it is, the motion vector amount can be considered to be equivalent to the correction residue amount.

[0080] Based on this idea, it is possible to calculate a reference value by subtracting (the angle converted value of the position of the blur correction lens 216 + the remaining shake angle) from the angle converted value of the output of the shake detection sensor 201. This remaining shake angle is the angle converted value of the motion vector.

[0081] Here, in order to realize the above method, the motion vector used for calculating the reference value needs to be calculated based on the shake of the video camera 200. For this purpose, the representative vector based on the background region calculated by the second representative vector calculation section 218 is always input to the shake detection sensor reference value calculation section 202, regardless of the electronic enlargement region.

[0082] As described above, in the second embodiment of the present invention, the output of representative vector switching control unit 114 that changes depending on the electronic enlargement magnification is supplied to image deformation amount calculation unit 115, and the motion vector based on the background area calculated by the second representative vector calculation unit is supplied to the reference value calculation. This makes it possible to realize good image shooting using optical image stabilization while maintaining the control of switching between electronic image stabilization and subject tracking depending on the electronic enlargement magnification as shown in the first embodiment.

[0083] In the above embodiment, a method has been described in which object tracking is performed by changing the image read position when the electronic magnification ratio is equal to or greater than a threshold value. However, this object tracking may also be performed using the blur correction lens 216. For example, a value obtained by performing a filter operation such as a predetermined phase compensation or amplification process on the representative vector is added to the drive amount calculated by the lens drive amount calculation unit 204. This allows the blur correction lens 216 to correct even low-frequency components. By applying this, it is also possible to perform object tracking while correcting high-frequency blur with the blur correction lens 216.

[0084] 8 is obtained by changing the inputs and outputs of representative vector switching control section 301 and lens drive amount calculation section 303 from video camera 200, and further adding vector control filter 302. Vector control filter 302 is a circuit that performs filter calculation processing such as phase compensation described above. When the electronic enlargement magnification is equal to or greater than a threshold, as described above, a representative vector based on the movement of the subject is output from representative vector switching control section 301. This representative vector is subjected to phase compensation and the like through vector control filter 302, and is input to lens drive amount calculation section 303.

[0085] In the lens driving amount calculation unit 303, the subject movement amount output from the vector control filter 302 is superimposed on the driving amount of the blur correction lens 216 calculated based on the shake amount detected by the shake detection sensor 201. As a result, the blur correction lens 216 performs image blur correction and subject tracking, and the subject image input to the image sensor 102 is an image in which the deviation of the main subject position has been corrected. In this case, the value detected by the motion vector detection unit 110 is the amount that the blur correction lens 216 could not track, that is, the remaining tracking amount, but by executing the processing as described above, the remaining tracking amount is electronically corrected in the image transformation unit 105 as well. From the above, by using a configuration like the video camera 300, optical subject tracking by the lens and electronic subject tracking by changing the read position are used together. Therefore, it is possible to track even a larger subject movement than the video camera 100 or the video camera 200.

[0086] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0087] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100: video camera, 105: image transformation section, 110: motion vector detection section, 111: first representative vector calculation section, 112: second representative vector calculation section, 113: electronic enlargement information acquisition section, 114: representative vector switching control section, 115: image transformation amount calculation section, 120: control section, 121: memory

Claims

1. an electronic enlargement means for enlarging and outputting a predetermined area of ​​a captured image; A motion vector detection means for detecting a motion vector from a plurality of captured images; a first representative vector calculation means for calculating a first representative motion vector by making a weight of a first motion vector detected in the predetermined region larger than a weight of a second motion vector detected in a region outside the predetermined region; a second representative vector calculation means for calculating a second representative motion vector by weighting the second motion vector with respect to the first motion vector more heavily than the first representative vector calculation means; a third representative motion vector calculation means for calculating a third representative motion vector based on the first representative motion vector and the second representative motion vector, The third representative vector calculation means when the enlargement factor in the electronic enlargement means is equal to or greater than a threshold value, a weight of the first representative motion vector is made greater than a weight of the second representative motion vector to determine the third representative motion vector; an electronic enlargement unit that determines a third representative motion vector by making a weight of the second representative motion vector greater than a weight of the first representative motion vector when the enlargement magnification of the electronic enlargement unit is less than the threshold value;

2. the first representative vector calculation means calculates the first motion vector and outputs the first motion vector as the first representative motion vector; 2. The image processing apparatus according to claim 1, wherein said second representative vector calculation means calculates said second motion vector and outputs said second motion vector as said second representative motion vector.

3. 3. The image processing device according to claim 1, wherein the third representative vector calculation means determines the first representative motion vector as the third representative motion vector when the enlargement magnification in the electronic enlargement means is equal to or greater than a threshold value, and determines the second representative motion vector as the third representative motion vector when the enlargement magnification in the electronic enlargement means is less than the threshold value.

4. 3. The image processing apparatus according to claim 1, wherein the third representative vector calculation means outputs a value obtained by averaging the first representative motion vector and the second representative motion vector at a predetermined ratio.

5. 5. The image processing device according to claim 4, wherein, when the enlargement ratio in the electronic enlargement means is equal to or greater than the threshold value, the third representative vector calculation means changes the addition ratio of the first representative motion vector so as to gradually increase and the addition ratio of the second representative motion vector so as to gradually decrease, and outputs an average value.

6. 6. The image processing apparatus according to claim 5, wherein said third representative vector calculation means determines the amount of change in said addition rate in accordance with the amount of change in the enlargement magnification in said electronic enlargement means.

7. 7. The image processing apparatus according to claim 6, wherein said third representative vector calculation means increases the amount of change in said addition rate as the amount of change in the enlargement magnification in said electronic enlargement means increases.

8. 8. An image processing device according to claim 1, further comprising a reference value calculation means for calculating a reference value which is an output of a shake detection sensor, the output of which changes depending on an amount of shake, when no shake is occurring, and the reference value calculation means calculates the reference value based on the output of the second representative vector calculation means, regardless of an enlargement magnification of the electronic enlargement means.

9. 9. The image processing apparatus according to claim 8, further comprising a control unit for controlling movement of a blur correction lens, which is an optical system that is provided in an imaging optical system and deflects an optical axis, based on an output of the shake detection sensor.

10. 10. The image processing apparatus according to claim 9, wherein the control means further moves the motion compensation lens based on an output of the third representative vector calculation means.

11. an electronic enlargement means for enlarging and outputting a predetermined area of ​​a captured image; A motion vector detection means for detecting a motion vector using a plurality of captured images; a control unit that controls an image blur correction based on the motion vector detected by the motion vector detection unit, wherein the control means controls the image stabilization so as to reduce an amount of movement of a subject within the specified area in the image when the enlargement magnification of the electronic enlargement means is equal to or greater than a threshold, and controls the image stabilization so as to reduce an amount of movement of a background in the image when the enlargement magnification is less than the threshold.

12. 12. The image processing device according to claim 11, wherein, when the enlargement ratio is equal to or greater than the threshold value, the control means controls the image blur correction by making a weight of a first motion vector detected in the specified region greater than a weight of a second motion vector detected in a region other than the specified region.

13. 13. The image processing device according to claim 12, wherein the control means controls the image blur correction so as not to weight the first motion vector greater than the weight of the second motion vector when the enlargement ratio is less than the threshold value.

14. having electronic correction means; 14. The image processing apparatus according to claim 11, wherein the control unit controls the electronic correction unit to thereby control the image blur correction.

15. 14. The image processing device according to claim 11, wherein the control unit controls the image blur correction by controlling an optical correction unit included in a photographing optical system.

16. an electronic enlargement step of enlarging and outputting a predetermined area of ​​the captured image; a motion vector detection step of detecting a motion vector from a plurality of captured images; a first representative vector calculation step of calculating a first representative motion vector by making a weight of a first motion vector detected in the predetermined region larger than a weight of a second motion vector detected in a region outside the predetermined region; a second representative vector calculation step of calculating a second representative motion vector by making a weight of the second motion vector greater than a weight of the first motion vector; a third representative motion vector calculation step of calculating a third representative motion vector based on the first representative motion vector and the second representative motion vector, In the third representative vector calculation step, When the enlargement factor in the electronic enlargement step is equal to or greater than a threshold value, a weight of the first representative motion vector is made greater than a weight of the second representative motion vector to determine the third representative motion vector; an electronic enlargement process for determining a third representative motion vector by making a weight of the second representative motion vector greater than a weight of the first representative motion vector when the enlargement ratio in the electronic enlargement process is less than the threshold value;

17. an electronic enlargement step of enlarging and outputting a predetermined area of ​​the captured image; a motion vector detection step of detecting a motion vector using a plurality of captured images; a control step of controlling an image blur correction based on the motion vector detected in the motion vector detection step, the control step controls the image stabilization so as to reduce an amount of movement of a subject within the specified area in the image when the enlargement magnification in the electronic enlargement step is equal to or greater than a threshold, and controls the image stabilization so as to reduce an amount of movement of a background in the image when the enlargement magnification in the electronic enlargement step is less than the threshold.

18. A program for causing a computer to execute the image processing method according to claim 16 or 17.

19. 18. A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 16.

Images