Image processing device, image processing method, and program

Abstract

To achieve the improvement of performance of detecting a motion vector in addition to the reduction of calculation costs.SOLUTION: An image processing device (104) includes: acquisition means (104) for acquiring a plurality of images having parallax caused by a plurality of optical systems (101, 131); extraction means (121) for extracting feature points from the images; reflection means (121) for reflecting the feature point extracted from the first image among the plurality of images by adding the parallax to the feature point extracted from the second image; and detection means (121) for detecting motion vectors on the basis of the feature points. The detection means (121) performs the detection of the motion vector for the second image on the basis of the feature point reflecting the feature point of the first image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing technique for detecting motion in an image. [Background technology]

[0002] Conventionally, in order to provide a realistic video experience, a stereo camera capable of capturing images for stereoscopic display has been provided. When detecting a motion vector using image data captured by a stereo camera, since the image data contains images of almost the same subject, it is inefficient to detect the motion vector in the entire image data. Patent Document 1 discloses a technology that can solve the problem by devising a template matching method. In template matching, one of two frame images in an image is used as an original image and the other is used as a reference image. Then, a rectangular area arranged on the original image is used as a template block, and a rectangular area arranged on the reference image is used as a search area, and a correlation with the distribution of pixel values ​​in the template block is obtained at each position of the search area. Here, when a stereo camera captures a left eye image and a right eye image, there is an overlapping area between the search area in the left eye image and the search area in the right eye image. In the technology of Patent Document 1, template matching is performed on only one of the left eye image or the right eye image for the overlapping search area, thereby reducing the computation cost. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-165909 A Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, in the method of Patent Document 1, in order to reduce the calculation cost, template matching is performed on only one of the left eye image or the right eye image for the search area overlapping between the left eye image and the right eye image. However, depending on the camera movement caused by hand shake or the like, the image in the template block may move in different directions in the left eye image and the right eye image. In this case, template matching cannot be performed correctly, and as a result, the motion vector cannot be detected correctly, and the detection performance of the motion vector is reduced.

[0005] Therefore, an object of the present invention is to enable improvement in the detection performance of motion vectors in addition to reducing the calculation cost. [Means for solving the problem]

[0006] The image processing device of the present invention comprises an acquisition means for acquiring a plurality of images having parallax due to a plurality of optical systems, an extraction means for extracting feature points from the images, a reflection means for reflecting the feature points extracted from a first image of the plurality of images on feature points extracted from a second image while taking into account the parallax, and a detection means for detecting a motion vector based on the feature points, wherein the detection means detects the motion vector for the second image based on the feature points into which the feature points of the first image are reflected. Effect of the Invention

[0007] According to the present invention, it is possible to reduce the calculation cost and also improve the detection performance of the motion vector. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of a digital camera according to an embodiment. [Diagram 2] FIG. 4 is a diagram illustrating an example of the configuration of a motion vector detection unit. [Diagram 3] 1A and 1B are explanatory diagrams of a grid arrangement, feature points, and a template matching area. [Figure 4]13 is a flowchart showing a process performed by a motion vector detection unit. [Diagram 5] FIG. 4 illustrates an example of the configuration of a feature point extraction unit. [Figure 6] 13 is a diagram showing the relationship between pixel values ​​and judgments in an accuracy judgment unit. FIG. [Figure 7] 13 is a flowchart showing a process of a tracking target feature point determining unit. [Figure 8] 11A and 11B are diagrams illustrating an example of motion vector distribution when camera shake occurs in a rotational direction; [Figure 9] FIG. 1 is a diagram showing examples of two image circles. [Figure 10] FIG. 13 is a diagram showing an example of motion vector distribution of two image circle images. [Figure 11] FIG. 11 is a diagram used to explain feature point reflection processing. [Figure 12] 13 is a flowchart of a feature point reflection process. [Figure 13] 13 is a flowchart of a feature point number setting process. [Figure 14] FIG. 13 is a diagram showing an example of feature point distribution when the number of feature points is appropriate. [Figure 15] 13 is a timing chart relating to motion vector detection in the case of 11 points. [Figure 16] FIG. 13 is a diagram showing an example of feature point distribution when the number of feature points is inappropriate. [Figure 17] 13 is a timing chart relating to motion vector detection in the case of 16 points. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. <First embodiment> 1 is a diagram showing a schematic functional block configuration of a digital camera as an application example of an image processing device according to a first embodiment of the present invention. The digital camera of this embodiment is an imaging device (hereinafter, referred to as a stereo camera) capable of acquiring data of two parallax images having parallax corresponding to the left and right eyes, which are captured by, for example, two optical systems with wide angles of view corresponding to the left and right eyes. Note that in this embodiment, a configuration in which two optical systems are arranged in parallel is given as an example, but the present invention is not limited to this, and the digital camera may be composed of three or more optical systems.

[0010] 1, the first imaging optical unit 101 and the second imaging optical unit 131 each include a lens and an aperture, and the two imaging optical units are arranged in parallel to form a pair of optical systems. When photographing, the first imaging optical unit 101 and the second imaging optical unit 131 perform focus adjustment and exposure adjustment, and form an image circle of the two optical systems on the image sensor 102. A specific example of the image circle of the two optical systems on the image sensor 102 will be described later with reference to FIG. 9.

[0011] The image sensor 102 has a photoelectric conversion function for converting an optical image into an electrical signal (analog image signal), and is composed of a CCD, CMOS sensor, etc. The A / D converter 103 converts the analog image signal from the image sensor 102 into digital image data (hereinafter, image data will be referred to simply as image, where appropriate).

[0012] DRAM 107, a volatile memory, is a memory that stores data and has a storage capacity sufficient to store a predetermined number of still images, a predetermined amount of moving images, audio data, and other data, as well as constants and programs for the operation of CPU 112. DRAM is an abbreviation for Dynamic Random Access Memory (Dynamic RAM). Since contents of DRAM are lost over time after writing, it is necessary to refresh the DRAM at regular intervals to maintain the memory. Memory control unit 106 writes and reads data to and from DRAM 107 in response to instructions from CPU 112 or data transfer unit 105.

[0013] The non-volatile memory control unit 108 writes and reads data to and from the ROM 109, which is a non-volatile memory, in response to an instruction from the CPU 112. The ROM 109 is an electrically erasable and recordable memory, and an EEPROM or the like is used. Constants, programs, etc. for the operation of the CPU 112 are stored in the ROM 109.

[0014] The bus 114 is a system bus, and the bus 115 is an image data bus. Data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that perform data transfer. DMA is an abbreviation for Direct Memory Access. DMA is one of the data transfer methods within a computer system, and is a method of transferring data directly between peripheral devices and main memory (RAM) without going through the CPU. DMAC is also an abbreviation for DMA controller. A DMA controller is a controller that controls direct data transfer between memories and between memories and hardware.

[0015] The CPU 112 is composed of a microcomputer or the like that controls the entire stereo camera, and issues operation instructions to each functional block and executes various control processes. The CPU 112 controls the image processing unit 104, the data transfer unit 105, the memory control unit 106, the non-volatile memory control unit 108, the display control unit 110, the operation unit 113, the image sensor 102, and the like via a bus 114. The microcomputer in the CPU 112 executes a program recorded in the ROM 109 to realize each process according to this embodiment.

[0016] The display unit 111 is made up of a liquid crystal monitor or the like, and is controlled by the display control unit 110 to display various image data and the like on the display unit 111 . An operation unit 113 includes switches, buttons, etc. that are operated by the user, and is used for operations such as turning the power on / off and the shutter on / off.

[0017] The image processing unit 104 is composed of a buffer memory and other components that perform various types of image processing, such as white balance adjustment, color interpolation, reduction / enlargement, filtering, encoding, and decoding, and further includes a motion vector detection unit 121 and a subject detection unit 122. The image processing unit 104 also performs various types of image processing on image data captured by the image circles of the two optical systems, taking into account the optical center shift, and can generate image data for stereoscopic display.

[0018] The subject detection unit 122 detects the main subject, such as the face or object of the subject, from the captured image by a known method. The effective range of the subject's coordinates is expressed by, for example, the start coordinates in the upper left corner and the end coordinates in the lower right corner of the image, and the subject's coordinate information is stored in the DRAM 107 via the data transfer unit 105 and the memory control unit 106.

[0019] The motion vector detection unit 121 detects a motion vector based on the image data. Here, as a technique for detecting a motion vector, a technique is known in which feature points are extracted and template blocks are arranged at the extracted feature points in order to improve the detection accuracy of the motion vector. However, when feature points are extracted from the entire image, the distribution of feature points often becomes uneven. If a motion vector obtained based on uneven feature points is used for, for example, image stabilization, the image stabilization will be dominated by the area where feature points are concentrated. Therefore, in order to distribute the feature points uniformly, there is a technique in which an image is divided into a grid, a feature value representing the size of the feature is calculated for each pixel, and the pixel with the largest feature value in each grid is extracted as the feature point. The motion vector detection unit 121 of this embodiment performs template matching processing using a rectangular search area and a template area of ​​a preset size centered on the feature point extracted in this way.

[0020] Fig. 2 is a block diagram showing an example of the configuration of the motion vector detection unit 121. Fig. 3 is a diagram showing the relationship between the grid arrangement, feature points, and template matching area read out by the motion vector detection unit 121. Fig. 4 is a flowchart showing the flow of processing by each block of the motion vector detection unit 121 shown in Fig. 2.

[0021] The motion vector detection unit 121 tracks feature points between multiple frame images and detects motion vectors. The motion vector detection unit 121 takes one of two consecutive frame images on the time axis as an original image (image of the current frame) and the other as a reference image (image of the previous frame), and performs correlation value calculation processing using pixel values ​​within a rectangular area of ​​the reference image centered on a feature point and a rectangular area of ​​the original image. The motion vector detection unit 121 then tracks the subject based on the motion vector detected by the correlation value calculation processing.

[0022] The configuration and operation of the motion vector detection unit 121 will be described below with reference to FIG. 2 and subsequent figures. First, the relationship between the grid arrangement, the feature points, and the template matching area shown in FIG. 3 will be described. As shown in FIG. 3, the motion vector detection unit 121 divides a frame image into a plurality of grids. The grids are arranged in a preset number in the horizontal and vertical directions as feature point arrangement grids 302 and peripheral grids 301 (grids with dot patterns in FIG. 3) of a preset size. The motion vector detection unit 121 extracts one feature point 303 from each feature point arrangement grid 302. The example in FIG. 3 shows an example in which a feature point 303 indicated by coordinates (x, y) is extracted from a feature point arrangement grid 302 whose start coordinates are (Xs, Ys) and whose end coordinates are (Xe, Ye). The peripheral grids 301 are arranged around a group of feature point arrangement grids made up of the feature point arrangement grids 302. The peripheral grids 301 are an area in which feature point extraction is not performed but which is used in template matching. The motion vector detection unit 121 performs template matching processing by setting a search area 305 and a template area 304, which are rectangular areas of a predetermined size in the reference image of the previous frame, with the coordinates (x, y) of a feature point 303 extracted from the original image, which is the current frame, as the center.

[0023] Next, an overview of the processing in each block of the motion vector detection unit 121 will be described with reference to the flowchart in FIG. In step S401, the first RDDMAC 221 included in the data transfer unit 105 reads input image data 241 of a current frame to be subjected to motion vector detection from the DRAM 107 via the bus 115. The RDDMAC is a controller that controls direct data reading from the memory to the hardware. The amount of data read at this time is set to an amount according to the sizes of the feature point arrangement grid 302 and the surrounding grid 301 set as shown in FIG. 3. The input image data 241 is data after various image processes are performed in the image processing unit 104. The image data 241 read in units of grids by the first RDDMAC 221 is output to the matching image generation unit 201 and the feature point extraction unit 202. The first RDDMAC 221 also outputs grid coordinate information 252 indicating the coordinate position of the read grid to the tracking target feature point determination unit 205 described later. After the process of step S401, the motion vector detection unit 121 performs the process of step S402 and the process thereafter, S403, and the process of step S404 in parallel.

[0024] In step S402, the matching image generating unit 201 generates a matching image to be used for template matching during motion vector detection, and outputs it to the WRDMAC 231. The WRDMAC is a controller that controls writing data directly from hardware to memory. The matching image generating unit 201 also has a band-pass filter circuit, and cuts out high-frequency and low-frequency components of the image signal that are not required for template matching processing.

[0025] Next, in step S403, the WRDMAC 231 included in the data transfer unit 105 performs processing to write out the matching image data 242 input to the DRAM 107 via the bus 115. Also stored in the DRAM 107 is matching image data 243 generated in the previous frame that precedes the current frame on the time axis. After step S403, the processing of the motion vector detection unit 121 proceeds to step S405.

[0026] In addition, when the process proceeds to step S404, the feature point extraction unit 202 or the feature point reflection unit 207 calculates new feature points for the current frame. Here, a description will be given of the feature point extraction process in the feature point extraction unit 202. Fig. 5 is a block diagram showing an example of the configuration of the feature point extraction unit 202. The feature point extraction unit 202 is configured to include a feature filter unit 501, a feature evaluation unit 502, and a feature point determination unit 503.

[0027] The feature filter unit 501 is configured with a plurality of filters, such as a band-pass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. The feature filter unit 501 first cuts unnecessary high-frequency components and low-frequency components from the image data using a band-pass filter. Next, the feature filter unit 501 further performs horizontal differential filtering using a horizontal differential filter and vertical differential filtering using a vertical differential filter on the data after the band-pass filtering process. Then, the feature filter unit 501 outputs the image data after smoothing filtering on each of the image data after the horizontal differential filtering process and the image data after the vertical differential filtering process to the feature evaluation unit 502.

[0028] The feature evaluation unit 502 calculates feature points from the image data in units of grids after filtering by the feature filter unit 501. For example, the feature evaluation unit 502 performs a calculation using a predetermined feature evaluation formula and calculates, as feature values, points where differential values ​​around a pixel are large in multiple directions, such as the intersection of two edges or a curved point with a maximum curvature, for each pixel.

[0029] Hereinafter, an example of calculation of feature points in the feature evaluation unit 502 will be described using the so-called Shi and Tomasi method as an example. First, the feature evaluation unit 502 uses the data filtered by the above-mentioned horizontal differential filter and the data filtered by the vertical differential filter to create an autocorrelation matrix H according to the following formula (1). Note that Ix in the formula represents the result of applying the horizontal differential filter, and Iy represents the result of applying the vertical differential filter, and formula (1) is a formula for convolving a Gaussian filter G to create the autocorrelation matrix H.

[0030]

number

[0031] Then, the feature evaluation unit 502 calculates the feature value by the feature evaluation formula of Shi and Tomasi shown in the following formula (2). Note that formula (2) indicates that the smaller eigenvalue of eigenvalues ​​λ1 and λ2 of the autocorrelation matrix H in formula (1) is set as the feature value.

[0032] Shi and Tomasi = min(λ1, λ2) Equation (2)

[0033] The feature value calculated for each pixel in this manner by the feature evaluation unit 502 is sent to a feature point determination unit 503 . The feature point determination unit 503 determines, for each grid, a pixel having the largest feature value among the feature values ​​calculated by the feature evaluation unit 502 for each pixel in the grid as a feature point in that grid. In the case of this embodiment, the feature point information is expressed as relative coordinates (PX, PY) with the upper left corner of the grid being coordinates (0, 0). Note that the feature point coordinates may be expressed as absolute coordinates in the frame image. Then, the feature point information determined by the feature point determination unit 503 (hereinafter referred to as feature point coordinate information) is stored in an internal memory provided in the feature point extraction unit 202. This internal memory has a capacity sufficient to store the information of each feature point acquired from the previous frame and the feature point coordinate information acquired from the current frame.

[0034] When the matching processing unit 203 starts template matching processing of the corresponding grid, the feature point extraction unit 202 reads out feature point coordinate information 251 calculated in the previous frame from the internal memory and outputs it to the tracked feature point determination unit 205. The tracked feature point determination unit 205 outputs the feature point coordinate information 251 sent from the feature point extraction unit 202 as tracked feature point coordinate information 257 to the second RDDMAC 222 included in the data transfer unit 105 .

[0035] Next, the feature point reflection process in the feature point reflection unit 207 will be described. Here, the image captured by the first imaging optical section 101 is called a first image, and the image captured by the second imaging optical section 131 is called a second image. The feature point reflecting unit 207 reflects the feature points extracted from the first image by the feature point extracting unit 202 to the feature points of the second image while taking into account the parallax. If the parallax is u, the distance to the subject is Z, the base line distance between the first imaging optical unit 101 and the second imaging optical unit 131 is d, and the focal length of these optical units is f, the parallax u can be calculated by the following formula (3).

[0036] u=(d×f) / Z Equation (3)

[0037] Furthermore, the feature point reflection unit 207 adds the parallax u to any feature point coordinates (x, y) of the first image to obtain feature point coordinates (x+u, y) of the second image with the parallax added. The current frame feature points that are the reflection source are stored in the SRAM 206, and new feature points that have been subjected to the feature point reflection process described above are stored in the internal memory of the feature point reflection unit 207.

[0038] Then, when the matching processing unit 203 starts template matching processing of the corresponding grid, the feature point reflection unit 207 reads out from the internal memory the feature point coordinate information 259 that has been subjected to the feature point reflection processing, and outputs it to the tracked target feature point determination unit 205. The tracked feature point determination unit 205 outputs the feature point coordinate information 259 sent from the feature point reflection unit 207 to the second RDDMAC 222 included in the data transfer unit 105 as tracked feature point coordinate information 257 in the second image. A specific description of how new feature points are calculated for images captured by the stereo camera through the feature point reflection process described above will be given later.

[0039] Returning to the flowchart of FIG. After the processes of steps S404 and S403, which are performed in parallel as described above, the motion vector detection unit 121 advances the process to step S405. In step S405, the second RDDMAC 222 reads data of a rectangular area of ​​a preset size centered on the tracking target feature point coordinate information 257 calculated in the previous frame as described above. That is, data of the search area 305 in FIG. 3 is read from the matching image data 242 generated in the current frame, while data of the template area 304 is read from the matching image data 243 generated in the previous frame. Then, the read search area image data 253 and template area image data 254 are sent to the matching processing unit 203. However, in the first template matching process, since the tracking target feature point coordinates cannot be calculated from the previous frame, the process is performed using all new feature point coordinates calculated in the current frame. After step S405, the process of the motion vector detection unit 121 proceeds to step S406.

[0040] In step S406, the matching processing unit 203 calculates a correlation value using the input search area image data 253 and template area image data 254, and calculates a vector value based on the correlation value. After step S406, the processing of the motion vector detection unit 121 proceeds to step S407.

[0041] In this embodiment, the Sum of Absolute Difference (hereinafter abbreviated as SAD) is used as a method for calculating the correlation value. Equation (4) is a calculation formula for SAD.

[0042]

number

[0043] In formula (4), f(i,j) represents a pixel value at coordinates (i,j) in the template region image data 254, and g(i,j) represents each pixel value in a target region for correlation value calculation (hereinafter referred to as correlation value calculation region) in the search region image data 253. The correlation value calculation region has the same size as the region in the template region image data 254. The matching processing unit 203 calculates the absolute value of the difference between each pixel value f(i,j) and g(i,j) in these regions, and calculates the correlation value S_SAD by calculating the sum of the absolute values. Here, the smaller the correlation value S_SAD, the smaller the difference in luminance value between the two regions, that is, the more similar the textures are between the template region image data 254 and the correlation value calculation region. In this embodiment, SAD is used as an example of a correlation value, but this is not limited thereto, and other correlation values ​​such as sum of squared differences (SSD) and normalized cross correlation (NCC) may be used.

[0044] Then, the matching processing unit 203 obtains the position of the minimum correlation value as the motion vector value of the grid. The matching processing unit 203 outputs the vector value and correlation value information 255 calculated as described above to the accuracy determination unit 204.

[0045] Next, in step S407, the accuracy determination unit 204 uses the correlation values ​​of the vector values ​​and correlation value information 255 calculated in the process of step S406 to determine the maximum, minimum, average, and minimum values ​​of the correlation values.The accuracy determination unit 204 then uses the maximum, minimum, average, and minimum values ​​of the correlation values ​​to perform processes of low contrast determination, maximum pixel value protrusion determination, and repetitive pattern determination, as described below.After step S407, the process of the motion vector detection unit 121 proceeds to step S408.

[0046] Figures 6(a) to 6(d) are graphs used to explain the relationship between the maximum, minimum, average, and minimum values ​​of the correlation value and examples of determination using them. However, in this embodiment, the smaller the correlation value, the higher the similarity, so in Figures 6(a) to 6(d), the maximum pixel value represents the minimum correlation value, the minimum pixel value represents the maximum correlation value, and the maximum pixel value represents the minimum correlation value.

[0047] FIG. 6(a) shows an example in which each determination result is good. FIG. 6(b) shows an example where a low contrast is determined in the low contrast determination. The low contrast determination is a process for determining whether or not the correlation value calculation region has low contrast. In the low contrast determination in this embodiment, when the difference between the maximum and minimum correlation values ​​in the correlation value calculation region is smaller than a predetermined threshold value that is set in advance, the correlation value calculation region is determined to have low contrast. In the case of FIG. 6(b), the difference between the maximum and minimum pixel values ​​is smaller than that in FIG. 6(a), and the difference between the maximum and minimum correlation values ​​in the correlation value calculation region is also smaller than a predetermined threshold value. Therefore, in the case of FIG. 6(b), the accuracy determination unit 204 determines that the correlation value calculation region has low contrast because the difference between the maximum and minimum correlation values ​​in the correlation value calculation region is smaller than a predetermined threshold value.

[0048] FIG. 6(c) shows an example where a pixel value is determined to be a low peak in the maximum value protrusion determination. The maximum value protrusion determination of pixel values ​​is a process for determining how prominent the minimum value of the correlation value in the correlation value calculation region is. In the maximum value protrusion determination of pixel values ​​in this embodiment, when the value of the division result of the difference between the maximum value and average value of pixel values ​​and the difference between the maximum value and minimum value of pixel values ​​is smaller than a predetermined threshold value set in advance, the correlation value calculation region is determined to be a low peak. On the other hand, when the value of the division result is larger than the predetermined threshold value, the correlation value calculation region is determined to be a high peak. In the case of FIG. 6(c), the value of the division result of the difference between the maximum value and average value of pixel values ​​and the difference between the maximum value and minimum value of pixel values ​​is smaller than the threshold value. Therefore, in the case of FIG. 6(c), the accuracy determination unit 204 determines that the correlation value calculation region is a low peak because the value of the division result is smaller than the predetermined threshold value.

[0049] FIG. 6(d) shows an example of a repeat pattern determined by the repeat pattern determination. The repeat pattern determination is a process for determining whether or not pixel values ​​are repeatedly fluctuating. In the repeat pattern determination in this embodiment, if the difference between the maximum value and the maximum value of the pixel value in the correlation value calculation area is smaller than a predetermined threshold value that is set in advance, the repeat pattern is determined. In the case of FIG. 6(d), the difference between the maximum value and the maximum value of the pixel value is smaller than the predetermined threshold value compared to FIG. 6(a). Therefore, in the case of FIG. 6(d), the accuracy determination unit 204 determines that the repeat pattern is present because the difference between the maximum value and the maximum value of the pixel value in the correlation value calculation area is smaller than the predetermined threshold value.

[0050] Return to the flow chart of FIG. 4. After step S407, the motion vector detection unit 121 proceeds to processing of step S408. When proceeding to step S408, the precision determination unit 204 writes the above-mentioned low contrast, low peak, and repeat pattern determination information and vector information 256 to the SRAM 206. As a result, the information is stored in the storage area 244 of the SRAM 206. Note that SRAM is an abbreviation for static random access memory (static RAM). Among semiconductor storage devices (semiconductor memories), SRAM is one of the volatile memories whose records are lost when the power is turned off, and has the characteristic that when data is written to the element, the contents are not lost while the power is turned on even if a refresh operation is not performed. After step S408, the motion vector detection unit 121 proceeds to processing of step S409.

[0051] In step S409, the tracked feature point determination unit 205 calculates tracked feature points to be used in the template matching process of the next frame. 7 is a flowchart showing the flow of processing in the tracked feature point determining unit 205. Below, an overview of the tracked feature point determination processing in the tracked feature point determining unit 205 will be described with reference to the flowchart in FIG.

[0052] First, in the process of step S701, the tracked feature point determination unit 205 acquires a vector value and the above-mentioned determination information 258 from the SRAM 206. The tracked feature point determination unit 205 calculates a tracked feature point of the current frame using the acquired vector value, and further calculates the coordinates of the tracked feature point of the next frame by adding the vector value calculated in the template matching process centered on the coordinates of the tracked feature point. After step S701, the tracked feature point determination unit 205 proceeds to step S702.

[0053] When proceeding to step S702, the tracking target feature point determination unit 205 determines whether the tracking target feature point calculated in step S701 is a valid tracking target feature point that can be used for the template matching process of the next frame based on the above-described determination information acquired from the SRAM 206. If the tracking target feature point determination unit 205 determines that it is low contrast in the low contrast determination, or determines that it is a low peak in the maximum pixel value prominence determination, or determines that it is a repeating pattern in the repeating pattern determination, the tracking target feature point determination unit 205 determines that the tracking target feature point is invalid (NG determination). On the other hand, when the tracking target feature point determination unit 205 is not any of low contrast, low peak, and repeating pattern, the tracking target feature point determination unit 205 determines that the tracking target feature point is valid (OK determination). Then, in step S702, if the tracking target feature point determination unit 205 makes an NG determination (NO), the process proceeds to step S704, while if it makes an OK determination (YES), the process proceeds to step S703.

[0054] When proceeding to step S703, the tracking target feature point determination unit 205 determines whether the coordinates of the tracking target feature point calculated in step S701 are within the grid. Specifically, the tracking target feature point determination unit 205 determines OK when the coordinates of the tracking target feature point are within the feature point arrangement grid 302. In the case of the example in FIG. 3 above, the tracking target feature point determination unit 205 determines OK when Xs ≤ x ≤ Xe and Ys ≤ y ≤ Ye from the start coordinates (Xs, Ys) and end coordinates (Xe, Ye) of the feature point arrangement grid 302 and the coordinates (x, y) of the feature point 303. On the other hand, the tracking target feature point determination unit 205 determines NG when the coordinates of the tracking target feature point are not within the feature point arrangement grid 302. That is, the tracking target feature point determination unit 205 determines NG when any of x < Xs, Xe < x, y < Ys, Ye < y is satisfied. Then, when the tracking target feature point determination unit 205 determines OK, the process proceeds to step S703, and when it determines NG, the process proceeds to step S704.

[0055] In step S704, the tracked feature point determination unit 205 deletes the tracked feature point coordinates corresponding to the coordinate position determined as NG in step S702 or step S703. Then, the tracked feature point determination unit 205 performs processing to replace the deleted tracked feature point coordinates with the new feature point coordinates calculated in step S404. After step S704, the tracked feature point determination unit 205 advances the processing to step S705.

[0056] In step S705, the tracked feature point determination unit 205 determines whether or not all the tracked feature point coordinates have been calculated, the number of which is equal to the number of feature points. Here, the number of feature points is set by the feature point number setting unit 208. If the tracked feature point determination unit 205 determines in step S705 that all the tracked feature point coordinates have not been calculated (NO), the process returns to step S701 and performs the above-mentioned process. The processes from step S701 to step S705 are repeated until it is determined in step S705 that all the tracked feature point coordinates have been calculated (YES). On the other hand, if the tracked feature point determination unit 205 determines in step S705 that all the tracked feature point coordinates have been calculated (YES), the process of the flowchart in FIG. 7 is terminated. This completes the process of each block of the motion vector detection unit 121 based on the process of each step in the flowchart in FIG. 4.

[0057] Next, the details of the calculation process of new feature points by the feature point reflection process performed by the feature point reflection unit 207 in step S404 of the flowchart shown in FIG. 4 will be described with reference to FIGS. Fig. 8 is a diagram showing an example of distribution of motion vectors in an image when, for example, camera shake occurs in which the camera rotates in an axis direction (roll direction) about the midpoint between the first imaging optical unit 101 and the second imaging optical unit 131, among the camera movements caused by camera shake and the like. Note that camera shake generally includes components of camera movement in the yaw direction and pitch direction in addition to the roll direction, but the motion vectors in the yaw direction and pitch direction are omitted in Fig. 8. In the following, an explanation will be given by taking as an example only camera shake in the axis direction at the midpoint between the two imaging optical units.

[0058] As shown in Fig. 8, when camera shake occurs in the rotational direction, each motion vector 802 of image data 801 is distributed in a point-symmetrical circular manner. In the example of image data 801 in Fig. 8, the motion vector 802 points upward on the left side and downward on the right side. It can be considered that such a distribution of rotational motion vectors is similar for images captured by a stereo camera. In other words, in the case of images captured by a stereo camera, the motion vectors are considered to be distributed in a point-symmetrical circular manner corresponding to the midpoint of the two imaging optical units.

[0059] 9 is a diagram showing an example of two image circles 901, 902 formed on the imaging surface of the image sensor 102 by two optical systems, the first imaging optical unit 101 and the second imaging optical unit 131, as in the stereo camera of this embodiment. The image sensor 102 captures images of the two image circles 901, 902 to obtain data of two parallax images having parallax in which the same subject 905 is captured. In the following description, the image corresponding to the image circle 902 on the left side is called the left eye image, and the image corresponding to the image circle 901 on the right side is called the right eye image.

[0060] Fig. 10 is a diagram showing an example of distribution of feature points and motion vectors for image data 1001 consisting of a left eye image 1002 and a right eye image 1003 captured by a stereo camera when hand shake occurs in the rotation direction around an axis at the midpoint between two imaging optical units. Note that Fig. 10 shows a feature point arrangement grid, surrounding grid, etc., similar to Fig. 3.

[0061] The motion vector detection unit 121 of this embodiment extracts feature points in all feature point arrangement grids of the left eye image 1002. On the other hand, the motion vector detection unit 121 extracts feature points from some of the feature point arrangement grids for the right eye image 1003, and reflects some of the feature points extracted from the left eye image 1002 in the feature points of the right eye image 1003. Then, the motion vector detection unit 121 detects a motion vector for each of the feature points of the left eye image 1002 and the right eye image 1003. That is, the motion vector detection unit 121 is used for detecting motion vectors of subject movement factors and camera shake factors for the left eye image 1002, and is used for detecting only motion vectors due to feature points suitable for camera shake detection in the rotational direction for the right eye image 1003. In other words, in this embodiment, the motion vector detection unit 121 basically performs vector detection using the left eye image 1002, and uses the right eye image 1003 to assist in camera shake detection in the rotational direction. Details regarding which grid should be used as the feature point arrangement grid for extracting feature points from the right eye image 1003, and which feature points of the left eye image 1002 should be reflected in the feature points of the right eye image 1003, will be described later.

[0062] Here, for example, if motion vector detection is performed using only the left eye image 1002, it is not possible to detect a rotational shake. This is because, in the case of FIG. 10, all the motion vectors in the left eye image 1002 point upward, and it is not possible to determine whether the shake is a rotational shake or an upward translational shake. In this case, if the motion vector in the right eye image points downward, for example, it can be determined that the shake is a rotational shake, and if it points upward, it can be determined that the shake is an upward translational shake. Therefore, if there is at least one motion vector in the right eye image 1003, it is possible to determine whether the shake is a rotational shake or a translational shake.

[0063] Also, for example, if the number of extracted feature points is increased, the accuracy of camera shake detection in the rotational direction can be improved, but in that case, the amount of processing for motion vector detection increases, and the calculation cost increases. For this reason, in this embodiment, the number of feature points handled in the right eye image is limited (restricted in number) to, for example, two points. In this way, in this embodiment, by reducing the number of feature points handled in the right eye image 1003 compared to the left eye image 1002, it is possible to reduce the amount of processing for motion vector detection and reduce the calculation cost.

[0064] On the other hand, if the feature points of the right eye image 1003 are feature points derived from the subject, when the motion vector of the subject is larger than the motion vector of the camera shake, the motion vector of the camera shake cannot be confirmed, and camera shake in the rotational direction cannot be detected correctly. Therefore, the motion vector detection unit 121 of this embodiment reflects feature points extracted from a feature point arrangement grid in which no subject exists in the left eye image 1002 to the right eye image 1003. In other words, in this embodiment, of the feature points extracted from the left eye image 1002, the feature points of the subject are not reflected to the feature points of the right eye image 1003.

[0065] Fig. 11 is a diagram showing an example of the distribution of feature points of image data 1101 consisting of a left eye image 1102 and a right eye image 1103 captured by a stereo camera, and an example of a subject 1105. Note that Fig. 11 also shows a feature point arrangement grid, a surrounding grid, and the like, as in Fig. 3. Fig. 11 shows a processing example in which, for image data 1101 consisting of a left eye image 1102 and a right eye image 1103 in which a subject 1105 exists, feature points of the left eye image 1102 are reflected in image data of the right eye image 1103.

[0066] Fig. 12 is a flowchart showing the flow of the feature point reflection process performed by the motion vector detection unit 121 of this embodiment. Hereinafter, the feature point reflection process as shown in Fig. 11 will be described with reference to the flowchart shown in Fig. 12. First, in step S1201, the motion vector detection unit 121 performs different processing depending on whether the area for which feature points are to be newly calculated is a left eye image or a right eye image. If the area for which feature points are to be newly calculated is a left eye image, the motion vector detection unit 121 proceeds to processing for extracting feature points from the left eye image, whereas if the area for which feature points are to be newly calculated is a right eye image, the motion vector detection unit 121 proceeds to processing for reflecting feature points from the left eye image to the right eye image.

[0067] If the area for calculating new feature points is the left eye image, and the process proceeds to step S1202, the feature point extraction unit 202 extracts feature points from the left eye image. Specifically, as in step S704 described above, the feature point extraction unit 202 performs a process of replacing the feature point arrangement grid at the position where the deleted tracked target feature point was first calculated as a new feature point with the new feature point coordinates of the current frame calculated at the corresponding position.

[0068] On the other hand, when the area for calculating new feature points is the right eye image and the process proceeds to step S1203, the feature point extraction unit 202 assigns a number Gn representing a priority order reflecting the feature points to the feature point arrangement grids of the right eye image. Here, in the case of this embodiment, feature points are not arranged in all feature point arrangement grids in the right eye image, but are limited to a part of them. Also, in the case of this embodiment, the number of feature points in the right eye image is determined based on the processing speed for calculating the motion vector and the accuracy of camera shake detection in the rotational direction.

[0069] Here, if the minimum number of feature points that can detect camera shake in the rotational direction with sufficient accuracy is Nmin, and the maximum number of feature points that can calculate a motion vector at a processing speed sufficient to maintain real-time performance is Nmax, the number of feature points N must satisfy Nmin≦N≦Nmax. To reduce the calculation cost, N=Nmin may be set. In the example of FIG. 11, the number of feature points in the right eye image is described as two. For this reason, in step S1203, the feature point extraction unit 202 assigns a number Gn=1-Gmax representing a priority order for reflecting feature points in the feature point reflection unit 207 described later to the feature point arrangement grid of the right eye image. In addition, the number Gn=1 indicates the highest priority order, and Gn=Gmax indicates the lowest priority order. In the example of FIG. 11, the feature point arrangement grid with the lowest priority order is Gmax=9. In other words, it is considered that the greater the distance from the midpoint between the left eye image and the right eye image, the greater the motion vector value of the camera shake factor in the rotational direction. For this reason, in this embodiment, the greater the distance from the midpoint between the left eye image and the right eye image, the higher the priority is set to the grid. That is, in the example of FIG. 11, it is considered that the motion vector value of the cause of camera shake in the rotational direction will be a large value in the feature point arrangement grid on the right side of the figure in the right eye image. For this reason, in this embodiment, the priority is set to be higher for the feature point arrangement grid on the right side of the right eye image (Gn=1 in the example of FIG. 11). In this embodiment, the feature point extraction unit 202 first initializes the priority number Gn to 1, and processing is performed in order starting from the feature point arrangement grid with the highest priority.

[0070] Next, in step S1204, the feature point extraction unit 202 determines whether or not a feature point already exists in the grid of interest. That is, in the present embodiment, since a maximum of one feature point is arranged in one feature point arrangement grid, the feature point extraction unit 202 divides the process according to whether or not a feature point already exists in the feature point arrangement grid corresponding to the priority order number Gn in the right eye image. If the feature point extraction unit 202 determines that a feature point already exists in the grid of interest, the process proceeds to step S1211, increments the number Gn (G=Gn+1), and then returns the process to step S1204. For example, in the case of the image data 1101 in FIG. 11, the feature point 1104 already exists in the feature point arrangement grid corresponding to Gn=3 in the right eye image 1103, so that the calculation process of the feature point is performed in a grid other than Gn=3. On the other hand, if the feature point extraction unit 202 determines that a feature point does not exist in the grid of interest, the process proceeds to step S1205.

[0071] When proceeding to step S1205, the feature point extraction unit 202 judges whether or not a subject exists in the grid of interest. That is, the feature point extraction unit 202 divides the process according to whether or not a subject detected by the subject detection unit 122 in FIG. 1 exists in the feature point arrangement grid corresponding to the number Gn in the left eye image, which is the reflection source of the feature points of the right eye image. If the feature point extraction unit 202 judges that a subject does not exist, the process proceeds to step S1210 performed by the feature point reflection unit 207. On the other hand, if it is judged that a subject exists, the feature point extraction unit 202 advances the process to step S1206 and subsequent steps. That is, if a subject exists, the camera shake detection in the rotation direction is not performed correctly, so the feature point extraction unit 202 performs a feature point calculation process for the feature point arrangement grids of other numbers Gn. In the case of the image data 1101 in FIG. 11, since the subject 1105 exists in the grids of Gn=1 and Gn=2, it is necessary to calculate feature points in grids other than Gn=1 and Gn=2.

[0072] In step S1206, the feature point extraction unit 202 determines whether or not Gn=Gmax has been achieved. If the feature point extraction unit 202 determines that Gn=Gmax has been achieved, the process proceeds to step S1209 performed by the feature point reflection unit 207. On the other hand, if the feature point extraction unit 202 determines that Gn=Gmax has not been achieved, the process proceeds to step S1207.

[0073] When the process proceeds to step S1207, the feature point extraction unit 202 determines whether or not this step has been passed for the first time in the feature point extraction process. If the feature point extraction unit 202 has already passed this step, the process proceeds to step S1211, sets Gn=Gn+1, and returns the process to step S1204. On the other hand, if the feature point extraction unit 202 determines that this step has been passed for the first time, the process proceeds to step S1208.

[0074] In step S1208, the feature point extraction unit 202 holds the value of the number Gn, and then proceeds to step S1201, where it sets Gn=Gn+1, and returns the process to step S1204.

[0075] When it is determined in step S1206 that Gn=Gmax and the process proceeds to step S1209, the feature point reflecting unit 207 performs processing to reflect feature points from the left eye image to the right eye image based on the value of the number Gn held in step S1208. The processing in step S1209 is a procedure to be taken when an appropriate feature point to be reflected from the left eye image to the right eye image cannot be found, and although the feature point is within the feature point arrangement grid in which the subject is detected, the feature point is reflected using data with a high priority. After step S1209, the feature point reflecting unit 207 proceeds to the process in step S1210.

[0076] When the process proceeds to step S1210, the feature point reflection unit 207 reflects the feature points of the left eye image to the feature points of the right eye image, taking into account the parallax between the left eye image and the right eye image, based on the value of the number Gn. In the case of image data 1101 in Fig. 11, an example in which feature points are reflected at Gn=4 is shown.

[0077] Thereafter, as described above, the tracking target feature point determination unit 205 and the matching processing unit 203 perform processing to calculate a motion vector, which enables the motion vector detection unit 121 to detect camera shake in the rotational direction.

[0078] As described above, the image processing device of the first embodiment detects the motion vector of the cause of rotational camera shake by reflecting some of the feature points of the left eye image in the feature points of the right eye image, thereby making it possible to reduce computational costs while detecting rotational camera shake.

[0079] In the above-mentioned FIG. 11, the priority order numbers are assigned so that the right side of the right eye image has a higher priority, but the priority order can be freely determined. For example, in image data captured from a stereo camera, image distortion is smaller near the center of each of the left eye image and the right eye image (a region within a predetermined range from the center coordinates or a rectangular region of a predetermined width and height). Therefore, the priority order may be set higher (preferentially set) near the center of each image. In addition, in order to prevent feature points from gathering at a certain location and causing bias in the obtained motion vector, the priority order may be set so that the feature points are evenly scattered throughout the entire image. This allows the feature points to be reflected so that the distance between each feature point in the right eye image is evenly distributed, for example.

[0080] <Second embodiment> Next, an image processing device according to a second embodiment of the present invention will be described. Note that the configuration of the image processing device according to the second embodiment is the same as that of Fig. 1 and Fig. 2 described above, and therefore illustration and description thereof will be omitted. Furthermore, the processing in the image processing device of the second embodiment is also generally similar to that described in Fig. 3 to Fig. 12, but whereas the number of feature points reflected from the left eye image to the right eye image is fixed in the first embodiment, the image processing device of the second embodiment dynamically controls the number of feature points over time.

[0081] Hereinafter, a feature point number setting process for dynamically controlling the number of feature points over time in the image processing device of the second embodiment will be described with reference to FIGS. FIG. 13 is a flowchart showing the flow of the feature point number setting process in the image processing apparatus of the second embodiment.

[0082] First, in step S 1301 , the rotational direction camera shake detection unit 209 detects camera shake in the rotational direction using the motion vector data stored in the SRAM 206 , and sends information on the amount of camera shake in the rotational direction to the feature point number setting unit 208 .

[0083] The feature point number setting unit 208 compares the amount of camera shake in the rotational direction detected in the current frame by the rotational direction camera shake detection unit 209 with the amount of camera shake in the rotational direction detected in the previous frame. Then, the feature point number setting unit 208 determines whether the amount of camera shake in the rotational direction has increased in the current frame compared to the previous frame. Then, if the feature point number setting unit 208 determines that the amount of camera shake in the rotational direction has increased in the current frame compared to the previous frame, the process proceeds to step S1304 and subsequent steps, which are steps for reducing the number of feature points in order to reduce calculation costs. On the other hand, if the feature point number setting unit 208 determines that the amount of camera shake in the rotational direction has not increased in the current frame compared to the previous frame, the process proceeds to step S1302 and subsequent steps, which are steps for increasing the number of feature points in order to improve the accuracy of camera shake detection in the rotational direction.

[0084] In step S1302, the feature point number setting unit 208 determines whether the number of feature points N is equal to the minimum number Nmin of feature points capable of detecting camera shake in the rotational direction. If N=Nmin, the feature point number setting unit 208 cannot further reduce the number of feature points N, and therefore ends the feature point number setting process according to the flowchart in Fig. 13. On the other hand, if the feature point number setting unit 208 determines that the number of feature points N is not equal to the minimum number Nmin, the process proceeds to step S1303.

[0085] When proceeding to step S1303, the feature point number setting unit 208 decreases the feature point number N within a range where the feature point number N does not become less than the minimum point number Nmin (N < Nmin). After this step S1303, the feature point number setting unit 208 ends the processing of the flowchart in FIG. 13.

[0086] Also, when proceeding to step S1304, the feature point number setting unit 208 determines whether the feature point number N is equal to the maximum point number Nmax of the feature points that can calculate the motion vector while maintaining real-time performance. And when N = Nmax, since the feature point number N cannot be further increased, the feature point number setting unit 208 ends the processing of the flowchart in FIG. 13. On the other hand, when the feature point number setting unit 208 determines that the feature point number N is not equal to the maximum point number Nmax, the processing proceeds to step S1305.

[0087] When proceeding to step S1305, the feature point number setting unit 208 increases the feature point number N within a range where the feature point number N does not exceed the maximum point number Nmax (N > Nmax). After this step S1305, the feature point number setting unit 208 ends the processing of the flowchart in FIG. 13.

[0088] Here, the method of determining the maximum point number Nmax will be described with reference to FIGS. 14 to 17. FIG. 14 is a diagram showing an example of the distribution of feature points for the image data 1401 composed of the left-eye image 1402 and the right-eye image 1403 captured from the stereo camera, similar to the example of FIG. 10 described above. Note that arrows representing motion vectors are not shown in FIG. 14. Also, FIG. 14 shows a feature point arrangement grid, a peripheral grid, etc., similar to FIG. 3. In the example of FIG. 14, a total of 11 feature points are arranged, with 9 feature points in the left-eye image 1402 and 2 feature points in the right-eye image 1403.

[0089] Fig. 15 is a timing chart relating to motion vector detection in the case where there are 11 feature points as illustrated in Fig. 14. As shown in Fig. 15, in the motion vector detection unit 121, during one frame, a matching image is generated from image data output from the image sensor 102, and template matching is performed sequentially according to the number of feature points N. Here, the time 1505 of one frame is RVD, the time 1506 required until template matching for the first feature point starts is T0, and the processing time 1507 required to perform one template matching obtained in advance is TT. In this case, these times need to satisfy the relationship TDV>T0>T0+N×TT.

[0090] As shown in Fig. 15, when the number of feature points is 11, the processing time (T0+N×TT) for detecting motion vectors for these 11 feature points falls within the time TDV of one frame shown as VD in Fig. 15. This shows that when the number of feature points is 11, motion vectors can be calculated in real time.

[0091] Fig. 16 is a diagram showing an example of the distribution of feature points for image data 1601 consisting of a left eye image 1602 and a right eye image 1603 captured by a stereo camera. In the example of Fig. 16, a total of 16 feature points are arranged, with nine feature points arranged in the left eye image 1602 and seven feature points arranged in the right eye image 1603. Note that Fig. 16 shows a feature point arrangement grid, surrounding grid, etc., similar to Fig. 3.

[0092] FIG. 17 is a timing chart for motion vector detection in the case where there are 16 feature points as illustrated in FIG. 16. In FIG. 17, the time for one frame is time 1705, the time until template matching for the first feature point starts is time 1706, and the processing time required for executing one template matching is time 1707. In the motion vector detection unit 121, as described above, a matching image is generated from image data output from the image sensor 102 during one frame, and template matching is sequentially performed according to the number of feature points N. In the case where there are 16 feature points, the processing time (T0+N×TT) for detecting motion vectors for the 16 feature points does not fall within the time TDV for one frame shown by VD in FIG. 17, and therefore it is understood that a motion vector cannot be calculated while maintaining real-time performance. That is, the number of feature points N of 16 points is not appropriate. From the example of FIG. 17, it can be said that the maximum number Nmax of feature points for which a motion vector can be calculated while maintaining real-time performance is 15 points.

[0093] As described above, the image processing device according to the second embodiment dynamically controls the number of feature points reflected from the left eye image to the right eye image over time. As a result, according to the second embodiment, it is possible to reduce the computation cost under optimal conditions based on the processing speed for calculating the motion vector and the accuracy of the camera shake detection in the rotational direction.

[0094] As described above, in the image processing device of the first and second embodiments, the calculation cost consumed for template matching can be reduced by limiting the number of feature points without limiting the search area. As a result, the image processing device of the first and second embodiments can inexpensively detect hand shake shake in the rotational direction for image data captured by the stereo camera, and can improve the performance of motion vector detection.

[0095] In the first and second embodiments, an example is given in which feature points are reflected from a left eye image to a right eye image, but feature points may be reflected from a right eye image to a left eye image. Also, in the first and second embodiments, the processing is not limited to the case of two images, a left eye image and a right eye image, and similar processing may be performed in the case of a plurality of images.

[0096] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0097] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0098] The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics. [Explanation of symbols]

[0099] 101: first imaging optical unit, 131: second imaging optical unit, 102: image sensor, 104: image processing unit, 112: CPU, 121: motion vector detection unit, 202: feature point extraction unit, 207: feature point reflection unit, 208: feature point number setting unit, 209: rotational direction camera shake detection unit

Claims

1. An acquisition means for acquiring a plurality of images having parallax by a plurality of optical systems; An extraction means for extracting feature points from the image; a reflecting means for reflecting the feature points extracted from a first image among the plurality of images on the feature points extracted from a second image while taking into account the parallax; a detection means for detecting a motion vector based on the feature points, The image processing device according to claim 1, wherein the detection means detects the motion vector for the second image based on feature points that reflect feature points of the first image.

2. 2. The image processing device according to claim 1, further comprising a camera shake detection means for detecting camera shake in a rotational direction of an imaging device that captured the plurality of images having parallax due to the plurality of optical systems, using a motion vector detected based on the feature points of the first image and a motion vector detected based on feature points of the second image in which the feature points of the first image are reflected.

3. 3. The image processing apparatus according to claim 2, wherein said reflecting means limits the number of feature points of said second image which reflect the feature points of said first image.

4. 4. The image processing apparatus according to claim 3, wherein said reflecting means determines the number of said feature points to be limited based on a processing speed at which said detecting means detects said motion vector.

5. 4. The image processing apparatus according to claim 3, wherein said reflecting means determines the number of said feature points to be limited based on the accuracy of the camera shake detection in the rotational direction.

6. 6. The image processing device according to claim 3, wherein the reflecting means reflects, among the feature points extracted from the first image, feature points suitable for detecting camera shake in the rotational direction of the imaging device, on the feature points of the second image.

7. a subject detection means for detecting a subject from the image, 7. The image processing device according to claim 1, wherein the reflecting means does not reflect feature points of the subject among the feature points extracted from the first image onto the feature points of the second image.

8. 8. The image processing device according to claim 1, wherein the reflection means sets a priority for each of the second image divided into a plurality of regions, and reflects the feature points extracted from the first image on the feature points of the regions of the second image based on the set priority for each region.

9. 9. The image processing apparatus according to claim 8, wherein the reflecting means sets a higher priority order for the area located farther away from the midpoint between the first image and the second image.

10. 9. The image processing apparatus according to claim 8, wherein the reflecting means reflects the feature points preferentially in an area within a predetermined range from the center of the second image.

11. 11. The image processing apparatus according to claim 1, wherein the reflecting means reflects the feature points in the second image so that the feature points are arranged at equal distances from one another.

12. 7. The image processing device according to claim 3, wherein the reflection means increases the number of feature points of the second image that reflect the feature points of the first image as the camera shake in the rotational direction becomes larger, and decreases the number of feature points of the second image that reflect the feature points of the first image as the camera shake in the rotational direction becomes smaller.

13. 13. The image processing device according to claim 1, wherein the detection means detects the motion vector by calculating a correlation value between pixel values ​​within a rectangular area of ​​the reference image centered on the feature point and a rectangular area of ​​the original image, using one of two consecutive images on a time axis as an original image and the other as a reference image.

14. An image processing method executed by an image processing device, comprising: An acquisition step of acquiring a plurality of images having parallax by a plurality of optical systems; An extraction step of extracting feature points from the image; a reflection step of reflecting the feature points extracted from a first image among the plurality of images on the feature points extracted from a second image while taking into account the parallax; a detection step of detecting a motion vector based on the feature points, An image processing method, characterized in that in the detection step, the motion vector of the second image is detected based on feature points that reflect feature points of the first image.

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images