Image processing device, imaging device, image processing method, program, and storage medium

Abstract

To prevent a parallax value to be erroneously calculated when a luminance difference exists between a standard image and a reference image.SOLUTION: An image processing apparatus calculates a parallax value from a first image and a second image. The image processing apparatus includes: image setting means configured to set a prescribed region of the first image as a standard image, and a prescribed region of the second image as a reference image; travel image generation means configured to generate a travel standard image obtained by moving the standard image according to a first travel amount and a travel reference image obtained by moving the reference image according to a second travel amount; correlation value calculation means configured to calculate a correlation value of the travel standard image and the travel reference image; and parallax calculation means configured to calculate the parallax value according to the first travel amount and the second travel amount. The travel image generation means sets one of the first travel amount and the second travel amount as a decimal pixel unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device that calculates parallax from a plurality of images. [Background technology]

[0002] Block matching is a method used to acquire multiple images and calculate 3D information. In this method, two images (hereafter referred to as image A and image B) taken from different viewpoints are first taken. An arbitrary region 1 in image A is set as the base image, and region 2 in image B is set as the reference image. The position of region 2 is then changed to search for region 2 that is most similar to region 1. The distance is then calculated using the principle of triangulation from the positional deviation between region 1 and region 2. Note that the search uses a correlation value that represents the degree of difference (or similarity) between regions in different images to determine which regions are similar. This positional deviation is called parallax, and distance information can be obtained using known techniques such as triangulation.

[0003] Furthermore, conventional block matching methods can only calculate disparity in integer pixel units. To obtain disparity with even greater precision, subpixel estimation methods are generally used. This method calculates disparity in decimal pixel units by fitting correlation values ​​to a function.

[0004] The brightness values ​​of the same object in image A and image B may differ due to factors such as the angle of view and exposure time. Preprocessing to correct this brightness difference is sometimes implemented, but the correction may not be sufficient, or the corrected value may differ due to some other influence. If the subpixel estimation method is used when there is a brightness difference between image A and image B, a correlation value that differs from the ideal state will be calculated, and as a result of incorrect function fitting, an incorrect disparity will be calculated.

[0005] To address the decline in accuracy of disparity calculation, Patent Document 1 discloses a method of performing filter processing on at least one of image A and image B. In this method, after the first calculation of disparity, image A and image B are relatively displaced by an amount corresponding to the disparity. After that, disparity is calculated again. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 4941565 Summary of the Invention [Problem to be solved by the invention]

[0007] In the method disclosed in Patent Document 1, if there is a difference in brightness between image A and image B, there is a risk that the first calculation of disparity will be incorrect and filtering will not be performed correctly, which could result in an error in the calculated disparity.

[0008] Therefore, an object of the present invention is to provide an image processing device that can calculate parallax with high accuracy even when there is a difference in brightness value between image A and image B. [Means for solving the problem]

[0009] An image processing device according to one aspect of the present invention is an image processing device that calculates a disparity value from a first image and a second image, and includes: an image setting means that sets a specific region of the first image as a base image and a specific region of the second image as a reference image; a moved image generation means that generates a moved base image obtained by moving the base image based on a first movement amount and a moved reference image obtained by moving the reference image based on a second movement amount; a correlation value calculation means that calculates a correlation value between the moved base image and the moved reference image; a parallax calculation means that identifies the first movement amount and the second movement amount based on the correlation value and calculates the disparity value based on the identified first movement amount and second movement amount; and a movement amount change means that changes the first movement amount and the second movement amount, wherein the movement amount change means sets either the first movement amount or the second movement amount in decimal pixel units. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image processing device capable of calculating parallax with high accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram of an imaging device including an image processing device according to a first embodiment; [Figure 2] FIG. 1 is an explanatory diagram of a light beam received by an image sensor according to a first embodiment; [Figure 3] Illustration of the subpixel estimation method when there is no deviation in brightness values [Figure 4] Illustration of the subpixel estimation method when there is a difference in brightness values [Figure 5] FIG. 1 is an explanatory diagram of an image processing apparatus according to a first embodiment; [Figure 6] FIG. 1 is an explanatory diagram of an image setting unit according to a first embodiment; [Figure 7] FIG. 1 is an explanatory diagram of an image setting unit according to a first embodiment; [Figure 8] FIG. 10 is an explanatory diagram of an imaging device according to a second embodiment; [Figure 9] FIG. 1 is an explanatory diagram of a movement image generation unit according to a first embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will be described in detail using embodiments and drawings. The present invention is not limited to the contents described in each embodiment. In addition, each embodiment may be combined appropriately.

[0013] First Embodiment (Device configuration) FIG. 1 is a diagram schematically showing the configuration of an imaging device according to an embodiment of the present invention.

[0014] In FIG. 1A, the imaging device 100 includes an image processing device 110 and an imaging unit 120.

[0015] The imaging unit 120 includes an imaging element 121 and an optical system 122. The image processing device 110 can be configured using a logic circuit. Alternatively, the image processing device 110 may be configured with a central processing unit (CPU) and a memory that stores a processing program.

[0016] The optical system 122 is a photographing lens of the imaging device 100, and has the function of forming an image of a subject on the imaging element 121 (on the imaging element). The optical system 122 is composed of a plurality of lens groups (not shown), an aperture (not shown), etc., and has an exit pupil 123 at a predetermined distance from the imaging element 121. Note that in this specification, the z-axis is parallel to the optical axis 130 of the optical system 122. Furthermore, the x-axis and y-axis are perpendicular to each other and to the optical axis.

[0017] The image sensor 121 is configured with a CMOS (complementary metal oxide semiconductor) or a CCD (charge coupled device). An image of a subject formed on the image sensor 121 (on the image sensor) via the optical system 122 is photoelectrically converted by the image sensor 121 to generate an image signal based on the subject image.

[0018] 1(B) is an xy cross-sectional view of the imaging element 121. The imaging element 121 is configured by arranging multiple pixel groups 150 in a 2-row by 2-column array. The pixel group 150 is configured by arranging green pixels 150G1 and 150G2 in the diagonal direction, and a red pixel 150R and a blue pixel 150B in the other two pixels.

[0019] FIG. 1C is a schematic diagram showing the I-I' cross section of the pixel group 150. Each pixel is composed of a light receiving layer 182 and a light guide layer 181. The light receiving layer 182 is provided with two photoelectric conversion units (a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162) for photoelectrically converting received light. The light guide layer 181 is provided with a microlens 183 for efficiently guiding a light beam incident on the pixel to the photoelectric conversion unit, a color filter (not shown) that transmits light of a predetermined wavelength band, wiring (not shown) for image reading and pixel driving, and the like. Each pixel is also provided with wiring (not shown), and each pixel can send an image signal (output signal) to the image processing device 110 via the wiring. 1B and 1C show examples of a photoelectric conversion unit that is divided into two in one pupil division direction (x-axis direction), but depending on the specifications, an image sensor having multiple photoelectric conversion units that are divided into two pupil division directions (x-axis direction and y-axis direction) may be used. The pupil division direction and the number of divisions are arbitrary.

[0020] 2 shows the exit pupil 123 of the optical system 122 as viewed from the intersection (central image height) of the optical axis 130 and the image sensor 121. A first light beam that has passed through a first pupil region 210 and a second light beam that has passed through a second pupil region 220, which are different regions of the exit pupil 123, are incident on the photoelectric conversion unit 161 and the photoelectric conversion unit 162, respectively. The photoelectric conversion unit 161 and the photoelectric conversion unit 162 in each pixel photoelectrically convert the incident light beams to generate image signals corresponding to image A (first image) and image B (second image), respectively. The generated image signals are transmitted to the image processing device 110.

[0021] 2 shows the center of gravity of the first pupil region 210 (first center of gravity position 211) and the center of gravity of the second pupil region 220 (second center of gravity position 221). In this embodiment, the first center of gravity position 211 is decentered (moved) from the center of the exit pupil 123 along a first axis 200. On the other hand, the second center of gravity position 221 is decentered (moved) along the first axis 200 in the opposite direction to the first center of gravity position 211. The direction connecting the first center of gravity position 211 and the second center of gravity position 221 is called the pupil division direction. The distance between the centers of gravity of the first center of gravity position 211 and the second center of gravity position 221 is the base length 230.

[0022] (Explanation of image processing device) Image processing device configuration The image processing device 110 of this embodiment will be described below. The image processing device 110 acquires information on a plurality of images and calculates the parallax between the images.

[0023] First, we will explain the effect of brightness value deviation on the detected disparity. First, we will use Figure 3 to explain general subpixel estimation when there is no brightness value deviation between image A and image B. In the following explanation, we assume that image A and image B have the same grayscale and a disparity of +0.1 pixels. Figure 3(A) is a diagram showing the positional relationship between image A 310, standard image 311, image B 320, and reference image 321 of an object having a line pattern with a dark area near the center, acquired using image capture device 100. Figure 3(B) shows the correlation value between standard image 311 and the reference image when the reference image set with respect to standard image 311 is shifted. Note that the correlation value here is assumed to be the sum of squared differences (SSD), with smaller values ​​indicating higher correlation and larger values ​​indicating lower correlation. When the correlation values ​​S(0), S(1), and S(-1) obtained when the position of the reference image is shifted by 0, +1, and -1 pixels are fitted with a quadratic function, the shape becomes curve 330. The position of the minimum value in the curve 330 is +0.1 pixels, which coincides with the disparity of +0.1 pixels between the A image and the B image.

[0024] Next, a typical subpixel estimation process when there is a difference in brightness between image A and image B will be described with reference to FIG. 4. FIG. 4(A) shows the positional relationship between image A 310, standard image 311, image B 420, and reference image 421 of the same subject as in FIG. 3(A). Here, image B 420 has lower brightness overall than image A 310. FIG. 4(B) shows the correlation value between standard image 311 and the reference image when the reference image set with respect to standard image 311 is moved. When the correlation values ​​S(0), S(1), and S(-1) obtained when the position of the reference image is moved by 0, +1, and -1 pixels are fitted with a quadratic function, the shape becomes curve 430. The minimum value in curve 430 is at +0.3 pixels, which does not match the disparity of +0.1 pixels between image A and image B.

[0025] In this way, if fitting to a function is performed while the correlation value is deviated from the true value in a state where there is a difference in brightness value between image A and image B, the calculated disparity will also contain an error.

[0026] In this case, this error can be reduced by not fitting to a function, but by generating images by shifting the standard image and reference image by decimal pixel units, and then calculating the correlation value between these images.

[0027] 5A is a diagram schematically illustrating the configuration of an image processing device 110 according to an embodiment of the present invention. In FIG. 5A, the image processing device 110 includes an image setting unit 111, a movement image generation unit 112, a correlation value calculation unit 113, a calculation completion determination unit 114, a movement amount change unit 115, and a parallax calculation unit 116.

[0028] 5B is a flowchart showing the operation of the image processing device 110 of this embodiment. When image processing according to this embodiment starts, the process proceeds to step S510.

[0029] In step S510, photography is performed using the imaging device 100, an image set including an image A and an image B is generated and acquired, and the acquired images are stored in the main body memory (not shown).

[0030] The image acquired in step S510 may be subjected to processing to correct imbalance in light intensity caused mainly by vignetting of the optical system 122. Specifically, the light intensity balance can be corrected by correcting the luminance value of the image so that it becomes approximately constant regardless of the angle of view, based on the results of imaging a surface light source with constant luminance previously captured by the imaging device 100. Furthermore, in order to reduce the influence of optical shot noise generated by the imaging element 121, for example, the acquired image may be subjected to filtering processing using a band-pass filter, a low-pass filter, or the like. Alternatively, the image may be reduced in size to reduce calculation costs.

[0031] Step S520 is performed by the image setting unit 111. Fig. 6 is a flowchart showing in detail the processing flow of step S520.

[0032] First, the flow of FIG. 6(A) will be described.

[0033] The method for setting the reference image in step S521A will be described with reference to Figure 7(A). Figure 7(A) shows image A 710A. In step S521A, a partial area including pixel of interest 720 and its neighboring pixels is extracted from image A 710A and set as reference image 711. This means that a specific area of ​​image A 710A has been set as reference image 711.

[0034] The method for setting the reference image in step S522A will be described with reference to FIG. 7(B). FIG. 7(B) shows image B 710B. In step S522A, an area of ​​image B 710B having the same area (image size) as the base image 711 is extracted and set as reference image 712. This results in a specific area of ​​image B 710B being set as reference image 712. For example, a position shifted a predetermined amount from the position of the base image may be set as the reference image. A specific example of the predetermined amount is 0.1 pixels. Alternatively, the same position as the base image may be set as the reference image. In other words, the same coordinates as those of the base image 711 for image A 710A may be set as the coordinates of reference image 712 for image B 710B.

[0035] Step S530 is performed by the moved image generation unit 112. In step S530, the base image and the reference image are moved by the base image movement amount and the reference image movement amount to generate a moved base image and a moved reference image. The generation of the moved base image and the moved reference image will be explained using FIG. 9. The A image 710A, base image 711, and pixel of interest 720 in FIG. 9(A) are the same as those in FIG. 7(A). An image obtained by moving the base image 711 by 0.1 pixels in the x-axis direction is defined as moved base image 911. The B image 710B and reference image 712 in FIG. 9(B) are the same as those in FIG. 7(B). An image obtained by moving the reference image 712 by -0.1 pixels in the x-axis direction is defined as moved reference image 912.

[0036] The shift between the base image and the reference image may be calculated using any known method, such as applying a window function or rotating a phase component in frequency space. The following describes a method for interpolating images.

[0037] Let the luminance value of pixel position (x, y) be I(x, y). In this case, consider the luminance value I(x+Δx, y+Δy) when moved by (Δx, Δy). When generated by linear interpolation, I(x+Δx, y+Δy) can be calculated using Equation 1.

[0038]

number

[0039] In the case of bilinear interpolation and bicubic interpolation, the luminance value I(x+Δx, y+Δy) can be calculated using pixel values ​​of neighboring pixels in the same way as above.

[0040] By performing such calculations for all pixels of the standard image and the reference image, a shifted standard image and a shifted reference image can be generated.

[0041] In this case, the reference image shift amount and the base image shift amount can be any value, and can be moved in two dimensions. It is preferable that the reference image shift amount and the base image shift amount are one pixel or less in each dimension, as this reduces image interpolation errors.

[0042] If the reference image movement amount has the same absolute value as the base image movement amount and the value has an inverted sign, the image correction error caused by the movement will be similar in the base image and the reference image, so step S540 described below can be performed with high accuracy.

[0043] Alternatively, a conventional sub-pixel estimation method may be used to calculate sub-pixel disparity, and then the base image shift amount and the reference image shift amount may be determined. Specifically, the difference between the base image shift amount and the reference image shift amount may be the sub-pixel disparity. In this case, a value close to the true value of disparity can be set, which is efficient.

[0044] Alternatively, the reference image shift amount and the reference image shift amount may be determined from the calculated disparity value of the neighboring pixels. Specifically, the difference between the reference image shift amount and the reference image shift amount may be the sub-pixel disparity of the neighboring pixels. In this case, a value close to the true value of the disparity can be set, which is efficient.

[0045] Step S540 is performed by the correlation value calculation unit 113. In step S540, the correlation value between the movement standard image and the movement reference image is calculated.

[0046] The correlation value may be calculated using any known method as long as it can evaluate the degree of correlation between the standard image 411 and the reference image candidate 412. For example, SSD, SAD, or NCC may be used.

[0047] Step S550 is performed by calculation end determination unit 114. Step S550 determines whether or not the calculation of the correlation value in step S540 is to be ended.

[0048] For example, if the difference between the movement amount of the standard image and the movement amount of the reference image in step S530 is outside a predetermined value range, it may be determined that the predetermined condition has been met and that the calculation of the correlation value should be terminated. Since the range in which parallax occurs is limited depending on the optical system, the predetermined value is set within that range. Specifically, if the range in which parallax occurs is plus or minus 3 pixels, the calculation of the correlation value ends after calculating the correlation value in the range from minus 3 pixels to plus 3 pixels.

[0049] Furthermore, when the correlation value calculated in step S540 is greater (low correlation) than the previously calculated correlation value and the previous correlation value is at its minimum (high correlation), it may be determined that a predetermined condition is satisfied and that the calculation of the correlation value is to be terminated. In other words, when the correlation value calculated this time is lower than the previously calculated correlation value, it may be determined that the calculation of the correlation value is to be terminated. In this case, the calculation can be completed in fewer iterations, which is more efficient.

[0050] Alternatively, it may be determined that the calculation is to be terminated when the correlation value calculated in step S540 is higher than a predetermined value. In this case, the calculation can be terminated in fewer iterations, which is more efficient.

[0051] Step S560 is performed by movement amount change unit 115. In step S560, the reference image movement amount and the reference image movement amount are changed. At this time, the reference image movement amount and the reference image movement amount are changed so that at least one of the reference image movement amount and the reference image movement amount is in decimal pixel units.

[0052] Note that the change amount of the base image movement amount and the change amount of the reference image movement amount may be different values, or may be any value. If the changed reference image movement amount has the same absolute value as the base image movement amount and a value with the opposite sign, the error caused by the movement will be about the same in the base image and the reference image, so step S540 can be performed with high accuracy, improving precision.

[0053] For example, the reference image movement amount and the reference image movement amount may be changed by a predetermined change amount. Specifically, if the predetermined change amount is 0.1 pixel, the first movement amount is -0.1 pixel for the reference image and +0.1 pixel for the reference image, and the second movement amount is -0.2 pixel for the reference image and +0.2 pixel for the reference image. The same applies to the third and subsequent movements.

[0054] Alternatively, the difference between the reference image movement amount and the reference image movement amount when the correlation value calculated in step S540 is highest may be calculated, and the reference image movement amount and the reference image movement amount may be changed so that they approach this value. For example, if the difference between the reference image movement amount and the reference image movement amount when the correlation value is highest is 0.5 pixels, the reference image movement amount and the reference image movement amount may be changed so that the difference between the reference image movement amount and the reference image movement amount becomes 0.4 pixels or 0.6 pixels. In this case, values ​​close to the true value of the parallax can be set as the reference image movement amount and the reference image movement amount, which is efficient.

[0055] Although the above examples show the cases where the reference image shift amount and the reference image shift amount are less than 1.0 pixel or more than -1.0 pixel, they may also be greater than 1.0 pixel or less than -1.0 pixel. For example, if the reference pixel shift amount is 1.6 pixels and the reference image shift amount is -1.7 pixels, the disparity value is 3.3 pixels.

[0056] Step S570 is performed by the parallax calculation unit 116. In step S570, the parallax is calculated from the correlation value between the movement standard image and the movement reference image.

[0057] Specifically, the standard image movement amount and the reference image movement amount when the correlation value is the highest are calculated, and the difference between the standard image movement amount and the reference image movement amount is calculated as the parallax.

[0058] (Variation 1) In the above example, the reference image is defined in advance in step S520. However, the reference image may be set at a position that matches the base image in integer pixel units by performing block matching. In other words, a position corresponding to the base image may be searched for in image B 710B and set as the reference image. The flow for this case is shown in Figure 6(B).

[0059] The method for performing block matching and determining the amount of movement of the reference image in step S522B will be described with reference to Fig. 7C. An area of ​​the B image 710B having the same area (image size) as the base image 711 is extracted and set as a reference image candidate 713.

[0060] Thereafter, the position from which the reference image candidate 713 is extracted is moved on the B image 710B, and the correlation value between the reference image candidate 713 and the standard image 711 at each amount of movement (each position) is calculated. This calculates a correlation value consisting of a correlation value data string corresponding to each amount of movement.

[0061] The amount of movement that maximizes the correlation is calculated from the correlation value data string, and this amount of movement is set as the amount of movement of the reference image.

[0062] At this time, the reference image candidate 713 may be moved in any direction. It may also be moved two-dimensionally. Preferably, it is moved in the same direction as the pupil division direction (the x-axis direction in this embodiment), so that the amount of movement that efficiently increases the correlation can be calculated.

[0063] The correlation value may be calculated using any known method that can evaluate the degree of correlation between the standard image 711 and the reference image candidate 713. For example, the sum of squared differences (SSD) or the sum of absolute differences (SAD) may be used. Normalized cross correlation (NCC) may also be used.

[0064] In step S523B, a position shifted from the position of the base image by the amount of movement calculated in step S522B is set as the reference image.

[0065] In this example, a position close to the true value of the parallax can be set as the reference image, so step S540 can be performed with high accuracy.

[0066] (Variation 2) Alternatively, in step S520, the reference image may be set from the calculated parallax values ​​of neighboring pixels. The flow in this case is shown in FIG.

[0067] In step S522C, the parallax value of the neighboring pixel is searched for, and the parallax value is rounded off and set as the amount of movement of the reference image.

[0068] In step S523C, a position shifted from the position of the base image by the amount of movement calculated in step S522C is set as the reference image.

[0069] In this example, a position close to the true value of the parallax can be set as the reference image without performing the block matching process described in the first modification, so that step S540 can be performed efficiently and with high accuracy.

[0070] Fitting to a function is based on the assumption that brightness values ​​change linearly, and calculates disparity based on correlation values ​​calculated using a partial combination of the base image and reference image. Therefore, if there is a discrepancy in brightness values, the error in the calculated disparity will be large. In contrast, this method does not fit to a function, but instead moves the base image and reference image to generate a moved base image and a moved reference image, and then calculates the correlation value. Then, disparity is calculated based on the correlation value with the highest correlation among multiple calculated correlation values, so disparity can be calculated with high accuracy even if there is a discrepancy in brightness values.

[0071] <Second embodiment> The second embodiment of the present invention will be described in detail below with reference to the drawings. Note that the components described in this embodiment are merely examples, and the scope of the present invention is not limited to the components described in this embodiment.

[0072] (Device configuration) Fig. 8 is a diagram schematically illustrating the configuration of an image processing device according to an embodiment of the present invention. In Fig. 8, the same components as those described in Fig. 1 are assigned the same numbers as in Fig. 1, and descriptions thereof will be omitted.

[0073] In FIG. 8, an imaging device 800 includes an image processing device 110 and an imaging unit 820 .

[0074] Image capture unit configuration The imaging unit 820 includes two imaging elements 821 and 822 and two optical systems 823 and 824. The optical systems 823 and 824 are photographing lenses of the imaging device 800, and have the function of forming an image of a subject on the imaging elements 821 or 822. The optical systems 823 and 824 are made up of a plurality of lens groups (not shown), an aperture (not shown), etc., and have exit pupils 825 and 826 at positions separated by a predetermined distance from the imaging elements 821 and 822. In this case, the optical axes of the optical systems 823 and 824 are 841 and 842, respectively.

[0075] By calibrating parameters such as the positional relationship of the optical systems in advance, it is possible to accurately calculate the parallax between images. In addition, by correcting the lens distortion in each optical system, it is also possible to accurately calculate the parallax between images.

[0076] In this embodiment, there are two optical systems that acquire image A and image B, which have parallax according to distance, but it may also be configured with a stereo camera consisting of three or more optical systems and corresponding image sensors.

[0077] The imaging device of this embodiment has improved design freedom for the base line length, enabling improved ranging resolution. In this case, even if there are differences in sensor sensitivity or exposure time between cameras, the parallax can be calculated with high accuracy.

[0078] The present invention encompasses not only a distance measuring device but also a computer program. The computer program of this embodiment causes a computer to execute predetermined processes to calculate distance or parallax. The program of this embodiment is installed in a computer of a distance measuring device or an imaging device such as a digital camera equipped with the same. The installed program is executed by the computer to realize the above functions, enabling high-speed and high-precision calculation of parallax.

[0079] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0080] 100 Imaging device 110 Image processing device 120 Imaging unit 121 Image sensor 122 Optical system 123 Exit pupil 130 Optical axis

Claims

1. An image processing device that calculates a parallax value from a first image and a second image, an image setting means for setting a specific region of the first image as a base image and a specific region of the second image as a reference image; a moved image generating means for generating a moved reference image by moving the reference image based on a first amount of movement and a moved reference image by moving the reference image based on a second amount of movement; a correlation value calculation means for calculating a correlation value between the movement standard image and the movement reference image; a parallax calculation means for specifying the first movement amount and the second movement amount based on the correlation value, and calculating the parallax value based on the specified first movement amount and the specified second movement amount; a movement amount changing means for changing the first movement amount and the second movement amount; and the movement amount changing means sets either the first movement amount or the second movement amount in decimal pixel units; 1. An image processing device comprising:

2. the disparity calculation means calculates, as the disparity value, a difference between the first movement amount and the second movement amount for which the correlation value has been calculated; 2. The image processing device according to claim 1, wherein:

3. the movement image generating means generates the movement standard image and the movement reference image by assuming that the absolute values ​​of the first movement amount and the second movement amount are the same and that the movement directions are opposite to each other.

2. The image processing device according to claim 1, wherein:

4. the image setting means searches for a position of the reference image corresponding to the base image on the second image in integer pixel units and sets the position as the reference image; 2. The image processing device according to claim 1, wherein:

5. the moved image generating means estimates the position of the reference image corresponding to the base image as the disparity value by fitting the correlation value to a function, and generates the moved base image and the moved reference image by setting the estimated disparity value as the disparity value of the first movement amount and the second movement amount.

2. The image processing device according to claim 1, wherein:

6. the image setting means sets a specific region of the second image as the reference image based on the calculated parallax value of the neighboring pixels; 2. The image processing device according to claim 1, wherein:

7. the movement image generating means generates the movement standard image and the movement reference image based on the calculated parallax values ​​of the neighboring pixels.

2. The image processing device according to claim 1, wherein:

8. The moving image generating means generating the moved reference image by using pixel values ​​of neighboring pixels of each pixel of the reference image when the first movement amount is a decimal pixel; generating the moved reference image by using pixel values ​​of neighboring pixels of each pixel of the reference image when the second movement amount is a decimal pixel; 2. The image processing device according to claim 1, wherein:

9. Further comprising a calculation completion determination means, The calculation completion determination means executes the parallax calculation means when a predetermined condition is satisfied.

9. The image processing device according to claim 1, wherein the image processing device is a computer.

10. the movement amount changing means changes the first movement amount and the second movement amount based on a predetermined change amount; 10. The image processing device according to claim 9,

11. the movement amount changing means changes the first movement amount and the second movement amount so that the first movement amount and the second movement amount become values ​​close to a difference between the first movement amount and the second movement amount that calculates the correlation value with the highest correlation among the at least two correlation values; 10. The image processing device according to claim 9,

12. the calculation completion determination means determines whether the predetermined condition is that the difference between the first movement amount and the second movement amount is outside a predetermined value range; 10. The image processing device according to claim 9,

13. The calculation completion determination means determines whether the predetermined condition is satisfied when the correlation value calculated this time is lower than the correlation value calculated up to the previous time.

10. The image processing device according to claim 9,

14. An imaging device comprising an imaging means and the image processing device according to claim 9.

15. The imaging means includes an optical system and an imaging element. the optical system forms an image of a subject on the imaging element; the imaging element includes a plurality of first photoelectric conversion units for generating the first image and a plurality of second photoelectric conversion units for generating the second image; 15. The imaging device according to claim 14.

16. The imaging means a first imaging element; a first optical system that forms an image of a subject on the first image sensor; a second imaging element; a second optical system that forms an image of a subject on the second image sensor; Equipped with acquiring the first image with the first imaging element and acquiring the second image with the second imaging element; 15. The imaging device according to claim 14.

17. An image processing method in which a CPU calculates a parallax value from a first image and a second image, an image setting step of setting a specific region of the first image as a base image and a specific region of the second image as a reference image; a moved image generating step of generating a moved reference image by moving the reference image based on a first amount of movement and a moved reference image by moving the reference image based on a second amount of movement; a correlation value calculation step of calculating a correlation value between the movement standard image and the movement reference image; a parallax calculation step of specifying the first movement amount and the second movement amount based on the correlation value, and calculating a parallax value based on the specified first movement amount and the specified second movement amount, a movement amount changing step of changing the first movement amount and the second movement amount; the movement amount changing step sets either the first movement amount or the second movement amount in decimal pixel units; An image processing method comprising:

18. A program for causing a computer to execute each step of the image processing method according to claim 17.

19. A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 17.

Images