Image processing device, image processing method, and program

Abstract

To provide an image processing device, method and program which reduce an error of parallax calculation using a plurality of images photographed from different view points.SOLUTION: An image processing device for calculating a parallax from a first image and a second image has: a cross correlation calculation unit for calculating a cross correlation value between a first standard image set on the first image and a first reference image set on the second image; a self-correlation calculation unit for calculating a self-correlation value between a second standard image and a second reference image which are set on any one of the first image and the second image; and a parallax calculation unit for calculating a parallax amount between the first image and the second image using the cross correlation value calculated by the cross correlation calculation unit and the self-correlation value calculated by the self-correlation calculation unit. The parallax calculation unit corrects the cross correlation value using the self-correlation value.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that calculates parallax from a plurality of images.

Background Art

[0002] There is a method for calculating three-dimensional information from a plurality of images. For example, as a method for calculating three-dimensional information, there is a sub-pixel estimation method. This method calculates the sub-pixel parallax by fitting a predetermined function according to the calculation method of the cross-correlation value to the cross-correlation with the lowest difference and the cross-correlation values in its vicinity.

[0003] Patent Document 1 discloses that after estimating the sub-pixel parallax by cross-correlation, autocorrelation is used to correct errors caused by sudden pixel value changes in the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in Patent Document 1, there are cases where errors caused particularly by the asymmetry of the edge texture cannot be correctly corrected.

[0006] The present case aims to reduce errors in parallax calculation using a plurality of images taken from different viewpoints, such as errors caused by the asymmetry of the above-mentioned edge texture.

Means for Solving the Problems

[0007] To achieve the above objective, the present invention provides an image processing apparatus for calculating disparity from a first image and a second image, comprising: cross-correlation calculation means for calculating a cross-correlation value between a first reference image set on the first image and a first reference image set on the second image; autocorrelation calculation means for calculating an autocorrelation value between a second reference image set for either the first image or the second image and a second reference image; and disparity calculation means for calculating the amount of disparity between the first image and the second image using the cross-correlation value calculated by the cross-correlation calculation means and the autocorrelation value calculated by the autocorrelation calculation means, wherein the disparity calculation means corrects the cross-correlation value using the autocorrelation value. The cross-correlation calculation means calculates at least two cross-correlation values ​​by changing the position of the first reference image; the autocorrelation calculation means calculates at least two autocorrelation values ​​by moving the position of the second reference image by at least +1 pixels and -1 pixels in an arbitrary direction; the disparity calculation means calculates a corrected cross-correlation value by correcting the cross-correlation value using the vicinity of the minimum value of the autocorrelation value; calculates the position of the first reference image with the highest correlation as the disparity amount by interpolating the corrected cross-correlation value or the cross-correlation value with the corrected cross-correlation value; and the disparity calculation means calculates the corrected cross-correlation value using the ratio of two autocorrelation values ​​with different amounts of movement of the second reference image for at least one cross-correlation value. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the error in calculating parallax using multiple images taken from different viewpoints. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating an imaging device equipped with an image processing device according to the first embodiment. [Figure 2] This diagram illustrates the light beam received by the image sensor of the first embodiment. [Figure 3] This figure illustrates the image processing apparatus of the first embodiment. [Figure 4] This figure illustrates the image processing apparatus of the first embodiment. [Figure 5] This diagram illustrates the relationship between a reference image and its autocorrelation and crosscorrelation. [Figure 6] This is a diagram illustrating the cross-correlation correction unit of the first embodiment. [Figure 7] This figure illustrates the image processing apparatus of the second embodiment. [Figure 8] This figure illustrates the image processing apparatus of the third embodiment. [Figure 9] This is a diagram illustrating the imaging device of the fourth embodiment.

Best Mode for Carrying Out the Invention

[0010] The present invention will be described in detail with reference to embodiments and drawings. The present invention is not limited to the contents described in each embodiment. Also, each embodiment may be combined as appropriate.

[0011] <First Embodiment> (Configuration of Imaging Device) FIG. 1 is a diagram schematically showing the configuration of an imaging device according to an embodiment of the present invention.

[0012] In FIG. 1(A), the imaging device 100 includes an image processing device 110, an imaging unit 120, and a distance calculation unit 130.

[0013] The imaging unit 120 includes an imaging element 121 and an optical system 122.

[0014] The optical system 122 is a photographing lens of the imaging device 100 and has a function of forming an image of a subject on the imaging element 121. The optical system 122 is composed of a plurality of lens groups (not shown) and an aperture (not shown), etc., and has an exit pupil 123 at a position separated from the imaging element 121 by a predetermined distance. In this specification, the z-axis is parallel to the optical axis 140 of the optical system 122. Further, the x-axis and the y-axis are perpendicular to each other and are axes perpendicular to the optical axis.

[0015] The imaging element 121 is composed of a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device). The subject image formed on the imaging element 121 via the optical system 122 is photoelectrically converted by the imaging element 121 to generate an image signal based on the subject image.

[0016] FIG. 1(B) is a cross-sectional view of the imaging element 121 in the xy plane. The imaging element 121 is configured by arranging a plurality of pixel groups 150 of 2 rows × 2 columns. In the pixel group 150, green pixels 150G1 and 150G2 are arranged in the diagonal direction, and red pixels 150R and blue pixels 150B are arranged in the other two pixels.

[0017] FIG. 1(C) is a diagram schematically showing an I-I' cross section of the pixel group 150. Each pixel is composed of a light-receiving layer 182 and a light-guiding layer 181. In the light-receiving layer 182, two photoelectric conversion units (a first photoelectric conversion unit 161 and a second photoelectric conversion unit 162) for photoelectrically converting received light are arranged. In the light-guiding layer 181, a microlens 183 for efficiently guiding the light beam incident on the pixel to the photoelectric conversion unit, a color filter (not shown) for passing light in a predetermined wavelength band, wirings for image reading and pixel driving (not shown), etc. are arranged. Also, each pixel is provided with a wiring (not shown), and each pixel can send an image signal (output signal) to the image processing device 110 via the wiring. FIGS. 1(B) and (C) are examples of the photoelectric conversion units divided into two in one pupil division direction (x-axis direction), but according to the specifications, an imaging device having photoelectric conversion units divided in two pupil division directions (x-axis direction and y-axis direction) is used. The pupil division direction and the number of divisions are arbitrary.

[0018] FIG. 2 shows the exit pupil 123 of the optical system 122 as seen from the intersection point (center image height) of the optical axis 140 and the imaging device 121. The first light beam that has passed through the first pupil region 210, which is a different region of the exit pupil 123, and the second light beam that has passed through the second pupil region 220 are incident on the photoelectric conversion unit 161 and the photoelectric conversion unit 162, respectively. The photoelectric conversion units 161 and 162 in each pixel can generate image signals corresponding to an A image (first image) and a B image (second image), respectively, by photoelectrically converting the incident light beams. The generated image signals are transmitted to the image processing device 110.

[0019] Figure 2 shows the center of gravity of the first pupil region 210 (first center of gravity 211) and the center of gravity of the second pupil region 220 (second center of gravity 221). In this embodiment, the first center of gravity 211 is eccentric (moves) along the first axis 200 from the center of the exit pupil 123. On the other hand, the second center of gravity 221 is eccentric (moves) along the first axis 200 in the opposite direction to the first center of gravity 211. The direction connecting the first center of gravity 211 and the second center of gravity 221 is called the pupil division direction. The distance between the centers of gravity of the first center of gravity 211 and the second center of gravity 221 is the baseline length 230.

[0020] Next, the image processing apparatus 110 shown in Figure 1(A) will be described with reference to Figure 3. Figure 3 is a schematic diagram showing the configuration of the image processing apparatus 110 according to an embodiment of the present invention. In Figure 3, the image processing apparatus 110 includes a cross-correlation calculation unit 111, an autocorrelation calculation unit 112, a cross-correlation correction unit 113, and a disparity calculation unit 114.

[0021] The image processing device 110 can be configured using logic circuits. Alternatively, the image processing device 110 and the distance calculation unit 130 may be configured as a control unit including a central processing unit (CPU) and a memory for storing calculation programs.

[0022] The image processing device 110 generates image A and image B based on the received image signal. The image processing device 110 calculates a disparity value using image A and image B through a disparity calculation process and stores the calculated disparity value in the main memory (not shown). The image processing device 110 also stores the image obtained by adding image A and image B as image information in the main memory, which can be used in subsequent processing. The image processing device 110 can also store image A and image B themselves in the main memory.

[0023] Return to the explanation of Figure 1(A).

[0024] The distance calculation unit 130 can be configured using logic circuits. Alternatively, the image processing device 110 and the distance calculation unit 130 may be configured with a central processing unit (CPU) and a memory for storing calculation programs.

[0025] The distance calculation unit 130 calculates the distance to the subject based on the received parallax information. Image A and Image B are shifted in the same direction as the pupil division direction (in this embodiment, the x-axis direction) due to defocusing. The relative positional shift between these images, i.e., the amount of parallax between Image A and Image B, corresponds to the amount of defocusing. Therefore, the conversion from the amount of parallax to the amount of defocusing can be performed using a geometric relationship using the baseline length. Furthermore, the conversion from the amount of defocusing to the subject distance can be performed using the imaging relationship of the optical system 122. Alternatively, the amount of parallax may be converted to the amount of defocusing or the subject distance by multiplying the amount of parallax by a predetermined conversion coefficient. In this way, the distance calculation unit 130 can generate distance information for the pixel of interest.

[0026] (Explanation of the processing performed by the image processing device) The processing of the image processing device 110 in this embodiment will now be described. Figure 3(B) is a flowchart showing an example of the operation of the image processing device 110 in this embodiment. The processing of this image processing device 110 is achieved by loading the software recorded in the non-volatile memory into the working memory and executing it on the control unit (CPU, GPU, etc.). This processing is triggered when an image signal is generated by the imaging unit 120.

[0027] In step S310, the control unit generates an image set including image A and image B from the image signal acquired from the imaging unit 120. The generated image set is stored in the main unit memory (not shown).

[0028] In step S311, the cross-correlation calculation unit 111 calculates the cross-correlation. For example, the cross-correlation calculation unit 111 sets an image of a partial region containing pixels (pixels of interest) for distance calculation on image A as the reference image, and sets a reference image on image B. Then, the cross-correlation calculation unit 111 calculates the cross-correlation value between the reference image and the reference image while moving the position of the reference image in a predetermined direction. The method of calculating the cross-correlation value by the cross-correlation calculation unit 111 in this embodiment will be described in detail below.

[0029] Figures 4(A), (B), and (C) are diagrams illustrating the positional relationship between the reference image and the reference image set in step S311. Figure 4(A) shows image A 410A, Figure 4(B) shows image B 410B, and Figure 4(C) shows image A 410A. In step S311, the cross-correlation calculation unit 111 calculates the cross-correlation value between image A 410A and image B 410B.

[0030] Specifically, the cross-correlation calculation unit 111 first sets a sub-region on image A 410A that includes the pixel of interest 420 and its neighboring pixels as the reference image 411. Next, the cross-correlation calculation unit 111 sets a region on image B 410B with the same area (image size) as the reference image 411 as the reference image 412. Subsequently, the cross-correlation calculation unit 111 moves (changes) the region designated as the reference image 412 on image B 410B, and calculates the cross-correlation value between the reference image 412 and the reference image 411 each time it is moved. As a result, the cross-correlation calculation unit 111 generates a correlation value data sequence from the cross-correlation values ​​corresponding to each movement amount (each region). At this time, the direction in which the reference image 412 is moved can be any direction. In this embodiment, the direction in which the reference image 412 is moved is called the disparity search direction. In particular, by setting the disparity search direction and the pupil division direction (x-axis direction) described above to the same direction, the calculations of the distance calculation unit 130 described above can be simplified. In this embodiment, the parallax search direction is the x-axis direction.

[0031] The cross-correlation value can be calculated using methods such as the sum of squared differences (SSD), the sum of absolute differences (SAD), or normalized cross-correlation (NCC). In this embodiment, the method using SSD will be described. However, the same concept can be used even if a different method is used. In this embodiment, SSD evaluates the degree of difference between the reference image 411 and the reference image 412, and a lower cross-correlation value indicates a higher degree of correlation.

[0032] Next, in step S312, the autocorrelation calculation unit 112 continues to set the region used as the reference image in step S311 as the reference image, and also sets the reference image within image A. The autocorrelation calculation unit 112 moves the position of the reference image within image A in a predetermined direction and calculates the correlation value (autocorrelation value) between at least two reference images and the reference image. Here, the amount of movement of the reference image is moved by 1 pixel or more in the -x direction and 1 pixel or more in the +x direction, and the autocorrelation value is calculated for each. In particular, the autocorrelation value can be calculated with greater accuracy by calculating the correlation value when the amount of movement is moved by 1 pixel in the -x direction and when it is moved by 1 pixel in the +x direction. In this embodiment, the autocorrelation calculation unit 112 calculates the correlation value when the amount of movement is moved by 1 pixel in the -x direction and when it is moved by 1 pixel in the +x direction. Note that the direction of movement of the reference image can be any direction, but if it is in the same direction as the disparity search direction of the cross-correlation value, the correction described later can be performed with greater accuracy.

[0033] Specifically, the autocorrelation calculation unit 112 first sets a reference image 411 in the same way as in step S311. Next, as shown in Figure 4(C), the autocorrelation calculation unit 112 sets a region on image A 410A with the same area (image size) as the reference image 411 as the reference image 413. Then, the autocorrelation calculation unit 112 moves (changes) the region designated as the reference image 413 on image A 410A in the disparity search direction, and calculates the correlation value between the reference image 413 and the reference image 411 each time it is moved. As a result, the autocorrelation calculation unit 112 generates a correlation value data column from the cross-correlation values ​​corresponding to each movement amount (each region).

[0034] The method for calculating the autocorrelation value may be any known method. However, if the similarity is calculated using the cross-correlation value, it is preferable to calculate the similarity using the autocorrelation value in the same way, and if the difference is calculated using the cross-correlation value, it is preferable to calculate the difference using the autocorrelation value in the same way. The autocorrelation calculation unit 112 of this embodiment uses SSD as the method for calculating the autocorrelation value, similar to the cross-correlation calculation unit 111.

[0035] Next, in step S313, the cross-correlation correction unit 113 calculates a corrected correlation value. In this embodiment, the cross-correlation correction unit 113 corrects the cross-correlation value using the ratio of two or more autocorrelation values ​​obtained in step S312, and calculates a corrected cross-correlation value. The method for correcting the cross-correlation value will be described below.

[0036] First, we will explain why changes in pixel values ​​at the edges of the reference image cause errors in the disparity calculation results. In the following explanation, we will assume that images A and B are images with the same grayscale and have a disparity of +0.1 pixels. Figures 5(A) to (E) illustrate the reasons for the occurrence of errors.

[0037] Figure 5(A) shows the positional relationship between image A 501, reference image 502, and reference image 503. Here, image A 501 is an image (subject) having a line pattern in which light and dark regions appear alternately. Reference image 502 contains the image edges 504 and 505 (boundaries) of image A 501 where the light and dark regions switch, within itself.

[0038] First, let's explain the case where there is no asymmetry in the edge texture (where the pixel value change at the edge of the reference image is equal in both the + and - directions along the same axis).

[0039] Figure 5(B) shows the autocorrelation value calculated by moving the set reference image relative to the reference image 502 and performing an autocorrelation calculation process between the reference image 502 and the reference image.

[0040] The autocorrelation values ​​C(0), C(+1), and C(-1) are the autocorrelation values ​​when the reference image is moved by 0, +1, and -1 pixels in the x-axis direction, respectively. Hereafter, C(x) is the autocorrelation value corresponding to the amount of movement (in the x-axis direction) from 0, where the pixel position where the reference image and the reference image are in the same position is defined as 0. Here, when the amount of movement is 0 pixels, the two images coincide, and the cross-correlation value C(0) is 0. When the reference image is moved by +1 or -1 pixels in the x-axis direction, a difference (different pixel values) occurs between the reference image 502 and the reference image due to image edges 504 and 505. Therefore, the autocorrelation values ​​C(+1) and C(-1) are greater than the autocorrelation value C(0). If there is no asymmetry in the edge texture, the absolute values ​​of the movement of the reference image corresponding to the autocorrelation value C(+1) and the autocorrelation value C(-1) are the same, and therefore the difference between images caused by image edges 504 and 505 on the line pattern is also the same. Therefore, the autocorrelation value C(1) and the autocorrelation value C(-1) are the same value (there is no asymmetry in the autocorrelation value). When this autocorrelation value is interpolated using a quadratic function, its shape is curve 510C. The autocorrelation curve showing the continuous change in the autocorrelation value coincides with curve 510C.

[0041] Figure 5(D) shows the cross-correlation values ​​calculated by performing a cross-correlation calculation process between the reference image 502 and the reference image set for the reference image 502. Since there is a parallax of +0.1 pixels between image A and image B, the cross-correlation curve showing the continuous change in the cross-correlation value is the shape of curve 510C shifted by +0.1 pixels, and coincides with the quadratic function curve 510S. Furthermore, the cross-correlation values ​​S(0), S(1), and S(-1) when the position of the reference image is moved by 0, +1, and -1 pixels in the x-axis direction are all values ​​on curve 510S. Hereafter, S(x) is the cross-correlation value corresponding to the amount of movement (in the x-axis direction) from 0, when the integer pixel position of the reference image where the cross-correlation value is minimized is set to 0.

[0042] When there is no asymmetry in the edge texture, the autocorrelation value is also asymmetry, and the cross-correlation curve becomes a quadratic function. In other words, the fitting function is symmetrical with respect to the boundary axis, like a quadratic function, with respect to the boundary axis. The image processing device 110 can accurately calculate the disparity by moving the curve 510C, which is a quadratic function calculated from the autocorrelation curve, and estimating the position to fit to the cross-correlation value.

[0043] On the other hand, we will now explain the case where there is asymmetry in the edge texture (where the amount of change in pixel values ​​at the edge of the reference image differs between the positive direction along the same axis and the negative direction along the axis).

[0044] In reference image 503, the image edge 504 and the right edge of reference image 503 overlap. Figure 5(C) shows the autocorrelation value calculated by moving the set reference image relative to reference image 503 and performing autocorrelation processing on reference image 503 and the reference image. When the amount of movement is 0, the two images coincide, and the autocorrelation value C(0) is 0. When the amount of movement is +1 pixel in the x-axis direction, the autocorrelation value C(+1) is greater than the autocorrelation value C(0), similar to the case of reference image 502. In contrast, when the amount of movement is -1 pixel in the x-axis direction, the difference between the reference image and the reference image arises only from the image edge 505, so the value of the autocorrelation value C(-1) from the autocorrelation value C(0) is smaller than C(+1). Therefore, in this case, the autocorrelation value is asymmetrical between the positive and negative sides of the amount of movement of the reference image. In this case, the autocorrelation curve is curve 511C on the positive side of the amount of movement, and curve 512C on the negative side of the amount of movement.

[0045] Figure 5(E) shows the cross-correlation values ​​calculated by performing a cross-correlation calculation process between the reference image 503 and the reference image set for the reference image 503. Since there is a parallax of +0.1 pixels between image A and image B, the curves 511S and 512S obtained by interpolating the cross-correlation values ​​with a quadratic function have shapes that are the same as curves 511S and 512S shifted by +0.1 pixels. Furthermore, the cross-correlation values ​​S(0) and S(-1) when the position of the reference image is moved by 0 and -1 pixels in the x-axis direction are values ​​on the cross-correlation curve 512S, and the cross-correlation value S(1) when the position of the reference image is moved by +1 pixel is a value on the cross-correlation curve 511S. The cross-correlation value curve becomes curve 511S when the amount of movement is positive from the true parallax value of +0.1 pixels, and curve 512S when the amount of movement is negative.

[0046] When there is asymmetry in the edge texture as described above, the autocorrelation value is also asymmetry, and the cross-correlation curve is no longer a function symmetric with respect to the boundary axis. In such cases, if the fitting function is a function symmetric with respect to the y-axis, such as a quadratic function, the error in calculating the disparity will be large. Thus, when the fitting function differs from the conventionally assumed known shape and is asymmetric with respect to the true disparity position as the boundary, this state is called asymmetry in the fitting function. When there is asymmetry in the fitting function, moving the quadratic curve 511C or 512C, which can be calculated from the autocorrelation curve, and estimating the position to fit to the cross-correlation value existing on a different correlation curve will be a factor that causes a large error in calculating the disparity.

[0047] As mentioned above, in this explanation, the autocorrelation and cross-correlation values ​​are calculated using SSD, so the fitting function assumed in conventional methods is a quadratic function. However, when SAD is used, the fitting function is a linear function of a contour line. Thus, the fitting function differs depending on the method used to calculate the autocorrelation and correlation values.

[0048] The above explains why the asymmetry of the edge texture causes errors in the parallax calculation results.

[0049] Similarly, errors can occur if the disparity between image A and image B is not 0.1 pixels. Furthermore, errors can also occur when subpixel estimation is performed using only cross-correlation values, such as in the well-known parabolic fitting method, for the same reasons.

[0050] Next, we will explain why errors in the calculated parallax cannot be accurately corrected when the asymmetry of the fitting function is not considered. For example, consider the case where correction is applied to the cross-correlation values ​​S(1) and S(-1), but not to S(0). In the example in Figure 5(E), the only quadratic function that passes through the point where the cross-correlation is 0 between S(0) and +0.1 pixels is the curve 512S. Therefore, if the correction is applied so that the corrected correlation value lies on the curve 512S, it can be judged that the correction is accurate. On the other hand, S(-1) originally lies on the curve 512S, so no correction is necessary. On the other hand, S(1) lies on the curve 511S, so a correction is necessary. When the asymmetry of the fitting function is not considered, the correction is performed by using a correction amount γ calculated from the autocorrelation value, subtracting γ from S(1) and adding γ to S(-1). This correction amount γ is the slope when C(-1) and C(1) are interpolated with a straight line. In other words, in the case of Figure 5(E), γ has a correction amount that is not 0, resulting in an inappropriate correction being applied to S(-1), which does not require correction.

[0051] The above explains why errors cannot be correctly corrected when the asymmetry of the fitting function is not taken into consideration. The problem in this case is that it does not take into account that the correction amount has different values ​​on the positive and negative sides of the disparity position (asymmetrical).

[0052] However, in the process of step S313, a corrected cross-correlation value (corrected cross-correlation value) is calculated, taking into account the asymmetry of the fitting function. By using the corrected cross-correlation value when calculating the disparity in step S314, which will be described later, the above error can be reduced.

[0053] As shown in Figure 5, the larger the autocorrelation value on the + side of the disparity search direction, the larger the cross-correlation value on the + side of the disparity search direction relative to the true subpixel disparity. Also, the larger the autocorrelation value on the - side of the disparity search direction, the larger the cross-correlation value on the - side of the disparity search direction relative to the true subpixel disparity. Therefore, the larger the autocorrelation value, the smaller the cross-correlation value for the same disparity search direction relative to the true subpixel disparity, thereby correcting the asymmetry of the fitting function included in the cross-correlation value.

[0054] For example, Figure 7(A) shows the cross-correlation value in Figure 5(E) after correction. In Figure 7(A), S(1), which is located on curve 511S, is corrected to move onto curve S512. In this way, the above correction can be made by correcting the cross-correlation value used for disparity calculation so that it lies on the same quadratic function. Any value can be used as the cross-correlation value to which this correction is applied, but here we will explain the case where S(0) is used. Specifically, when using S(0), the corrected cross-correlation value S' can be calculated using Equation 1 or Equation 2.

[0055]

number

[0056]

number

[0057] In equations 1 and 2, S'(x) is the corrected cross-correlation value corresponding to the amount of movement x from 0, where 0 is the integer pixel position of the reference image where the cross-correlation value is minimized. Here, equation 1 is a correction formula that is particularly effective when the true subpixel disparity is estimated to be located on the + side of the disparity search direction from the integer pixel position of the reference image where the cross-correlation value is minimized. Equation 2 is a correction formula that is particularly effective when the true subpixel disparity is estimated to be located on the + side of the disparity search direction from the integer pixel position of the reference image where the cross-correlation value is minimized.

[0058] The choice between formula 1 and formula 2 can be determined by estimating the true subpixel disparity position. In this case, known methods may be used. For example, one could compare the values ​​of S(1) and S(-1) and assume that the true subpixel disparity position lies on the side with the smaller value.

[0059] Alternatively, the asymmetry of the fitting function may be considered during this estimation, and the cross-correlation values ​​may be corrected. For example, the values ​​of S(1) / C(1) and S(-1) / C(-1) may be compared, and the true subpixel disparity position may be assumed to be on the side with the smaller value.

[0060] Alternatively, one may choose not to perform the process of estimating the true subpixel disparity position and instead always use a consistent calculation method.

[0061] Here, equations 1 and 2 do not limit the correction method; any method that corrects the cross-correlation value while considering the asymmetry of the fitting function is acceptable.

[0062] Furthermore, even when using different calculation methods for autocorrelation and cross-correlation, the same method as described above can be applied.

[0063] Furthermore, while the above describes applying a correction to a single point of the cross-correlation value, it is also acceptable to apply the correction to each point of the cross-correlation value used for sub-pixel estimation. For example, Figure 7(B) shows the cross-correlation value in Figure 5(E) after applying the correction to each point of the cross-correlation value. In this case, an arbitrary quadratic function is set, and the correction is applied so that all the cross-correlation values ​​used for disparity calculation lie on that function. If the slope of this arbitrary quadratic function is set to 1, the cross-correlation value can be corrected using the ratio of the cross-correlation value to the autocorrelation value. Herein, the above explanation does not limit the correction method; any method that corrects the cross-correlation value while considering the asymmetry of the fitting function is acceptable.

[0064] This concludes the explanation of step S313. Let's return to the explanation of Figure 3(B).

[0065] Next, in step S314, the disparity calculation unit 114 calculates the amount of disparity of the image set including image A and image B using any known method. The method used may be either a method using cross-correlation values ​​or a method using cross-correlation values ​​and autocorrelation values. In this embodiment, corrected cross-correlation values ​​calculated by the cross-correlation correction unit 113 are used instead of cross-correlation values. For example, conventionally, there is a method of calculating the subpixel disparity amount d as shown in equation 3. In this embodiment, the subpixel disparity amount d is calculated using equation 4, which replaces the cross-correlation values ​​in equation 3 with corrected cross-correlation values.

[0066]

number

[0067]

number

[0068] Equation 3 calculates the position of the minimum value when S(0) and S(1) lie on the same quadratic function of slope C(1). As mentioned above, due to the asymmetry of the fitting function, S(0) and S(1) do not necessarily lie on the same quadratic function, so the amount of parallax is not properly corrected. On the other hand, in Equation 4, a correction is applied so that S'(0) lies on the same quadratic function as S(1), so it is possible to calculate the position of the minimum value with less error.

[0069] Note that Equation 4 does not limit the correction method; any method that utilizes the cross-correlation value corrected to account for the asymmetry of the fitting function is acceptable.

[0070] In this embodiment, the autocorrelation calculation unit calculates the autocorrelation value using image A, but it may also calculate the autocorrelation value using image B. When calculating the autocorrelation value using image B, it is desirable to shift the position where the pixel of interest 420 is set using the amount of movement calculated by the disparity calculation unit.

[0071] The cross-correlation correction unit of this embodiment uses only the autocorrelation values ​​of +1 pixel and -1 pixel movement amounts in the reference image. Therefore, calculating the autocorrelation value only for these movement amounts is desirable in order to reduce the amount of computation. In addition, although the operation flow of the image processing apparatus 110 of this embodiment has been described as processing sequentially from step S311 to S314, step S312 may be performed before step S311.

[0072] The image processing apparatus 110 of this embodiment performs correction on all pixels, but a unit for determining whether or not to perform correction may be added, and correction may be performed only on the necessary pixels. In this case, the unit may use known methods to determine the reliability of the autocorrelation value and cross-correlation value, and perform correction only if the reliability is high. Alternatively, the degree of asymmetry of the fitting function may be determined from the autocorrelation value, and correction may be performed only if the asymmetry is strong. For example, a large difference between C(-1) and C(1) may be used to determine that the asymmetry is strong.

[0073] The image processing device 110 of this embodiment may perform processing on the image set acquired in step S310 to correct the imbalance in light intensity mainly caused by vignetting of the optical system 122. Specifically, the light intensity balance can be corrected by correcting the brightness values ​​of image A and image B to be approximately constant regardless of the field of view, based on the results of the imaging device 100 having previously photographed a surface light source with constant brightness. In addition, to reduce the effects of light shot noise generated in the image sensor 121, for example, a bandpass filter or a lowpass filter may be applied to the acquired image A and image B.

[0074] The image processing device 110 of this embodiment searches for disparity in one dimension, but it may also perform searches in two or more dimensions. In that case, any known method may be used to extend the dimension. For example, after calculating the disparity in the x direction using the image processing device 110 of this embodiment, the image processing device 110 of this embodiment may be used to calculate the disparity in the y direction to calculate a two-dimensional disparity.

[0075] In this embodiment, the image processing device calculates the amount of disparity using cross-correlation values ​​corrected with autocorrelation values. This process corrects the asymmetry of the fitting function that appears in the cross-correlation values, thereby eliminating the asymmetry of the fitting function during sub-pixel estimation. As a result, the calculation error of the amount of disparity that occurs in relation to the asymmetry of the fitting function is reduced, and the distance measurement error is reduced, enabling highly accurate distance measurement.

[0076] <Second Embodiment> A second embodiment of the present invention will be described in detail below with reference to the figures. It should be noted that the components described in this embodiment are merely illustrative, and the scope of the present invention is not limited to the components described in this embodiment.

[0077] (Configuration of the image processing device) The image processing apparatus 701 of this embodiment will now be described. Figure 7(A) is a schematic diagram showing the configuration of the image processing apparatus 701 according to an embodiment of the present invention. In Figure 7(A), the image processing apparatus 701 includes a cross-correlation calculation unit, an autocorrelation calculation unit, and a disparity calculation unit. Figure 7(B) is a flowchart showing the operation of the image processing apparatus 701 of this embodiment. When the image processing according to this embodiment is started, the process moves to step S310. In Figure 7, parts that are the same as the configuration or operation described in Figure 3 are given the same numbers as in Figure 3 and their description is omitted.

[0078] The image processing device 701 can be configured using logic circuits. Alternatively, the image processing device 701 may consist of a central processing unit (CPU) and a memory for storing processing programs.

[0079] The image processing device 701 generates image A and image B based on the received image signal. The image processing device 701 calculates a disparity value using image A and image B through a disparity calculation process and stores the calculated disparity value in the main memory (not shown). The image processing device 701 also stores the image obtained by adding image A and image B as image information in the main memory, which can be used in subsequent processing. The image processing device 701 can also store image A and image B themselves in the main memory.

[0080] In step S710, the disparity calculation unit 710 calculates the amount of disparity of the image set, including image A and image B, using a combination of cross-correlation and autocorrelation values ​​that are not affected by the asymmetry of the fitting function.

[0081] As shown in Figure 5, fitting correlation values ​​existing on different functions to a single function is one cause of error. Therefore, the above error can be reduced by calculating the disparity using only cross-correlation values ​​appropriate for the function being used.

[0082] For example, when using the autocorrelation curve 511C, the disparity can be calculated using the cross-correlation value S(1). The amount of movement by which the autocorrelation curve 511C is moved horizontally in the disparity search direction, and the amount by which the cross-correlation value S(1) exists on the autocorrelation curve 511C, is the subpixel disparity d. Specifically, the above amount of movement can be calculated using equation 5. Note that, from the positional relationship in Figure 5(E), the cross-correlation value S(1) is located on the + side of the disparity search direction from the minimum value of the autocorrelation curve 511C, so the subpixel disparity d can be uniquely determined.

[0083]

number

[0084] Using a similar approach, subpixel disparity can also be calculated when other cross-correlation values ​​such as S(-1) or S(0) are used.

[0085] Here, equation 5 does not limit the correction method; any method that calculates disparity using a combination of cross-correlation and autocorrelation values ​​that is unaffected by the asymmetry of the fitting function is acceptable.

[0086] Furthermore, while the above method calculates the amount of disparity using an autocorrelation curve and one cross-correlation value, it is also possible to calculate the amount of disparity using an autocorrelation curve and multiple cross-correlation values. For example, disparity may be calculated using the autocorrelation curve 512C and two cross-correlation values, S(-1) and S(0).

[0087] Alternatively, if using the autocorrelation curve 512C, the disparity can be calculated using conventional methods such as Equation 3 with the cross-correlation values ​​S(-1) and S(0).

[0088] In this case, we may estimate the true subpixel disparity position and modify the formula used to estimate which function the cross-correlation value lies on. In doing so, we may use known methods. For example, we may compare the values ​​of S(1) and S(-1) and assume that the true subpixel disparity position lies on the side with the smaller value.

[0089] Alternatively, the asymmetry of the fitting function may be considered during this estimation, and the cross-correlation values ​​may be corrected. For example, the values ​​of S(1) / C(1) and S(-1) / C(-1) may be compared, and the true subpixel disparity position may be assumed to be on the side with the smaller value.

[0090] Alternatively, the process of estimating the true subpixel disparity position can be omitted, and a constant calculation method can always be used.

[0091] In this embodiment, the image processing apparatus calculates the amount of disparity using only one combination of cross-correlation value and autocorrelation curve. However, the amount of disparity may be calculated using multiple combinations of cross-correlation value and autocorrelation curve. For example, disparity may be calculated using the autocorrelation curve 511C and the cross-correlation value S(1), and then again using the autocorrelation curve 512C and the cross-correlation value S(-1). In that case, the results of the multiple disparity calculations may be averaged. Furthermore, the final amount of disparity may be determined from the confidence level of the autocorrelation value or cross-correlation value or the combination of the autocorrelation value and cross-correlation value used.

[0092] In this embodiment, the image processing device calculates the amount of disparity using a combination of cross-correlation and autocorrelation values ​​that are not affected by the asymmetry of the fitting function. By correcting the disparity calculation formula, the disparity can be calculated without being affected by the asymmetry of the fitting function. As a result, the calculation error of the amount of disparity that occurs in relation to the asymmetry of the fitting function is reduced, and the distance measurement error is reduced, enabling highly accurate distance measurement.

[0093] The image processing apparatus of this embodiment can reduce the amount of computational processing required for disparity calculation.

[0094] <Third Embodiment> A third embodiment of the present invention will be described in detail below with reference to the figures. Note that the components described in this embodiment are merely illustrative, and the scope of the present invention is not limited to the components described in this embodiment.

[0095] (Configuration of the image processing device) The image processing apparatus 801 of this embodiment will now be described. Figure 8(A) is a schematic diagram showing the configuration of the image processing apparatus 801 according to an embodiment of the present invention. In Figure 8(A), the image processing apparatus 801 includes a cross-correlation calculation unit, an autocorrelation calculation unit, and a disparity calculation unit. Figure 8(B) is a flowchart showing the operation of the image processing apparatus 801 of this embodiment. When the image processing according to this embodiment is started, the process moves to step S310. In Figure 8, parts that are the same as the configuration or operation described in Figure 3 are given the same numbers as in Figure 3 and their description is omitted.

[0096] The image processing device 801 can be configured using logic circuits. Alternatively, the image processing device 801 may consist of a central processing unit (CPU) and a memory for storing processing programs.

[0097] The image processing device 801 generates image A and image B based on the received image signal. The image processing device 801 calculates a disparity value using image A and image B through a disparity calculation process and stores the calculated disparity value in the main memory (not shown). The image processing device 801 also stores the image obtained by adding image A and image B as image information in the main memory, which can be used in subsequent processing. The image processing device 801 can also store image A and image B themselves in the main memory.

[0098] In step S810, the fitting function correction unit 810 estimates the asymmetry of the fitting function using the autocorrelation value and performs the correction.

[0099] In the disparity calculation process performed in step S314, a function determined by the method used to calculate the cross-correlation value is fitted to the cross-correlation value. Conventionally, this function uses a shape that is symmetrical with respect to the disparity boundary, but as mentioned above in Figure 5, due to the asymmetry of the edge texture, it actually has a shape that is symmetrical with respect to the disparity boundary. In step S810, this asymmetry is estimated and the shape of the fitting function that fits to the cross-correlation value is calculated.

[0100] Specifically, this asymmetry is estimated using the autocorrelation value calculated in step S312. For example, in Figure 5, two slopes of the quadratic function can be calculated using C(-1) and C(1). C(-1) is the coefficient of the quadratic function to the left of the parallax boundary, and C(1) is the coefficient of the quadratic function to the right of the parallax boundary.

[0101] In this way, by correcting the conventionally symmetrical fitting function to an asymmetric shape, it becomes possible to fit an asymmetric fitting function to the cross-correlation value in the subsequent step S314.

[0102] As a result, the calculation error of the disparity amount that occurs in relation to the asymmetry of the fitting function is reduced, and the distance measurement error is reduced, enabling highly accurate distance measurement.

[0103] The image processing device of this embodiment can perform high-precision distance measurement even in noisy environments.

[0104] <Fourth Embodiment> A fourth embodiment of the present invention will be described in detail below with reference to the figures. Note that the components described in this embodiment are merely illustrative, and the scope of the present invention is not limited to the components described in this embodiment.

[0105] (Device configuration) Figure 9 is a schematic diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. In Figure 9, parts that are the same as those described in Figure 1 are given the same numbers as in Figure 1 and their descriptions are omitted.

[0106] In Figure 9, the imaging device 900 comprises an image processing device 110, an imaging unit 920, and a distance calculation unit 130.

[0107] The imaging unit 920 comprises two image sensors 921 and 922, and two optical systems 923 and 924. The optical systems 923 and 924 are the photographic lenses of the imaging device 900 and have the function of forming an image of the subject on the image sensor 921 or 922. The optical systems 923 and 924 are composed of multiple lens groups (not shown) and apertures (not shown), etc., and have exit pupils 925 or 926 at a predetermined distance from the image sensor 921 or 922. At this time, the optical axes of the optical systems 923 and 924 are 941 and 942, respectively.

[0108] By calibrating parameters such as the relative positions of the optical systems in advance, the parallax between images can be calculated accurately. Furthermore, correcting lens distortion in each optical system also allows for accurate calculation of the parallax between images.

[0109] In this case, the design flexibility of the baseline length is increased, and the distance measurement resolution can be improved.

[0110] In this embodiment, there are two optical systems that acquire A image and B image with parallax depending on the distance, but it may also be composed of a stereo camera consisting of three or more optical systems and corresponding image sensors.

[0111] In this embodiment, the image processing apparatus described in the first embodiment is used as the image processing apparatus, but the image processing apparatuses in the second and third embodiments may also be used.

[0112] This invention includes not only a distance measuring device but also a computer program. The computer program of this embodiment causes the computer to perform predetermined steps in order to calculate distance or parallax. The program of this embodiment is installed in the computer of the distance measuring device or an imaging device such as a digital camera equipped with it. The above functions are realized when the installed program is executed by the computer, enabling high-speed and high-precision calculation of parallax.

[0113] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. An image processing device that calculates disparity from a first image and a second image, A cross-correlation calculation means for calculating the cross-correlation value between a first reference image set on the first image and a first reference image set on the second image, Autocorrelation calculation means for calculating the autocorrelation value between a second reference image and a second reference image set for either the first image or the second image, The system includes a disparity calculation means that calculates the amount of disparity between the first image and the second image using the cross-correlation value calculated by the cross-correlation calculation means and the autocorrelation value calculated by the autocorrelation calculation means, The disparity calculation means corrects the cross-correlation value using the autocorrelation value, The cross-correlation calculation means changes the position of the first reference image and calculates at least two of the cross-correlation values. The autocorrelation calculation means calculates at least two autocorrelation values ​​by moving the position of the second reference image by at least +1 pixels and -1 pixels in an arbitrary direction. The disparity calculation means calculates a corrected cross-correlation value by correcting the cross-correlation value using the vicinity of the minimum value of the autocorrelation value, and calculates the position of the first reference image with the highest correlation by interpolating the corrected cross-correlation value or the cross-correlation value with the corrected cross-correlation value as the disparity amount. The disparity calculation means calculates the corrected cross-correlation value using the ratio of two autocorrelation values ​​with different displacement amounts of the second reference image for at least one of the cross-correlation values. An image processing apparatus characterized by the following:

2. The disparity calculation means calculates the ratio of the cross-correlation value to the autocorrelation value for at least two of the cross-correlation values ​​as the corrected cross-correlation value. The image processing apparatus according to feature 1.

3. An image processing device that calculates disparity from a first image and a second image, A cross-correlation calculation means for calculating the cross-correlation value between a first reference image set on the first image and a first reference image set on the second image, Autocorrelation calculation means for calculating the autocorrelation value between a second reference image and a second reference image set for either the first image or the second image, The system includes a disparity calculation means that calculates the amount of disparity between the first image and the second image using the cross-correlation value calculated by the cross-correlation calculation means and the autocorrelation value calculated by the autocorrelation calculation means, The disparity calculation means calculates the amount of disparity using the autocorrelation value and the cross-correlation value belonging to the same fitting function, The autocorrelation calculation means calculates at least one autocorrelation value by moving the second reference image by at least +1 pixels or -1 pixels in any direction. The disparity calculation means calculates the amount of disparity using the ratio of the cross-correlation value and the autocorrelation value that belong to the same fitting function, using the autocorrelation value. An image processing apparatus characterized by the following:

4. The image processing apparatus according to claim 3, characterized in that the disparity calculation means calculates the amount of disparity using a plurality of combinations of the cross-correlation values ​​and the autocorrelation values ​​belonging to the same fitting function.

5. The image processing apparatus according to claim 4, characterized in that the disparity calculation means calculates the amount of disparity using a plurality of combinations of the cross-correlation values ​​and autocorrelation values ​​belonging to the same fitting function, and uses the average of these as the final amount of disparity.

6. The image processing apparatus according to claim 4, characterized in that the disparity calculation means calculates the amount of disparity using a plurality of combinations of the cross-correlation values ​​and autocorrelation values ​​belonging to the same fitting function, and determines the final amount of disparity from the autocorrelation values ​​used.

7. The cross-correlation calculation means calculates the cross-correlation value using either the sum of squares of the differences between the first reference image and the first reference image, the sum of absolute values ​​of the differences, or the normalized cross-correlation. The autocorrelation calculation means calculates the sum of the squares of the differences between the second reference image and the second reference image. Alternatively, the cross-correlation value can be calculated using either the sum of the absolute values ​​of the differences or the normalized cross-correlation. The image processing apparatus according to any one of claims 1 or 3.

8. The disparity calculation means estimates the disparity position using the cross-correlation value or the ratio of the cross-correlation value to the autocorrelation value, and changes the processing method based on the result. The image processing apparatus according to any one of claims 1 to 7.

9. The image processing apparatus according to any one of claims 1 or 3, further comprising a determination means for determining whether or not to perform correction by the disparity calculation means based on the reliability of the cross-correlation value or the autocorrelation value.

10. The image processing apparatus according to any one of claims 1 or 3, further comprising a determination means for determining whether or not to perform correction by the disparity calculation means based on the magnitude of the asymmetry of the edge texture.

Images