Image processing device, imaging device, control method, program, and storage medium

Abstract

To provide an image processing apparatus that can determine a perspective conflict area with high accuracy.SOLUTION: An image processing apparatus (110) determines the reliability of parallax calculated based on a first image and a second image having parallax, and the image processing apparatus has: acquisition means (112, 114) that acquire first parallax with the second image calculated by setting a reference image to the first image, and second parallax with the first image calculated by setting a reference image to the second image; and determination means (115) that determines the reliability of a pixel of interest. The determination means determines the reliability by comparing a parallax change rate representing sharpness of the change of the first parallax and a parallax change rate representing sharpness of the change of the second parallax around the position of the pixel of interest.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method, a program, and a storage medium.

Background Art

[0002] Conventionally, a method of determining whether or not the result of disparity calculation falls within a foreground / background conflict region and calculating the reliability of disparity calculation is known. Patent Document 1 discloses a method of determining a foreground / background conflict region by comparing the contrast distributions between images. Patent Document 2 discloses a method of determining a foreground / background conflict region using the defocus values of a reference pixel and two peripheral pixels that are displaced by a predetermined amount in opposite directions with respect to the reference pixel. Patent Document 3 discloses a method of calculating corresponding points between an image other than a reference image using each image as the reference image and determining whether or not the corresponding points match when the reference image is changed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] The method disclosed in Patent Document 1 cannot make a correct determination if an object with a pattern is present at a distance where the parallax is not zero pixels. The method disclosed in Patent Document 2 assumes that the parallax change in the near-far competition region is gradual, and therefore cannot make a correct determination if the calculated parallax contains many errors. The method disclosed in Patent Document 3 cannot make a correct determination if an image taken with an imaging device with a short baseline length is used, or if the image size of the reference image is large when calculating the parallax.

[0005] Therefore, the present invention aims to provide an image processing device capable of determining near-far competitive areas with high accuracy. [Means for solving the problem]

[0006] An image processing apparatus as one aspect of the present invention is an image processing apparatus for determining the reliability of a parallax calculated based on a first image and a second image having parallax, comprising acquisition means for acquiring a first parallax between the first image and the second image calculated by setting a reference image in the first image, and a second parallax between the second image and the first image calculated by setting a reference image in the second image, and a pixel of interest. Parallax The determination means for determining the reliability of the first parallax around the position of the pixel of interest compares the parallax change rate representing the steepness of the parallax change of the first parallax and the parallax change rate representing the steepness of the parallax change of the second parallax, and Parallax Determine the level of confidence.

[0007] Other objects and features of the present invention are described in the following embodiments. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that can determine near and far competing regions with high accuracy. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram of the imaging device in the first embodiment. [Figure 2]This is an explanatory diagram of the exit pupil of the optical system in the first embodiment. [Figure 3] This is an explanatory diagram of the reliability calculation device in the first embodiment. [Figure 4] This is an explanatory diagram illustrating the positional relationship between the reference image and the reference image in the first embodiment. [Figure 5] This is an explanatory diagram of the parallax change in the near / far competition region in the first embodiment. [Figure 6] This is an explanatory diagram of the parallax change in the near / far competition region in the first embodiment. [Figure 7] This is an explanatory diagram of the parallax change rate comparison process in the first embodiment. [Figure 8] This is an explanatory diagram of the parallax change rate comparison process in the first embodiment. [Figure 9] This is an explanatory diagram of the imaging device in the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] First, before describing each embodiment, an overview of the present invention will be provided. As a method for acquiring multiple images and calculating three-dimensional information, for example, there is a block matching method for obtaining three-dimensional information from multiple image data. In this method, first, for two images, image A and image B, taken from different viewpoints, an arbitrary region 1 is set as the reference image in image A, and region 2 is set as the reference image in image B. The position of region 2 is then changed to search for the region 2 that is most similar to region 1. The distance is then calculated from the positional difference between region 1 and region 2. In the search, cross-correlation, which represents the degree of difference (or similarity) between regions of different images, is used to determine similar regions. This positional difference is called parallax, and distance information can be obtained using known methods such as triangulation. Hereafter, the shift in texture on the image will be called the amount of shift, and the positional difference between region 1 and region 2, which is the result of the parallax search, will be called parallax.

[0012] In parallax calculation methods such as the block matching method, errors may occur in the results. Assume an environment where there are an object 1 placed at a position far from the shooting position and an object 2 placed at a position close to the shooting position as subjects. Depending on the conditions of the imaging device or the like, the amount of deviation between the distance from the shooting position to the object and the images in which the object is depicted is determined by a unique relationship. Therefore, the deviation amount 1 on the image of object 1 captured in image A and the image of object 1 captured in image B and the deviation amount 2 on the image of object 2 captured in image A and the image of object 2 captured in image B are different values. Here, consider the case where both object 1 and object 2 are observed in the reference image. The reference image is in a state where the texture of object 1 shifted by the deviation amount 1 and the texture of object 2 shifted by the deviation amount 2 are mixed from the reference image. An area including subjects at different distances in the reference image or the reference image is called a perspective competition area. When performing a parallax calculation method represented by the above block matching in the perspective competition area, an incorrect parallax calculation is performed because there are differences in the textures between the images.

[0013] Therefore, in each embodiment, a configuration and method for determining whether the result of parallax calculation applies to the perspective competition area and determining the reliability of the parallax calculation with high accuracy will be described.

[0014] (First Embodiment) First, referring to FIGS. 1(A) to 1(C), the imaging device 100 in the first embodiment of the present invention will be described. FIG. 1(A) is a schematic diagram of the imaging device 100. The imaging device 100 includes a reliability calculation device (image processing device) 110, an imaging unit (imaging means) 120, and a parallax correction unit (correction means) 130. The reliability calculation device 110 and the parallax correction unit 130 are configured using a logic circuit, but are not limited thereto. The reliability calculation device 110 and the parallax correction unit 130 may be configured, for example, from a central processing unit (CPU) and a memory that stores an arithmetic processing program.

[0015] The imaging unit 120 includes an imaging element 121 and an optical system 122. The optical system 122 is an imaging lens (imaging optical system) of the imaging device 100, and forms an image of a subject (optical image) on the imaging element 121. The optical system 122 is composed of a plurality of lens groups (not shown), an aperture (not shown), etc., and has an exit pupil 123 at a position separated from the imaging element 121 by a predetermined distance. In this embodiment, the z-axis is an axis parallel to the optical axis 140 of the optical system 122, and the x-axis and the y-axis are perpendicular to each other and perpendicular to the optical axis 140.

[0016] The imaging element 121 is composed of a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor. The imaging element 121 photoelectrically converts the subject image formed through the optical system 122 and generates an image signal based on the subject image. In this embodiment, the imaging device 100 is integrally configured with a camera body including the imaging element 121 and the optical system 122, but is not limited thereto. The optical system 122 may be a lens device (interchangeable lens) detachable from the imaging device 100.

[0017] FIG. 1(B) is an xy cross-sectional view of the imaging element 121. The imaging element 121 is configured by arranging a plurality of pixel groups 150 of 2 rows × 2 columns. The pixel group 150 is configured such that green pixels 150G1, 150G2 are arranged in the diagonal direction, and red pixel 150R and blue pixel 150B are arranged in the other two pixels.

[0018] Figure 1(C) is a cross-sectional view along the line I-I' in Figure 1(B). Each pixel consists of a light-receiving layer 182 and a light-guiding layer 181. The light-receiving layer 182 has multiple photoelectric conversion units (first photoelectric conversion unit 161 and second photoelectric conversion unit 162, which acquire a first image and a second image with parallax, respectively) for converting the received light into photoelectric power. In other words, the image sensor 121 has multiple first photoelectric conversion units 161 and multiple second photoelectric conversion units 162. The light-guiding layer 181 has microlenses 183 for efficiently guiding the light beam incident on the pixel to the photoelectric conversion unit, a color filter (not shown) that allows light in a predetermined wavelength band to pass through, and wiring (not shown) for image readout and pixel driving. In addition, each pixel is provided with wiring (not shown), and each pixel can send an image signal (output signal) to the reliability calculation device 110 via the wiring. Figures 1(B) and (C) show an example of a photoelectric conversion unit divided into two in one pupil division direction (x-axis direction). However, depending on the specifications, an image sensor equipped with a photoelectric conversion unit divided into two pupil division directions (x-axis direction and y-axis direction) may be used. The pupil division direction and the number of divisions are arbitrary.

[0019] Figure 2 is an explanatory diagram of the exit pupil 123 of the optical system 122, as viewed from the intersection point (central image height) of the optical axis 140 and the image sensor 121. The first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 are each incident on the first and second photoelectric conversion units 161, respectively, which are different regions of the exit pupil 123: the first light beam that has passed through the first pupil region 210 and the second light beam that has passed through the second pupil region 220. The first photoelectric conversion unit 161 and the second photoelectric conversion unit 162 at each pixel can generate image signals corresponding to image A (first image) and image B (second image), respectively, by photoelectric conversion of the incident light beam. The generated image signals are transmitted to the reliability calculation device 110.

[0020] Figure 2 shows the centroid position of the first pupil region 210 (first centroid position 211) and the centroid position of the second pupil region 220 (second centroid position 221). In this embodiment, the first centroid position 211 is eccentric (moves) along the first axis 200 in a predetermined direction (x-axis direction) from the center of the exit pupil 123. On the other hand, the second centroid position 221 is eccentric (moves) along the first axis 200 in the opposite direction to the first centroid position 211 (-x-axis direction). The direction connecting the first centroid position 211 and the second centroid position 221 is called the pupil division direction. The baseline length 230 corresponds to the distance between the centroids of the first centroid position 211 and the second centroid position 221.

[0021] Next, the parallax correction unit 130 in this embodiment will be described. The parallax correction unit 130 is a correction means for correcting parallax that has been determined to have low reliability. That is, the parallax correction unit 130 interpolates the parallax value with low reliability based on the received A image reference parallax value or B image reference parallax value and reliability, and stores the result in memory (not shown). Now, let's consider the case where the A image reference parallax value and reliability are obtained. If the reliability of the parallax of the pixel of interest is lower than the threshold, interpolation may be performed using the parallax of the neighbors of the pixel of interest. In this case, only the parallax of the neighbors of the pixel of interest with a reliability higher than the threshold may be used, and the average value or median may be used for interpolation. Alternatively, interpolation may be performed using parallax calculated from images close to the time of capture of the images (first image and second image).

[0022] Next, the reliability calculation device 110 in this embodiment will be described with reference to Figures 3(A) and (B). The reliability calculation device 110 is an image processing device that determines the reliability (reliability representing the certainty of the parallax) of the parallax (parallax information) calculated based on an A image (first image) and a B image (second image) having parallax. Figure 3(A) is a block diagram of the reliability calculation device 110. In Figure 3(A), the reliability calculation device 110 includes a first correlation calculation unit 111, a first parallax calculation unit 112, a second correlation calculation unit 113, a second parallax calculation unit 114, and a comparison unit 115. The first parallax calculation unit 112 and the second parallax calculation unit 114 function as acquisition means for acquiring a first parallax between the first image and the second image calculated by setting a reference image in the first image, and a second parallax between the second image and the first image calculated by setting a reference image in the second image. The comparison unit 115 functions as a determination means for determining the reliability of the pixel of interest.

[0023] Figure 3(B) is a flowchart showing the operation of the reliability calculation device 110. When image processing in this embodiment is started, the process moves to step S310.

[0024] First, in step S310, the imaging device 100 takes an image, and the reliability calculation device 110 generates and acquires an image set including image A and image B with parallax corresponding to the distance, and stores the acquired image set in the memory (not shown) of the imaging device 100. Here, the reliability calculation device 110 may perform processing on the image set acquired in step S310 to correct the imbalance in light intensity mainly caused by vignetting of the optical system 122. Specifically, the light intensity balance can be corrected by correcting the brightness values ​​of image A and image B to be approximately constant regardless of the field of view, based on the results of imaging device 100 having previously photographed a surface light source with constant brightness. In addition, a bandpass filter or a lowpass filter may be applied to the acquired image A and image B to reduce the effects of light shot noise, for example, generated in the image sensor 121.

[0025] Next, in step S311, the first correlation calculation unit 111 performs A-pixel-based cross-correlation processing. That is, the first correlation calculation unit 111 sets the image of a partial region on image A that includes the pixel for which distance calculation is performed (the pixel of interest) as the reference image, and sets the reference image in image B. Then, the first correlation calculation unit 111 calculates the A-image-based cross-correlation value between the reference image and the reference image while moving the position of the reference image in a predetermined direction.

[0026] Figures 4(A) and 4(B) are explanatory diagrams illustrating the positional relationship between the reference image and the reference image set in step S311. Figure 4(A) shows image A 410A, and Figure 4(B) shows image B 410B. In step S311, the first correlation calculation unit 111 calculates the cross-correlation value of image A 410A and image B 410B based on image A. Specifically, the first correlation calculation unit 111 first extracts a subregion from image A 410A that includes the pixel of interest 420 and its neighboring pixels, and sets it as the reference image 411. Next, the first correlation calculation unit 111 extracts a region on image B 410B with the same area (image size) as the reference image 411 and sets it as the reference image 412. Subsequently, the first correlation calculation unit 111 moves the position from which the reference image 412 is extracted on the B image 410B and calculates the A image reference cross-correlation value between the reference image 412 and the reference image 411 at each movement amount (each position). This allows the first correlation calculation unit 111 to calculate the A image reference cross-correlation value, which consists of a data sequence of correlation values ​​corresponding to each movement amount. In this case, the direction of movement of the reference image 412 can be any direction. The direction in which the reference image 412 is moved and the cross-correlation calculation is performed is called the disparity search direction. By setting the disparity search direction and the pupil division direction to the same direction, the calculation by the first disparity calculation unit 112, described later, can be simplified.

[0027] The image-referenced cross-correlation value only needs to evaluate the degree of correlation between reference image 411 and reference image 412, and can be calculated using any known method. For example, the sum of squared differences (SSD), the sum of absolute differences (SAD), or normalized cross-correlation (NCC) can be used. The following describes the method using SSD, but the same idea can be used even if a different method is used. SSD evaluates the degree of difference between reference image 411 and reference image 412, and the lower the cross-correlation value, the higher the correlation.

[0028] Next, in step S312 of Figure 3, the first disparity calculation unit 112 performs the A-image reference disparity calculation process. That is, the first disparity calculation unit 112 calculates the amount of disparity (A-image reference disparity) relative to the A-image using any known method. For example, the position where the A-image reference cross-correlation value is minimized may be used as the amount of disparity. Alternatively, subpixel estimation may be performed to obtain the disparity in units of fractional pixels. For example, if the cross-correlation value is SSD, the minimum value can be obtained by interpolating with a quadratic function. Also, if the cross-correlation value is SAD, the minimum value can be obtained by interpolating with an isometric line.

[0029] Next, in step S313, the second correlation calculation unit 113 calculates the B-image-based cross-correlation value between image A 410A and image B 410B (B-image-based cross-correlation calculation process). Specifically, the second correlation calculation unit 113 performs the same process as in step S311, but with image A and image B reversed. Alternatively, the B-image-based cross-correlation value may be calculated using the A-image-based cross-correlation value obtained in step S311. The A-image-based cross-correlation value when the position is (x, y) and the amount of movement of the reference image is (a, 0) will be the same as the B-image-based cross-correlation value when the position is (x+a, y) and the amount of movement of the reference image is (-a, 0). This occurs because the images from which the difference is taken are the same. In other words, the B-image-based cross-correlation value can be calculated by rearranging the A-image-based cross-correlation value.

[0030] Next, in step S314, the second disparity calculation unit 114 performs B-image reference disparity calculation processing. In step S314, the amount of disparity relative to the B image (B-image reference disparity) is calculated using any known method. For example, the position where the B-image reference cross-correlation value is minimized may be used as the amount of disparity. Alternatively, subpixel estimation may be performed to obtain disparity in units of fractional pixels.

[0031] The confidence calculation device 110 of this embodiment has a one-dimensional parallax search direction, but it may also perform searches in two or more dimensions. In that case, any known method may be used to extend the dimensions. For example, the confidence calculation device 110 of this embodiment may be applied to a two-dimensional simultaneous estimation method. In this method, parallax in two directions, x and y, can be obtained. The confidence levels of the two parallaxes obtained in this way can be calculated by performing the processing in step S315, which will be described later, for each of them.

[0032] In this embodiment, the processing in steps S310 to S314 is not limited to the method described above, and other methods may be used as long as the A-image reference disparity and B-image reference disparity can be calculated. For example, a method for calculating disparity may be learned using machine learning, and the respective disparities may be calculated. Alternatively, a disparity calculation unit may be provided outside the confidence calculation device 110, and disparity information may simply be received from there.

[0033] Next, in step S315, the comparison unit 115 performs a parallax change rate comparison process. That is, the comparison unit 115 determines the steepness of the parallax change around the pixel of interest for the two parallax values ​​obtained in steps S312 and S314, and calculates the reliability by comparing them.

[0034] Here, we will explain the characteristics of parallax in the near-far competition region by referring to Figures 5(A) to (D). Figures 5(A) to (D) are diagrams illustrating the changes in parallax in the near-far competition region.

[0035] Figures 5(A) and (B) show image A 501A or image B 501B acquired using the imaging device 100. Image A and image B contain object 1, indicated by 511A and 511B, and object 2, indicated by 512A and 512B. A texture, indicated by 513, exists on a portion of object 1. Due to differences in viewpoint, texture 513 is only observed in image A 501A (texture 513 exists in the occlusion area). If the reference parallax of object 1 and object 2 in image A are disp1 and disp2, then the reference parallax of their respective images in image B are -disp1 and -disp2. 520 indicates the image size of the reference image.

[0036] Figures 5(C) and (D) show the relationship between pixel position and disparity when the search direction for the near and far competition region is set to the x-direction on Figures 5(A) and (B). The reference disparity of image A shown in Figure 5(C) is calculated as disp1 when the reference image is entirely within object 1. In region 531 where the reference image overlaps with the occlusion area, there are no areas in image B that match the reference image, so the calculated disparity value is more likely to contain errors. In particular, when a large portion of the reference image is in the occlusion area, there is a high possibility that a significantly different disparity value will be calculated. Note that the disparity value shown in region 531 is an illustration and does not necessarily change as shown in the figure. When the reference image moves beyond the occlusion area, the disparity is calculated as disp2.

[0037] On the other hand, the B-image reference disparity shown in Figure 5(D) is calculated as -disp1 when the reference image is entirely present in object 1. In region 531, where the reference image includes both object 1 and object 2, there is no region in image A that matches the reference image, so the calculated disparity value is more likely to contain errors. However, unlike the case of A-image reference, the texture exists somewhere in image A, so it does not contain large errors. The disparity gradually changes from -disp1 to -disp2, depending on the proportion of the texture in the reference image. When the reference image is entirely present in object 2, the disparity is calculated as -disp2. Hereafter, the A-image reference disparity at pixel position x will be Da(x), and the B-image reference disparity will be Db(x).

[0038] Next, we will explain the relationship between the A-image reference disparity and the B-image reference disparity when there is no near / far competition. The A-image reference disparity at the position indicated by 541A is Da(x1)=disp1. This disparity value correctly represents the disparity value of object 1. The corresponding pixel position for the B-image reference disparity is x1+disp1, indicated by 541B. In this case, Db(x1+disp1)=-disp1. When the sum of the corresponding disparity values ​​is 0, this disparity value is judged to be reliable because there is no near / far competition.

[0039] Next, we will explain the relationship between the A-image reference disparity and the B-image reference disparity when near-field competition occurs. The A-image reference disparity at the position indicated by 542A is Da(x2)=disp3. This disparity value is not the disparity value of object 2, but a value in which an error has occurred due to near-field competition. The corresponding pixel position of the B-image reference disparity is x1+disp3, indicated by 542B. In this case, Db(x1+disp3)=-disp1. When the corresponding disparity values ​​are added together in this way, the disparity value is judged to be unreliable because near-field competition has occurred.

[0040] Next, with reference to Figures 6(A) to 6(D), the problems to be solved in this embodiment will be explained. Figures 6(A) to 6(D) are explanatory diagrams of parallax changes in the near and far competition region, and are diagrams used to explain the problems to be solved in this embodiment.

[0041] Figures 6(A) and 6(B) show image A 501A or image B 501B acquired using the imaging device 100. Image A 501A and image B 501B are the same as image A 501 and image B 501B shown in Figures 5(A) and 5(B), respectively. 610 indicates the image size of the reference image, which is larger than the image size 520 shown in Figure 5(A).

[0042] Figures 6(C) and (D) show the relationship between pixel position and disparity when the search direction for the near-far competition region is set to the x-direction on Figures 6(A) and (B). The reference disparity of image A shown in Figure 6(C) is calculated as disp1 when the reference image is entirely within object 1. In region 621 where the reference image overlaps with the occlusion region, there is no region in image B that matches the reference image, so the calculated disparity value is more likely to contain errors. However, since the image size of the reference image is larger than the occlusion region, some of the texture of the reference image is present in image B. Therefore, it rarely contains large errors. From disp1 to disp2, the disparity gradually changes according to the proportion of texture within the reference image. When the reference image leaves the occlusion region, the disparity is calculated as disp2.

[0043] On the other hand, the B-image reference disparity shown in Figure 6(D) is calculated as -disp1 when the reference image is entirely present in object 1. In region 622 where the reference image includes both object 1 and object 2, there is no region in image A that matches the reference image, so the calculated disparity value is more likely to contain errors. As mentioned above, the texture exists somewhere in image A, so it rarely contains large errors. The disparity gradually changes from -disp1 to -disp2 depending on the proportion of the texture in the reference image. When the reference image is entirely present in object 2, the disparity is calculated as -disp2.

[0044] Next, we will explain the relationship between the A-image reference disparity and the B-image reference disparity when near-field competition is occurring. Note that the relationship between the A-image reference disparity and the B-image reference disparity when near-field competition is not occurring is the same as in Figures 5(C) and (D). The A-image reference disparity at the position indicated by 631A is Da(x2) = disp4. This disparity value is the disparity value between object 1 and object 2, and is the value in which an error occurs due to near-field competition. The corresponding pixel position of the B-image reference disparity is x1 + disp4, indicated by 631B. In this case, Db(x1 + disp4) = -disp4. Thus, when the corresponding disparity values ​​are added together, it becomes 0. In this way, Da(x2) is incorrectly judged to be reliable even though an error occurs due to near-field competition.

[0045] Thus, when the image size of the reference image is larger than the occlusion area, comparing the corresponding disparity values ​​can lead to an incorrect determination of the near-far competition area. This error in determination can occur not only when the reference image size is large, but also when the baseline length of the imaging unit is short.

[0046] Next, referring to Figure 7, the process of step S315 (parallax change rate comparison process) will be explained. Figure 7 is an explanatory diagram of the parallax change rate comparison process, and shows the A-image reference parallax and B-image reference parallax shown in Figures 6(C) and (D) on the same axis. The distance 701 from the pixel position where the parallax change begins in the A-image reference parallax to the pixel position where the parallax change begins in the B-image reference parallax represents the amount of shift in the edge position of object 1 between image A and image B, and is therefore disp1. Similarly, the distance 702 from the pixel position where the parallax change ends in the A-image reference parallax to the pixel position where the parallax change ends in the B-image reference parallax represents the amount of shift in the boundary position of the texture of object 2 observed in both image A and image B, and is therefore disp2. Furthermore, if the width of the region where the parallax change occurs in the B-image reference parallax is W, then the width of the region where the parallax change occurs in the A-image reference parallax is W+D. Here, D is the absolute value of the parallax difference between disp1 and disp2. Furthermore, W is a value determined by the shape of the object boundary, the pixel size of the reference image, and the filter size if a bandpass or lowpass filter was applied in step S310.

[0047] The steepness of the parallax change Sa with reference parallax of image A and the steepness of the parallax change Sb with reference parallax of image B are calculated using the following equations (1) and (2).

[0048]

number

[0049] In areas where near-near-far competition does not occur, the signs of the parallax referenced by image A and image B are reversed, and their absolute values ​​are the same, so the parallax steepness Sa and Sb have the same property. That is, Sa + Sb = 0. Hereafter, this state will be referred to as the parallax steepness Sa and Sb being the same. In near-near-far competition, as shown in equations (1) and (2), Sa + Sb ≠ 0, and the parallax steepness is not the same. In this way, it is possible to determine whether or not an area is near-near-far competition by comparing the parallax steepness. In this embodiment, an area is determined to be near-near-far competition when the parallax steepness is not the same. That is, the comparison unit 115 compares the parallax change rate, which represents the steepness of the parallax change of the first parallax around the position of the pixel of interest, with the parallax change rate, which represents the steepness of the parallax change of the second parallax, and determines the reliability.

[0050] Next, with reference to Figure 8, the specific method of step S315 of this embodiment will be described. Figure 8 is an explanatory diagram of the parallax change rate comparison process, in which the parallax (A+B parallax) obtained by adding the A image reference parallax (first parallax) and the B image reference parallax (second parallax) at the same pixel positions as in Figure 7 is added by a dashed line.

[0051] Let 810 be the pixel (pixel of interest) used to determine the near-far competition region. The evaluation value 801 is used for evaluation. This is the difference between the maximum and minimum values ​​of the A+B parallax around pixel 810. If the evaluation value 801 is greater than or equal to a predetermined value, it is determined to be a near-far competition region. The evaluation value 801 increases as the degree of parallax difference between the A-image reference parallax and the B-image reference parallax differs, and is equivalent to determining the consistency of the parallax degrees.

[0052] When the evaluation value 801 is denoted as E, the evaluation value E is calculated using the following formula (3).

[0053]

number

[0054] When the parallax difference D is kept constant, the evaluation value E' takes its minimum value when disp2=0, and this value is D×D÷(W+D). Even if the relative magnitudes or signs of disp1 and disp2 are changed, if the parallax difference is D, the evaluation value will be greater than or equal to D×D÷(W+D). If the amount of error included in the parallax can be estimated in advance, that amount may be set as a threshold. In this way, the comparison unit 115 calculates the sum of the first parallax and the second parallax at the same pixel position around the position of the pixel of interest (A+B parallax), and if the difference between the maximum and minimum values ​​of the sum is greater than a predetermined difference, it determines that the reliability of the parallax at the pixel of interest is low.

[0055] Alternatively, instead of using the difference between the maximum and minimum values ​​of the A+B parallax as the evaluation value, the parallax changes around the pixel of interest in the A-image reference parallax and the B-image reference parallax may be estimated using straight lines, and their slopes may be compared. When near-far competition occurs, the slopes of the parallax changes differ between the A-image reference parallax and the B-image reference parallax. For example, in Figure 7, these slopes (first slope, second slope) correspond to the steepness Sa and Sb. By using the difference in these slopes as an evaluation value, the near-far competition region can be determined. That is, the comparison unit 115 determines that the reliability of the parallax at the pixel of interest is low if the first slope obtained by linearly interpolating the parallax change of the first parallax around the position of the pixel of interest is different from the second slope obtained by linearly interpolating the parallax change of the second parallax.

[0056] Alternatively, instead of using the difference between the maximum and minimum values ​​of the A+B parallax as the evaluation value, the width of the parallax change between the A-image reference parallax and the B-image reference parallax may be compared. When near-far competition occurs, the width of the region where parallax change occurs (the size of the first region and the second region) differs between the A-image reference parallax and the B-image reference parallax. For example, in Figure 7, these widths are W and W+D. By using these differences in width as evaluation values, the near-far competition region can be determined. Any known method may be used to estimate the width of the region where parallax change occurs. For example, the parallax change may be fitted with a straight line, and the distance from the starting point to the ending point may be determined. That is, the comparison unit 115 determines that the reliability of the parallax at the pixel of interest is low if the size of the first region with parallax change in the first parallax and the size of the second region with parallax change in the second parallax are different.

[0057] Furthermore, the width of the parallax change of either the A-image reference parallax or the B-image reference parallax is a constant value regardless of the parallax. This value (the size of the first or second region) is denoted as Wc. If the shape of the object boundary is perpendicular to the direction of determination of the near-far competition region, its size is determined by the pixel size BS of the reference image and the filter size FS of the filter, such as the bandpass filter or lowpass filter, performed in step S310. Specifically, Wc = BS + FS. Therefore, after estimating the width of the parallax change region, a comparison with Wc may be performed to determine the validity of the estimation result. Alternatively, the width of the parallax change region may be estimated again based on the result. In other words, the comparison unit 115 determines the validity of the confidence level determination. The comparison unit 115 also determines the validity based on whether the size of the first or second region matches a value based on at least one of the size of the reference image, the size of the filters applied to the first and second images, or the texture of the subject.

[0058] In this embodiment, the parallax correction unit 130 corrects parallax based on reliability, but is not limited to this. For example, the parallax correction unit 130 may also function as an extraction means that extracts only parallax determined to have high reliability without correcting parallax. The reliability calculation device 110 of this embodiment determines the near-far competition region by comparing the steepness of the parallax change between the A image reference parallax and the B image reference parallax. With this process, the near-far competition region can be determined regardless of the relationship between occlusion and the image size of the reference image at the time of parallax calculation. As a result, the near-far competition region can be correctly determined even when the image size of the reference image at the time of parallax calculation is large or when the baseline length of the imaging unit is short.

[0059] (Second Embodiment) Next, with reference to Figure 9, the imaging device 900 in the second embodiment of the present invention will be described. Figure 9 is a schematic diagram of the imaging device 900. In Figure 9, parts that are the same as those described with reference to Figure 1(A) of the imaging device 100 are given the same numbers and their descriptions are omitted.

[0060] The imaging device 900 comprises a reliability calculation device (image processing device) 110, an imaging unit (imaging means) 920, and a parallax correction unit (distance calculation unit) 130. The imaging unit 920 comprises an image sensor (first image sensor) 921, an image sensor (second image sensor) 922, and an optical system (first optical system) 923 and an optical system (second optical system) 924. Optical systems 923 and 924 are imaging lenses (imaging optical systems) of the imaging device 900, respectively. Optical system 923 forms an image of the subject on the image sensor 921. Optical system 924 forms an image of the subject on the image sensor 922. Optical systems 923 and 924 are composed of a plurality of lens groups (not shown) and an aperture (not shown), etc., and have an exit pupil 925 or an exit pupil 926 at a predetermined distance from the image sensor 921 or the image sensor 922. 941 and 942 are the optical axes of the optical systems 923 and 924, respectively. With this configuration, the image sensors 921 and 922 acquire a first image and a second image, respectively, which have parallax.

[0061] By pre-calibrating parameters such as the relative positions of optical systems 923 and 924, the parallax between images can be accurately calculated. Furthermore, by correcting lens distortion in both optical systems 923 and 924, the parallax between images can be accurately calculated. In this case, the design flexibility of the baseline length is improved, and the distance measurement resolution can be enhanced.

[0062] In this embodiment, two optical systems 923 and 924 are provided to acquire A image and B image, which have parallax depending on the distance. However, it may also be configured as a stereo camera consisting of three or more optical systems and three or more corresponding image sensors.

[0063] Each embodiment includes a computer program in addition to the distance measuring device. The computer program in this embodiment causes the computer to perform predetermined steps in order to calculate distance or parallax. The program in this embodiment is installed on the computer of the distance measuring device or an imaging device such as a digital camera equipped with it. The above functions are realized when the installed program is executed by the computer, enabling high-speed and high-precision calculation of parallax.

[0064] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0065] According to each embodiment, even when using images captured by an imaging device with a short baseline length or when block matching is performed with a large block size during disparity calculation, it is possible to determine whether the calculated disparity is in a near-far competition area and to calculate the reliability. Therefore, according to each embodiment, it is possible to provide an image processing device, imaging device, control method, program, and storage medium that can determine near-far competition areas with high accuracy.

[0066] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0067] 110 Confidence Calculation Device (Image Processing Device) 112 First parallax calculation unit (acquisition means) 114 Second parallax calculation unit (acquisition means) 115 Comparison section (judgment means)

Claims

1. An image processing device for determining the reliability of a parallax calculated based on a first image and a second image having parallax, An acquisition means for acquiring a first parallax between the first image and the second image calculated by setting a reference image to the first image, and a second parallax between the second image and the first image calculated by setting a reference image to the second image, It includes a determination means for determining the reliability of the parallax at the pixel of interest, The image processing apparatus is characterized in that the determination means determines the reliability of the parallax by comparing the parallax change rate, which represents the steepness of the parallax change of the first parallax around the position of the pixel of interest, with the parallax change rate, which represents the steepness of the parallax change of the second parallax.

2. The image processing apparatus according to claim 1, characterized in that the determination means determines that the reliability of the parallax at the pixel of interest is low if the first slope obtained by linearly interpolating the parallax change of the first parallax around the position of the pixel of interest is different from the second slope obtained by linearly interpolating the parallax change of the second parallax.

3. The image processing apparatus according to claim 1, characterized in that the determination means calculates the sum of the first parallax and the second parallax at the same pixel position around the position of the pixel of interest, and determines that the reliability of the parallax at the pixel of interest is low if the difference between the maximum and minimum values ​​of the sum is large.

4. The image processing apparatus according to claim 1, characterized in that the determination means determines that the reliability of the parallax at the pixel of interest is low when the size of the first region with parallax change of the first parallax and the size of the second region with parallax change of the second parallax are different.

5. The image processing apparatus according to claim 4, characterized in that the determination means determines the validity of the determination of the reliability of the parallax.

6. The image processing apparatus according to claim 5, characterized in that the determination means determines the validity based on whether the size of the first region or the second region matches a value based on at least one of the size of the reference image, the size of the filter applied to the first image and the second image, or the texture of the subject.

7. The image processing apparatus according to any one of claims 1 to 6, further comprising an extraction means for extracting only the parallaxes that have been determined to have a high degree of reliability.

8. The image processing apparatus according to any one of claims 1 to 6, further comprising a correction means for correcting parallax that has been determined to have low reliability.

9. The image processing apparatus according to claim 8, characterized in that the correction means performs interpolation using the parallax of neighboring pixels of the pixel of interest when it is determined that the reliability of the parallax in the pixel of interest is low.

10. The image processing apparatus according to claim 8, characterized in that, if the correction means determines that the reliability of the parallax is low in the pixel of interest, it performs interpolation using the parallax of the image closest to the time of capture of the first image and the second image.

11. Imaging means, An imaging apparatus characterized by having an image processing apparatus according to any one of claims 1 to 10.

12. The imaging means comprises an optical system and an image sensor. The optical system forms an image of the subject on the image sensor, The imaging apparatus according to claim 11, characterized in that the image sensor has a plurality of first photoelectric conversion units for acquiring the first image and a plurality of second photoelectric conversion units for acquiring the second image.

13. The imaging means is First image sensor and A first optical system for forming an image of the subject on the first image sensor, The second image sensor and It comprises a second optical system for forming an image of the subject on the second image sensor, The imaging apparatus according to claim 11, characterized in that the first image sensor acquires a first image and the second image sensor acquires a second image.

14. A control method performed by a control device that determines the reliability of the disparity calculated based on a first image and a second image having disparity, An acquisition step to acquire a first parallax between the first image and the second image calculated by setting a reference image to the first image, and a second parallax between the second image and the first image calculated by setting a reference image to the second image, The method includes a determination step of determining the reliability of the parallax at the pixel of interest, A control method characterized in that, in the determination step, the reliability of the parallax is determined by comparing the rate of parallax change representing the steepness of the parallax change of the first parallax around the position of the pixel of interest with the rate of parallax change representing the steepness of the parallax change of the second parallax.

15. A program characterized by causing a computer to execute the control method described in claim 14.

16. A computer-readable storage medium characterized by storing the program described in claim 15.

Images