Image processing device

Abstract

To properly screen a wrongly calculated displacement amount of a corresponding point between images derived from miss-matching on an image-captured environment between a set of images, or a low texture property of a subject.SOLUTION: An image processing device 100 comprises: a search unit 102 that searches for a corresponding point between a set of images; a degree-of-reliability calculation unit 104 that calculates a degree of array similarity, in a search range of the corresponding point, indicative of similarity between a luminance array in a pixel group within one image and a luminance array in a pixel group within other image, and calculates a degree of reliability with respect to a displacement amount (parallax) of the corresponding point between the set of images on the basis of the calculated degree-of-array similarity; and a correction unit 106 that, when the calculated degree of reliability is below a prescribed threshold, voids or corrects the displacement amount (parallax).SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] There is known a stereo camera system that uses the principle of triangulation to measure the distance to an object present within a field of view based on a pair of images captured by a pair of cameras whose fields of view overlap at least partially. The principle of triangulation is to calculate the distance from the camera to the object using the positional shift (amount of displacement) of the images of the same object captured by a pair of cameras, that is, parallax. Parallax is calculated by searching for where the image of the object in one image is present in the other image.

[0003] Various methods have been proposed for calculating disparity. For example, a classical method is block matching, which searches for an area of ​​multiple pixels in one image that is most similar (an area with low dissimilarity) to another image.

[0004] In an ideal shooting environment, accurate disparity can be calculated from a pair of images. However, in reality, matching errors may occur between a pair of images due to inconsistencies in the shooting environment, such as out-of-focus or lighting, or due to low texture of the subject, such as the road surface, resulting in erroneous calculation of disparity. This may cause errors in object recognition, which may hinder recognition of objects that have fallen on the road or the condition of the road surface. Note that low texture means that the texture of the subject is poorly varied, and the edges of the texture are difficult to detect in the image of the subject.

[0005] As a measure for detecting the position of a subject with high accuracy regardless of the shooting environment, for example, Patent Document 1 discloses a method for screening out disparities with low reliability by utilizing a dissimilarity index between a pair of images used to calculate disparity. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-38886 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, the reliability of disparity is determined by a change in an index itself that indicates similarity or dissimilarity, such as SAD (Sum of Absolute Difference), used when searching for corresponding points between a pair of images. Specifically, in Patent Document 1, the reliability of disparity is determined by a sharpness that indicates the degree of prominence of the maximum value of the index. As a result, in Patent Document 1, when a matching error with high sharpness occurs due to a mismatch in the shooting environment between a pair of images, it may not be possible to screen disparities with low reliability.

[0008] In addition, in Patent Document 1, a dissimilarity index derived from a comparison between single pixels in each image is used as the reliability of the disparity. As a result, in Patent Document 1, the features of the subject represented by the arrangement of multiple pixels in each image cannot be reflected in the reliability of the disparity. Therefore, in Patent Document 1, it is difficult to detect matching errors caused by low texture of the subject, and even if an erroneous disparity is calculated due to this matching error, there is a possibility that this erroneous disparity cannot be screened.

[0009] This problem applies not only to stereo camera systems, but also to monocular camera systems that search for corresponding points between a set of images taken by the same camera at different times and calculate the amount of displacement of the corresponding points over time.

[0010] The present invention has been made in consideration of the above, and aims to appropriately screen the amount of displacement of corresponding points between a pair of images that is erroneously calculated due to a discrepancy in the shooting environments between the images or due to low texture of the subject. [Means for solving the problem]

[0011] In order to solve the above problem, the image processing device of the present invention is characterized in that it comprises a search unit that searches for corresponding points between a pair of images, a reliability calculation unit that calculates an array similarity indicating the similarity between a luminance array of a pixel group in one image and a luminance array of a pixel group in the other image within a search range for the corresponding points, and calculates a reliability of the displacement amount of the corresponding points between the pair of images based on the calculated array similarity, and a correction unit that invalidates or corrects the displacement amount if the calculated reliability is below a predetermined threshold. Effect of the Invention

[0012] According to the present invention, it is possible to appropriately screen the amount of displacement of corresponding points between a pair of images that is erroneously calculated due to a mismatch in the shooting environment between the pair of images or due to low texture of the subject. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing the configuration of a camera system including an image processing device according to a first embodiment. [Diagram 2] FIG. 13 is a diagram for explaining an example of calculating sequence similarity. [Diagram 3] FIG. 4 is a diagram for explaining an example of calculating a reliability in the first embodiment. [Figure 4] 5 is a flowchart showing a procedure for invalid parallax determination in the first embodiment. [Diagram 5] FIG. 11 is a diagram for explaining an example of calculating a reliability in the second embodiment. [Figure 6] 10 is a flowchart showing a procedure for invalid parallax determination in the second embodiment. [Figure 7] 13 is a flowchart showing a procedure for invalid parallax determination in the third embodiment. [Figure 8] FIG. 13 is a diagram showing the configuration of a camera system including an image processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that configurations or functions denoted by the same reference numerals in each embodiment have the same configurations or functions in each embodiment unless otherwise specified, and description thereof will be omitted.

[0015] [First embodiment] An image processing device 100 according to a first embodiment will be described with reference to Figures 1 to 4. Figure 1 is a diagram showing the configuration of a camera system 1 equipped with the image processing device 100 according to the first embodiment.

[0016] The camera system 1 is a camera system mounted on a vehicle or the like, and includes a stereo camera 101 and an image processing device 100. The stereo camera 101 is configured with a pair of cameras whose fields of view overlap at least partially. The pair of cameras of the stereo camera 101 simultaneously capture the same subject from different viewpoints. When the pair of cameras of the stereo camera 101 is configured with a right camera and a left camera spaced apart in the left-right direction of the vehicle, one image of the pair of images is a right image captured by the right camera, and the other image of the pair of images is a left image captured by the left camera. The stereo camera 101 outputs the pair of captured images to the image processing device 100.

[0017] Instead of the stereo camera 101, the camera system 1 may include a multi-camera configured as a set of three or more cameras whose fields of view at least partially overlap each other.

[0018] The image processing device 100 includes a search unit 102, a disparity calculation unit 103, a reliability calculation unit 104, an invalid disparity determination unit 105, and a correction unit .

[0019] The image processing device 100 includes a CPU, a ROM, and a RAM. The image processing device 100 realizes the functions of a search unit 102, a disparity calculation unit 103, a reliability calculation unit 104, an invalid disparity determination unit 105, and a correction unit 106 by the CPU reading and executing a program stored in the ROM.

[0020] The search unit 102 performs a matching process to search for corresponding points between a pair of images. For example, the search unit 102 searches for corresponding points by block matching. Specifically, the search unit 102 searches for a small area (matching block) consisting of multiple pixels in one image of the pair of images, and searches for a small area having the highest similarity in the other image of the pair of images to search for corresponding points. The search unit 102 can use, as the similarity, SAD or ZSAD (Zero-mean Sum of Absolute Difference), which is calculated based on the luminance difference between corresponding pixels in the pair of images. The search range for corresponding points may be the entire image or a local range. In this embodiment, an example will be described in which the search range for corresponding points is a local search range limited to a part of the image. The search unit 102 outputs similarity information indicating the similarity for each pixel calculated when searching for corresponding points to the disparity calculation unit 103.

[0021] Furthermore, when searching for corresponding points, the search unit 102 focuses on a pixel group at a predetermined position in a small region of one image (for example, a pixel group located in the vertical center of the small region and aligned horizontally) and a pixel group at a predetermined position in a small region of the other image.The search unit 102 then outputs brightness array information indicating the arrangement of the brightness in the pixel group of one image (hereinafter also referred to as the "brightness array of one image") and the arrangement of the brightness in the pixel group of the other image (hereinafter also referred to as the "brightness array of the other image") to the reliability calculation unit 104.

[0022] The disparity calculation unit 103 calculates the disparity between the pair of images (i.e., the amount of displacement of corresponding points between the pair of images) for each pixel based on the similarity information output from the search unit 102. The disparity calculation unit 103 outputs disparity information indicating the calculated disparity to the invalid disparity determination unit 105.

[0023] The reliability calculation unit 104 calculates the reliability of the disparity calculated by the disparity calculation unit 103. At this time, the reliability calculation unit 104 calculates an array similarity indicating the similarity between the luminance array of one image and the luminance array of the other image in the search range of the corresponding point, and calculates the reliability of the disparity based on the calculated array similarity. An example of the calculation of the array similarity will be described later with reference to FIG. 2.

[0024] Specifically, the reliability calculation unit 104 calculates a correlation value between the luminance array of one image and the luminance array of the other image based on the luminance array information output from the search unit 102. The reliability calculation unit 104 regards the calculated correlation value as array similarity. The reliability calculation unit 104 calculates the reliability based on the calculated correlation value. The reliability calculation unit 104 outputs reliability information indicating the calculated reliability to the invalid parallax determination unit 105. An example of the reliability calculation will be described later with reference to FIG. 3.

[0025] The function of calculating the sequence similarity may be included in the search unit 102, not in the reliability calculation unit 104. That is, the search unit 102 may calculate the similarity for each pixel between a pair of images to search for corresponding points between the images, and may also calculate the sequence similarity between the pair of images. Then, the reliability calculation unit 104 may calculate the reliability for the disparity based on the sequence similarity calculated by the search unit 102.

[0026] The invalid parallax determination unit 105 determines which parallaxes are to be invalid among the parallaxes indicated by the parallax information output from the parallax calculation unit 103, based on the reliability information output from the reliability calculation unit 104. Specifically, the invalid parallax determination unit 105 determines whether or not each reliability for each parallax in the search range is below a predetermined threshold. The invalid parallax determination unit 105 determines a parallax whose reliability is below the threshold as invalid, and determines a parallax whose reliability is equal to or greater than the threshold as valid. The invalid parallax determination unit 105 outputs a determination result to the correction unit 106.

[0027] The correction unit 106 maintains the value of the disparity determined to be valid by the invalid parallax determination unit 105. The correction unit 106 replaces the disparity determined to be invalid by the invalid parallax determination unit 105 with invalid information to invalidate it. Alternatively, the correction unit 106 corrects the disparity of the target pixel to a reliable value by using the disparity of a pixel surrounding the target pixel having the disparity determined to be invalid by the invalid parallax determination unit 105. For example, the correction unit 106 corrects the disparity of the target pixel to an average value of the disparity of the surrounding pixels. The correction unit 106 outputs corrected disparity information, which is disparity information obtained by invalidating or correcting the disparity determined to be invalid by the invalid parallax determination unit 105, to the ECU 107, which is an electronic control unit of the vehicle.

[0028] FIG. 2 is a diagram for explaining an example of calculation of sequence similarity.

[0029] The top part of Figure 2 shows how a matching block that is part of one image (e.g., the right image) of a pair of images taken by a pair of cameras is compared with a matching block of the same size in the other image (e.g., the left image).

[0030] The image processing device 100 fixes the matching block of one image, and shifts the position of the matching block of the other image in the other image by x1, x2, ..., xn, and sequentially calculates the similarity between the matching block of one image and the matching block of the other image. The image processing device 100 searches for the position of the matching block of the other image with the highest similarity as a corresponding point with the matching block of the one image. Here, for example, ZSAD can be used as the similarity between the matching blocks.

[0031] When searching for the corresponding points, the image processing device 100 compares the luminance array of one image with the luminance array of the other image. The luminance array of one image refers to the arrangement of the luminance of a pixel group at a predetermined position in the matching block of one image. The luminance array of the other image is similar. The pixel group at a predetermined position refers to, for example, a pixel group located at the vertical center in the matching block and arranged along the horizontal direction. However, the position and arrangement of the pixel group in the matching block are not particularly limited. Then, the image processing device 100 calculates n array similarities r1, r2, ..., rn for each position of the matching block of the other image shifted as x1, x2, ..., xn.

[0032] The image processing device 100 calculates a correlation value between the luminance array of one image and the luminance array of the other image as the array similarity. This correlation value may be, for example, a normalized cross-correlation value ZNCC (Zero-means Normalized Cross-Correlation). When the luminance arrays to be compared are 2m+1 pixels, the image processing device 100 can calculate the correlation value expressed by Equation 1 as the array similarity r.

[0033]

number

[0034] In Equation 1, x represents the shift amount (search position) of the matching block of the other image. Lx+i represents the luminance of a pixel that is i pixels to the right (in the shift direction) from the center pixel of the matching block of the other image when the shift amount is x pixels. Rx+i represents the luminance of a pixel that is i pixels to the right from the center pixel of the matching block of one image. Also, in Equation 1, R or L represents the average luminance value in the matching block of one image or the other image, respectively. The correlation value represented by Equation 1 can absorb differences in gain and brightness between a pair of images, and is therefore effective as an index that represents a more accurate sequence similarity.

[0035] FIG. 3 is a diagram for explaining an example of calculating the reliability in the first embodiment.

[0036] The horizontal axis in Fig. 3 represents the shift amount (search position) of the matching block of the other image. The vertical axis at the top of Fig. 3 represents the ZSAD value (similarity used to search for corresponding points between a pair of images) corresponding to the shift amount. The vertical axis at the bottom of Fig. 3 represents the sequence similarity (correlation value expressed by Equation 1) corresponding to the shift amount. The "matching position" shown in Fig. 3 is the position where the ZSAD value is the minimum in the search range and represents the position of the corresponding points between a pair of images. The shift amount at which the ZSAD value is the minimum represents the disparity of the position indicated by the shift amount.

[0037] The image processing device 100 adopts the array similarity at the position indicated by the shift amount where the ZSAD value is the minimum as the reliability Ryc for the disparity. In the example of Fig. 3, the reliability Ryc = 0.8. Due to the characteristics of the correlation value expressed by Equation 1, the larger the value of the reliability Ryc, the more similar the luminance arrays between a pair of images at the matching position are, and the higher the reliability of the disparity at the matching position is.

[0038] FIG. 4 is a flowchart showing a flow of invalid parallax determination in the first embodiment.

[0039] The image processing device 100 executes invalid parallax determination using the invalid parallax determination unit 105. That is, the image processing device 100 determines whether or not the reliability Ryc for the parallax is below a predetermined threshold THyc (step S11). If the reliability Ryc for the parallax is below the predetermined threshold THyc, the image processing device 100 determines the parallax corresponding to the reliability Ryc as invalid (step S12). The parallax determined to be invalid is replaced with invalid information or corrected to the average value of the parallax of the surrounding pixels. On the other hand, if the reliability Ryc for the parallax is equal to or greater than the predetermined threshold THyc, the image processing device 100 determines the parallax corresponding to the reliability Ryc as valid (step S13). The parallax determined to be valid is maintained.

[0040] As described above, the image processing device 100 of the first embodiment includes a search unit 102 that searches for corresponding points between a pair of images, a reliability calculation unit 104 that calculates an array similarity indicating the similarity between a luminance array of a pixel group in one image and a luminance array of a pixel group in the other image within a search range for corresponding points, and calculates a reliability for the displacement amount (parallax) of corresponding points between the pair of images based on the calculated array similarity, and a correction unit 106 that invalidates or corrects the displacement amount (parallax) if the calculated reliability is below a predetermined threshold.

[0041] As a result, the image processing device 100 can calculate the reliability of the displacement amount (parallax) of the corresponding points between a pair of images by using the similarity of the luminance array calculated by comparing the pixel groups between the pair of images, rather than using the similarity or dissimilarity of the luminance calculated by comparing the pixels between the pair of images. Therefore, even if a matching error occurs due to a mismatch in the shooting environment between the pair of images and an erroneous displacement amount (parallax) is calculated, the image processing device 100 can appropriately screen this. In addition, even if a matching error occurs due to low texture of the subject and an erroneous displacement amount (parallax) is calculated, the image processing device 100 can appropriately screen this. Therefore, the image processing device 100 can appropriately screen the displacement amount (parallax) calculated erroneously due to a mismatch in the shooting environment between the pair of images or low texture of the subject.

[0042] Furthermore, in the image processing device 100 of the first embodiment, the reliability calculation unit 104 calculates a correlation value between a luminance array in one image and a luminance array in the other image in the search range as the array similarity, and calculates the reliability based on the calculated correlation value.

[0043] As a result, the image processing device 100 can calculate the sequence similarity using an existing algorithm, and can calculate the reliability relatively easily. Therefore, the image processing device 100 can appropriately and easily screen the displacement amount (parallax) that is erroneously calculated due to a mismatch in the shooting environment between a pair of images or due to low texture of the subject.

[0044] Furthermore, in the image processing device 100 of the first embodiment, the set of images are images captured by a plurality of cameras whose fields of view overlap at least partially, and the amount of displacement is the parallax between the set of images.

[0045] As a result, the image processing device 100 can be applied to a set of images captured by the stereo camera 101 or a multi-camera, and can appropriately screen the displacement amount (disparity) that is erroneously calculated due to a discrepancy in the shooting environment between the set of images or due to low texture of the subject.

[0046] [Second embodiment] An image processing device 100 according to the second embodiment will be described with reference to Fig. 5 and Fig. 6. In the image processing device 100 according to the second embodiment, the description of the same configuration and operation as in the first embodiment will be omitted.

[0047] The reliability calculation unit 104 in the first embodiment calculates the reliability based on a correlation value between the luminance sequence of one image and the luminance sequence of the other image, which is calculated as the sequence similarity. In contrast, the reliability calculation unit 104 in the second embodiment calculates the reliability based on a difference value between the maximum and minimum values ​​of the correlation value.

[0048] Fig. 5 is a diagram for explaining an example of calculating the reliability in the second embodiment, and corresponds to Fig. 3.

[0049] The horizontal axis of Fig. 5 represents the shift amount (search position) of the matching block of the other image that is shifted in block matching. The vertical axis in the upper part of Fig. 5 represents the ZSAD value according to the shift amount. The vertical axis in the lower part of Fig. 5 represents the sequence similarity (correlation value expressed by Equation 1) according to the shift amount.

[0050] The image processing device 100 of the second embodiment adopts the fluctuation range of the sequence similarity in the search range, i.e., the difference between the maximum and minimum correlation values, as the reliability Rtex for the disparity. Due to the characteristics of the correlation value, the smaller the reliability Rtex value, the more pixels having similar luminance are aligned horizontally in each image, i.e., the pixels have low texture, and the lower the reliability of the disparity at the matching position.

[0051] FIG. 6 is a flowchart showing a flow of invalid parallax determination in the second embodiment.

[0052] The image processing device 100 of the second embodiment determines whether or not the reliability Rtex for the disparity is below a predetermined threshold THtex (step S21). If the reliability Rtex for the disparity is below the predetermined threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is invalid (step S22). The disparity determined to be invalid is replaced with invalid information or corrected to the average value of the disparities of the surrounding pixels. On the other hand, if the reliability Rtex for the disparity is equal to or greater than the predetermined threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is valid (step S23). The disparity determined to be valid is maintained.

[0053] As described above, in the image processing device 100 of the second embodiment, the reliability calculation unit 104 calculates the correlation value between the luminance array in one image and the luminance array in the other image in the search range as the array similarity, and calculates the reliability based on the difference value between the maximum and minimum values ​​of the calculated correlation values.

[0054] As a result, even if the displacement amount (parallax) of corresponding points between a pair of images is erroneously calculated due to low texture of the subject, the image processing device 100 of the second embodiment can appropriately and simply screen the erroneously calculated displacement amount (parallax). Therefore, the image processing device 100 of the second embodiment can more appropriately and simply screen the erroneously calculated displacement amount (parallax) due to a mismatch in the shooting environment between the pair of images or due to low texture of the subject.

[0055] [Third embodiment] An image processing device 100 according to the third embodiment will be described with reference to Fig. 7. In the image processing device 100 according to the third embodiment, the description of the same configuration and operation as those of the first or second embodiment will be omitted.

[0056] The reliability calculation unit 104 of the third embodiment calculates the reliability based on both the correlation value between the luminance array of one image and the luminance array of the other image and the difference value between the maximum and minimum values ​​of the correlation value. Specifically, the reliability calculation unit 104 of the third embodiment calculates the first reliability Ryc based on the correlation value, and calculates the second reliability Rtex based on the difference value.

[0057] The invalid disparity determination unit 105 of the third embodiment performs a first determination of determining whether or not the first reliability Ryc is below a first threshold THyc, and a second determination of determining whether or not the second reliability Rtex is below a second threshold THtex.

[0058] The correction unit 106 of the third embodiment invalidates or corrects the disparity when the first reliability Ryc is lower than the first threshold THyc, or when the second reliability Rtex is lower than the second threshold THtex.

[0059] FIG. 7 is a flowchart showing a flow of invalid parallax determination in the third embodiment.

[0060] The image processing device 100 of the third embodiment performs a first determination to determine whether or not a first reliability Ryc for the disparity is below a first threshold THyc (step S31). If the first reliability Ryc is below the first threshold THyc, the image processing device 100 determines that the disparity corresponding to the first reliability Ryc is invalid (step S32). On the other hand, if the first reliability Ryc is equal to or greater than the first threshold THyc, the image processing device 100 performs a second determination to determine whether or not a second reliability Rtex for the disparity is below a second threshold THtex (step S33). If the second reliability Rtex is below the second threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is invalid (step S32). The disparity determined to be invalid is replaced with invalid information or corrected to the average value of the disparities of the surrounding pixels. On the other hand, when the second reliability Rtex is equal to or greater than the second threshold THtex, the image processing device 100 determines that the disparity corresponding to the second reliability Rtex is valid (step S34). The disparity determined to be valid is maintained. In this manner, in the image processing device 100 of the third embodiment, the invalid disparity determination unit 105 performs the first determination in preference to the second determination.

[0061] As described above, in the image processing device 100 of the third embodiment, the reliability calculation unit 104 calculates a correlation value between a luminance sequence in one image and a luminance sequence in the other image in a search range as the sequence similarity, and calculates a difference value between the maximum and minimum values ​​of the calculated correlation value. Then, the reliability calculation unit 104 calculates a first reliability Ryc as the reliability based on the calculated correlation value, and calculates a second reliability Rtex as the reliability based on the calculated difference value. If the first reliability Ryc is lower than the first threshold THyc or if the second reliability Rtex is lower than the second threshold THtex, the correction unit 106 invalidates or corrects the displacement amount (parallax) of corresponding points between a pair of images.

[0062] As a result, the image processing device 100 of the third embodiment can appropriately and easily screen the erroneously calculated displacement amount (parallax) of the corresponding points between a pair of images, when the displacement amount (parallax) is erroneously calculated due to the inconsistency of the shooting environment between the pair of images. Furthermore, even if the displacement amount (parallax) is erroneously calculated due to the low texture of the subject, the image processing device 100 of the third embodiment can appropriately and easily screen the erroneously calculated displacement amount (parallax). Therefore, the image processing device 100 of the third embodiment can more appropriately and easily screen the erroneously calculated displacement amount (parallax) due to the inconsistency of the shooting environment between the pair of images or the low texture of the subject.

[0063] Furthermore, the image processing device 100 of the third embodiment further includes an invalid parallax determination unit 105 that performs a first determination of whether or not the first reliability Ryc is below a first threshold THyc and a second determination of whether or not the second reliability Rtex is below a second threshold THtex. The invalid parallax determination unit 105 performs the first determination in preference to the second determination.

[0064] As a result, the image processing device 100 of the third embodiment can screen the erroneously calculated amount of displacement more efficiently than when the second determination is performed in priority over the first determination, or when the first determination and the second determination are performed in parallel. Therefore, the image processing device 100 of the third embodiment can more appropriately and more simply screen the amount of displacement (parallax) that is erroneously calculated due to a mismatch in the shooting environment between a pair of images or due to low texture of the subject.

[0065] [Fourth embodiment] The image processing device 100 of the fourth embodiment will be described with reference to Fig. 8. In the image processing device 100 of the fourth embodiment, the description of the configuration and operation similar to those of the first to third embodiments will be omitted.

[0066] Fig. 8 is a diagram showing a configuration of a camera system 1 including an image processing device 100 of the fourth embodiment. In Fig. 8, a displacement amount calculation unit corresponding to the parallax calculation unit 103 of the first to third embodiments, an invalid displacement amount determination unit corresponding to the invalid parallax determination unit 105, a correction unit corresponding to the correction unit 106, and an ECU 107 are not shown.

[0067] The camera system 1 of the fourth embodiment includes a monocular camera 801 instead of the stereo camera 101. A current frame image captured by the monocular camera 801 is input to a search unit 802 and also input to and recorded in memory 803. The search unit 802 searches for corresponding points between the current frame image output from the monocular camera 801 and the previous frame image recorded in memory 803. That is, in the image processing device 100 of the fourth embodiment, a set of images are images captured at different times by the same camera (monocular camera 801), one image of the set of images is a current frame image, and the other image of the set of images is a previous frame image.

[0068] The search unit 802 performs a matching process to search for corresponding points between a pair of images, similar to the search unit 102 of the first to third embodiments. The search unit 802 may output similarity information, similar to the search unit 102 of the first to third embodiments, or may output corresponding position information indicating the positions of corresponding points between a pair of images, as shown in FIG. 8. The similarity information or corresponding position information output from the search unit 802 is input to a displacement amount calculation unit and used to calculate the displacement amount of corresponding points between a pair of images. This displacement amount is a time-series displacement amount between a pair of images. Furthermore, the search unit 802 outputs luminance array information to a reliability calculation unit 804, similar to the search unit 102 of the first to third embodiments.

[0069] The reliability calculation unit 804 of the fourth embodiment, like the reliability calculation unit 104 of the first to third embodiments, calculates the reliability of the displacement calculated by the displacement calculation unit based on the luminance array information, and outputs the reliability information. At this time, like the first to third reliability calculation units 104, the reliability calculation unit 804 of the fourth embodiment calculates the reliability Ryc or the reliability Rtex, or both. The reliability information output from the reliability calculation unit 804 of the fourth embodiment is input to an invalid displacement determination unit and is used to determine whether or not the displacement calculated by the displacement calculation unit needs to be invalidated or corrected.

[0070] As described above, in the image processing device 100 of the fourth embodiment, a set of images are images taken at different times by the same camera (monocular camera 801), and the amount of displacement of corresponding points between the set of images is the amount of displacement between the set of images over time.

[0071] As a result, the image processing device 100 of the fourth embodiment can be applied to a set of images captured by a monocular camera 801, and even when calculating a time-series displacement amount between a set of images such as optical flow, it can appropriately screen the displacement amount that is erroneously calculated due to a discrepancy in the shooting environment between the set of images or low texture of the subject.

[0072] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0073] In addition, the above-mentioned configurations, functions, processing units, or processing means may be realized in part or in whole by hardware, for example, by designing them in an integrated circuit. The above-mentioned configurations or functions may be realized by software, in which a processor interprets and executes a program that realizes each function. Information such as a program, table, or file that realizes each function may be stored in a recording device such as a memory, a hard disk, or an SSD (solid state drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0074] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0075] 100: image processing device, 102: search unit, 103: parallax calculation unit, 104: reliability calculation unit, 105: invalid parallax determination unit (determination unit), 106: correction unit, 802: search unit, 804: reliability calculation unit

Claims

1. a search unit for searching for corresponding points between a pair of images; a reliability calculation unit that calculates an array similarity indicating a similarity between an array of luminance in a pixel group in one image and an array of luminance in a pixel group in the other image in a search range for the corresponding points, and calculates a reliability of a displacement amount of the corresponding points between the pair of images based on the calculated array similarity; a correction unit that invalidates or corrects the amount of displacement when the calculated reliability is below a predetermined threshold.

13. An image processing device comprising:

2. The reliability calculation unit is calculating, as the sequence similarity, a correlation value between the sequence of luminance in the one image and the sequence of luminance in the other image in the search range; Calculating the reliability based on the calculated correlation value 2. The image processing device according to claim 1,

3. The reliability calculation unit is calculating, as the sequence similarity, a correlation value between the sequence of luminance in the one image and the sequence of luminance in the other image in the search range; The reliability is calculated based on a difference between the maximum and minimum values ​​of the calculated correlation values.

2. The image processing device according to claim 1,

4. The reliability calculation unit is calculating, as the sequence similarity, a correlation value between the sequence of luminance in the one image and the sequence of luminance in the other image in the search range; Calculating a difference between a maximum value and a minimum value of the calculated correlation value; calculating a first reliability as the reliability based on the calculated correlation value, and calculating a second reliability as the reliability based on the calculated difference value; The correction unit invalidates or corrects the amount of displacement when the first reliability falls below a first threshold or when the second reliability falls below a second threshold.

2. The image processing device according to claim 1,

5. a determination unit that performs a first determination of whether or not the first reliability is below the first threshold value and a second determination of whether or not the second reliability is below the second threshold value, The determination unit performs the first determination in priority to the second determination.

5. The image processing device according to claim 4.

6. the set of images being images taken by a plurality of cameras whose fields of view overlap at least in part; The displacement is the disparity between the pair of images.

2. The image processing device according to claim 1,

7. The set of images are images taken by the same camera at different times, The displacement amount is a time-series displacement amount between the pair of images.

2. The image processing device according to claim 1,

Images