Image processing device

Abstract

The purpose of the present invention is to appropriately screen the amount of displacement of a corresponding point between images erroneously calculated due to mismatch of the image capture environment between a pair of images or low texture of a subject. This image processing device comprises: a search unit that searches for a corresponding point between a pair of images; a reliability calculation unit that calculates, in a search range for corresponding points, an array similarity indicating the degree of similarity between a luminance array for a pixel group in one image and a luminance array for a pixel group in the other image, and calculates reliability with respect to the amount of displacement (parallax) of a corresponding point between the set of images on the basis of the calculated array similarity; and a correction unit that invalidates or corrects the amount of displacement (parallax) when the calculated reliability falls below a predetermined threshold.

Description

Image processing device

[0001] The present invention relates to an image processing device.

[0002] A stereo camera system is known that uses the principle of triangulation to measure the distance to an object within a field of view based on a pair of images captured by a pair of cameras whose fields of view overlap at least partially. The principle of triangulation is to calculate the distance from the camera to the object using the positional shift (amount of displacement) between the images of the same object captured by the pair of cameras, i.e., parallax. Parallax is calculated by searching for the location of the object image in one image on the other image.

[0003] Various methods have been proposed for calculating disparity. For example, a classical method is block matching, which searches for the area with the highest similarity (area with the lowest dissimilarity) between a region consisting of multiple pixels in one image and another image.

[0004] In an ideal shooting environment, accurate disparity can be calculated from a set of images. However, in reality, matching errors may occur between a set of images due to inconsistencies in the shooting environment, such as out-of-focus or lighting, or due to low texture of the subject, such as the road surface, resulting in an erroneous calculation of disparity. This may result in erroneous object recognition, which may hinder the recognition of, for example, fallen objects on the road or the condition of the road surface. Note that low texture means that the texture of the subject is poorly varied, making it difficult to easily detect texture edges in the image of the subject.

[0005] As a measure to detect the position of a subject with high accuracy regardless of the shooting environment, for example, Patent Document 1 is known, which is a method of screening out disparities with low reliability by using a dissimilarity index between a pair of images used to calculate disparity.

[0006] JP 2016-38886 A

[0007] In Patent Document 1, the reliability of disparity is measured by a change in an index representing similarity or dissimilarity, such as Sum of Absolute Difference (SAD), which is used when searching for corresponding points between a pair of images. Specifically, in Patent Document 1, the reliability of disparity is measured by a sharpness indicating the degree to which the maximum value of the index protrudes. As a result, in Patent Document 1, if a matching error with high sharpness occurs due to a mismatch in the shooting environment between a pair of images, it may not be possible to screen disparities with low reliability.

[0008] Furthermore, in Patent Document 1, a dissimilarity index derived from a comparison between single pixels in each image is used as the reliability of the disparity. As a result, in Patent Document 1, the characteristics of the object represented by the arrangement of multiple pixels in each image cannot be reflected in the reliability of the disparity. Therefore, in Patent Document 1, it is difficult to detect matching errors caused by low texture of the object, and even if an erroneous disparity is calculated due to this matching error, there is a possibility that this erroneous disparity cannot be screened.

[0009] This problem applies not only to stereo camera systems, but also to monocular camera systems that search for corresponding points between a set of images taken by the same camera at different times and calculate the amount of displacement of those corresponding points over time.

[0010] The present invention has been made in consideration of the above, and aims to appropriately screen for displacement amounts of corresponding points between a pair of images that are erroneously calculated due to inconsistencies in the shooting environments between the images or low texture of the subject.

[0011] In order to solve the above problem, the image processing device of the present invention is characterized by comprising: a search unit that searches for corresponding points between a pair of images; a reliability calculation unit that calculates an array similarity that indicates the similarity between the luminance array of a pixel group in one image and the luminance array of a pixel group in the other image within the search range for the corresponding points, and calculates a reliability for the amount of displacement of the corresponding points between the pair of images based on the calculated array similarity; and a correction unit that invalidates or corrects the amount of displacement if the calculated reliability is below a predetermined threshold.

[0012] According to the present invention, it is possible to appropriately screen for the amount of displacement between corresponding points between a pair of images that is erroneously calculated due to a mismatch in the shooting environment between the pair of images or due to low texture of the subject. Other problems, configurations, and advantages will become clear from the description of the following embodiments.

[0013] FIG. 10 is a diagram showing the configuration of a camera system equipped with an image processing device of a first embodiment. FIG. 11 is a diagram explaining an example of calculation of sequence similarity. FIG. 12 is a diagram explaining an example of calculation of reliability in the first embodiment. FIG. 13 is a flowchart showing the flow of invalid parallax determination in the first embodiment. FIG. 14 is a diagram explaining an example of calculation of reliability in a second embodiment. FIG. 15 is a flowchart showing the flow of invalid parallax determination in the second embodiment. FIG. 16 is a flowchart showing the flow of invalid parallax determination in a third embodiment. FIG. 17 is a diagram showing the configuration of a camera system equipped with an image processing device of a fourth embodiment.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that components or functions denoted by the same reference numerals in each embodiment have the same components or functions in each embodiment unless otherwise specified, and description thereof will be omitted.

[0015] First Embodiment An image processing device 100 according to a first embodiment will be described with reference to Figures 1 to 4. Figure 1 is a diagram showing the configuration of a camera system 1 equipped with the image processing device 100 according to the first embodiment.

[0016] The camera system 1 is a camera system mounted on a vehicle or the like, and includes a stereo camera 101 and an image processing device 100. The stereo camera 101 is configured with a pair of cameras whose fields of view at least partially overlap. The pair of cameras of the stereo camera 101 simultaneously capture images of the same subject from different viewpoints. When the pair of cameras of the stereo camera 101 is configured with a right camera and a left camera arranged at an interval in the left-right direction of the vehicle, one image of the pair of images is a right image captured by the right camera, and the other image of the pair of images is a left image captured by the left camera. The stereo camera 101 outputs the pair of captured images to the image processing device 100.

[0017] Instead of the stereo camera 101, the camera system 1 may include a multi-camera configured as a set of three or more cameras whose fields of view overlap at least partially with each other.

[0018] The image processing device 100 includes a search unit 102 , a parallax calculation unit 103 , a reliability calculation unit 104 , an invalid parallax determination unit 105 , and a correction unit 106 .

[0019] The image processing device 100 includes a CPU, a ROM, and a RAM. The CPU reads and executes a program stored in the ROM, thereby realizing the functions of a search unit 102, a disparity calculation unit 103, a reliability calculation unit 104, an invalid disparity determination unit 105, and a correction unit 106.

[0020] The search unit 102 performs a matching process to search for corresponding points between a pair of images. For example, the search unit 102 searches for corresponding points using block matching. Specifically, the search unit 102 searches for a small area in the other image of the pair that has the highest similarity to a small area (matching block) consisting of multiple pixels in one of the images, thereby searching for corresponding points. The search unit 102 can use, as the similarity measure, SAD or ZSAD (Zero-Mean Sum of Absolute Difference), calculated based on the luminance difference between corresponding pixels in the pair of images. The search range for corresponding points may be the entire image or a local area. In this embodiment, an example will be described in which the search range for corresponding points is a local search range limited to a portion of the image. The search unit 102 outputs similarity information indicating the similarity for each pixel calculated during the search for corresponding points to the disparity calculation unit 103.

[0021] Furthermore, when searching for corresponding points, the search unit 102 focuses on a group of pixels at a predetermined position within a small region of one image (for example, a group of pixels located in the vertical center of the small region and lined up horizontally) and a group of pixels at a predetermined position within the small region of the other image.The search unit 102 then outputs brightness array information indicating the arrangement of the brightness levels in the group of pixels in one image (hereinafter also referred to as the "brightness array of one image") and the arrangement of the brightness levels in the group of pixels in the other image (hereinafter also referred to as the "brightness array of the other image") to the reliability calculation unit 104.

[0022] The disparity calculation unit 103 calculates the disparity between the pair of images (i.e., the amount of displacement between corresponding points between the pair of images) for each pixel based on the similarity information output from the search unit 102. The disparity calculation unit 103 outputs disparity information indicating the calculated disparity to the invalid disparity determination unit 105.

[0023] The reliability calculation unit 104 calculates the reliability of the disparity calculated by the disparity calculation unit 103. In this case, the reliability calculation unit 104 calculates an array similarity indicating the similarity between the luminance array of one image and the luminance array of the other image in the search range for corresponding points, and calculates the reliability of the disparity based on the calculated array similarity. An example of calculating the array similarity will be described later with reference to FIG. 2.

[0024] Specifically, the reliability calculation unit 104 calculates a correlation value between the luminance array of one image and the luminance array of the other image based on the luminance array information output from the search unit 102. The reliability calculation unit 104 sets the calculated correlation value as array similarity. The reliability calculation unit 104 calculates the reliability based on the calculated correlation value. The reliability calculation unit 104 outputs reliability information indicating the calculated reliability to the invalid parallax determination unit 105. An example of calculating the reliability will be described later with reference to FIG. 3 .

[0025] The function of calculating the sequence similarity may be included in the search unit 102, rather than in the reliability calculation unit 104. That is, the search unit 102 may calculate the similarity for each pixel between a pair of images to search for corresponding points between the images, and may also calculate the sequence similarity between the pair of images. Then, the reliability calculation unit 104 may calculate the reliability of the disparity based on the sequence similarity calculated by the search unit 102.

[0026] The invalid parallax determination unit 105 determines which parallaxes are to be invalid among the parallaxes indicated by the parallax information output from the parallax calculation unit 103, based on the reliability information output from the reliability calculation unit 104. Specifically, the invalid parallax determination unit 105 determines whether or not each reliability for each parallax in the search range is below a predetermined threshold. The invalid parallax determination unit 105 determines a parallax whose reliability is below the threshold to be invalid, and determines a parallax whose reliability is equal to or greater than the threshold to be valid. The invalid parallax determination unit 105 outputs the determination result to the correction unit 106.

[0027] The correction unit 106 maintains the value of the disparity determined to be valid by the invalid parallax determination unit 105. The correction unit 106 invalidates the disparity determined to be invalid by the invalid parallax determination unit 105 by replacing it with invalid information. Alternatively, the correction unit 106 corrects the disparity of the target pixel having the disparity determined to be invalid by the invalid parallax determination unit 105 to a reliable value using the disparities of pixels surrounding the target pixel. For example, the correction unit 106 corrects the disparity of the target pixel to the average value of the disparities of the surrounding pixels. The correction unit 106 outputs corrected disparity information, which is disparity information obtained by invalidating or correcting the disparity determined to be invalid by the invalid parallax determination unit 105, to the ECU 107, which is an electronic control unit of the vehicle.

[0028] FIG. 2 is a diagram illustrating an example of calculation of sequence similarity.

[0029] The top part of Figure 2 shows how a matching block that is part of one image (e.g., the right image) of a set of images taken by a pair of cameras is compared with a matching block of the same size in the other image (e.g., the left image).

[0030] The image processing device 100 fixes the matching block of one image and sequentially calculates the similarity between the matching block of the other image and the matching block of the one image while shifting the position of the matching block in the other image by x1, x2, ..., xn. The image processing device 100 searches for the position of the matching block of the other image that has the highest similarity as a corresponding point with the matching block of the one image. Here, the similarity between the matching blocks can be calculated using, for example, ZSAD.

[0031] When searching for corresponding points, the image processing device 100 compares the luminance array of one image with the luminance array of the other image. The luminance array of one image refers to the arrangement of luminance values ​​of a group of pixels at a predetermined position in the matching block of the one image. The luminance array of the other image is similar. The group of pixels at a predetermined position refers, for example, to a group of pixels located in the vertical center of the matching block and arranged horizontally. However, the position and arrangement of the group of pixels within the matching block are not particularly limited. The image processing device 100 then calculates n array similarities r1, r2, ..., rn for each position of the matching block of the other image, which is shifted as x1, x2, ..., xn.

[0032] The image processing device 100 calculates a correlation value between the luminance array of one image and the luminance array of the other image as the array similarity. This correlation value may be, for example, a normalized cross-correlation value ZNCC (Zero-means Normalized Cross-Correlation). When the luminance arrays to be compared have 2m+1 pixels, the image processing device 100 can calculate the correlation value expressed by Equation 1 as the array similarity r.

[0033]

[0034] In Equation 1, x represents the shift amount (search position) of the matching block of the other image. Lx+i represents the luminance of the pixel i pixels to the right (shift direction) from the center pixel of the matching block of the other image when the shift amount is x pixels. Rx+i represents the luminance of the pixel i pixels to the right from the center pixel of the matching block of one image. Also, in Equation 1, R or L represents the average luminance value within the matching block of one image or the other image, respectively. The correlation value represented by Equation 1 can absorb differences in gain and brightness between a pair of images, making it effective as an index representing more accurate sequence similarity.

[0035] FIG. 3 is a diagram illustrating an example of calculating the reliability in the first embodiment.

[0036] The horizontal axis in Figure 3 represents the shift amount (search position) of the matching block of the other image. The vertical axis at the top of Figure 3 represents the ZSAD value (similarity used to search for corresponding points between a pair of images) corresponding to the shift amount. The vertical axis at the bottom of Figure 3 represents the sequence similarity (correlation value expressed by Equation 1) corresponding to the shift amount. The "matching position" shown in Figure 3 indicates the minimum ZSAD value in the search range and represents the position of corresponding points between a pair of images. The shift amount at which the ZSAD value is minimum represents the disparity of the position indicated by the shift amount.

[0037] The image processing device 100 uses the array similarity at the position indicated by the shift amount that minimizes the ZSAD value as the reliability Ryc for the disparity. In the example of Figure 3, the reliability Ryc = 0.8. Due to the characteristics of the correlation value expressed by Equation 1, a larger value of the reliability Ryc indicates a greater similarity in the luminance arrays between a pair of images at the matching position, and thus indicates a higher reliability of the disparity at the matching position.

[0038] FIG. 4 is a flowchart showing the flow of invalid parallax determination in the first embodiment.

[0039] The image processing device 100 performs invalid parallax determination using the invalid parallax determination unit 105. That is, the image processing device 100 determines whether the reliability Ryc for the parallax is below a predetermined threshold THyc (step S11). If the reliability Ryc for the parallax is below the predetermined threshold THyc, the image processing device 100 determines the parallax corresponding to the reliability Ryc as invalid (step S12). The parallax determined to be invalid is replaced with invalid information or corrected to the average value of the parallaxes of the surrounding pixels. On the other hand, if the reliability Ryc for the parallax is equal to or greater than the predetermined threshold THyc, the image processing device 100 determines the parallax corresponding to the reliability Ryc as valid (step S13). The parallax determined to be valid is maintained.

[0040] As described above, the image processing device 100 of the first embodiment includes a search unit 102 that searches for corresponding points between a pair of images, a reliability calculation unit 104 that calculates array similarity indicating the similarity between the luminance array of a pixel group in one image and the luminance array of a pixel group in the other image within the search range for corresponding points, and calculates a reliability for the displacement amount (parallax) of the corresponding points between the pair of images based on the calculated array similarity, and a correction unit 106 that invalidates or corrects the displacement amount (parallax) if the calculated reliability is below a predetermined threshold.

[0041] As a result, the image processing device 100 can calculate the reliability of the displacement (parallax) between corresponding points between a pair of images using the similarity of the luminance arrays calculated by comparing groups of pixels between the pair of images, rather than using the luminance similarity or dissimilarity calculated by comparing pixels between the pair of images. Therefore, even if a matching error occurs due to a mismatch in the shooting environment between the pair of images and an erroneous calculation of the displacement (parallax), the image processing device 100 can appropriately screen for this. In addition, even if a matching error occurs due to low texture of the subject and an erroneous calculation of the displacement (parallax), the image processing device 100 can appropriately screen for this. Therefore, the image processing device 100 can appropriately screen for an erroneous calculation of the displacement (parallax) due to a mismatch in the shooting environment between the pair of images or low texture of the subject.

[0042] Furthermore, in the image processing device 100 of the first embodiment, the reliability calculation unit 104 calculates the correlation value between the luminance array in one image and the luminance array in the other image in the search range as the array similarity, and calculates the reliability based on the calculated correlation value.

[0043] This allows the image processing device 100 to calculate the sequence similarity using an existing algorithm, making it possible to calculate the reliability relatively easily. Therefore, the image processing device 100 can appropriately and easily screen for the displacement amount (parallax) that has been erroneously calculated due to a mismatch in the shooting environment between a pair of images or low texture of the subject.

[0044] Furthermore, in the image processing device 100 of the first embodiment, the set of images are images taken by a plurality of cameras whose fields of view overlap at least partially, and the amount of displacement is the parallax between the set of images.

[0045] As a result, the image processing device 100 can be applied to a set of images captured by a stereo camera 101 or a multi-camera, and can appropriately screen for the displacement (parallax) that is incorrectly calculated due to inconsistencies in the shooting environments between the set of images or low texture of the subject.

[0046] Second Embodiment An image processing device 100 according to a second embodiment will be described with reference to Figures 5 and 6. In the image processing device 100 according to the second embodiment, the description of the same configuration and operation as in the first embodiment will be omitted.

[0047] The reliability calculation unit 104 in the first embodiment calculates the reliability based on a correlation value between the luminance sequence of one image and the luminance sequence of the other image, which is calculated as the sequence similarity. In contrast, the reliability calculation unit 104 in the second embodiment calculates the reliability based on a difference value between the maximum and minimum values ​​of the correlation value.

[0048] 5 is a diagram illustrating an example of calculating the reliability in the second embodiment, and corresponds to FIG.

[0049] The horizontal axis of Fig. 5 represents the shift amount (search position) of the matching block of the other image that is shifted in block matching. The vertical axis at the top of Fig. 5 represents the ZSAD value corresponding to the shift amount. The vertical axis at the bottom of Fig. 5 represents the sequence similarity (correlation value expressed by Equation 1) corresponding to the shift amount.

[0050] The image processing device 100 of the second embodiment uses the fluctuation range of the sequence similarity in the search range, i.e., the difference between the maximum and minimum correlation values, as the reliability Rtex for the disparity. Due to the characteristics of the correlation value, the smaller the reliability Rtex, the more pixels with similar brightness are aligned horizontally in each image, i.e., the lower the texture of the pixels, and the lower the reliability of the disparity at the matching position.

[0051] FIG. 6 is a flowchart showing the flow of invalid parallax determination in the second embodiment.

[0052] The image processing device 100 of the second embodiment determines whether the reliability Rtex for the disparity is below a predetermined threshold THtex (step S21). If the reliability Rtex for the disparity is below the predetermined threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is invalid (step S22). A disparity determined to be invalid is replaced with invalid information or corrected to the average value of the disparities of the surrounding pixels. On the other hand, if the reliability Rtex for the disparity is equal to or greater than the predetermined threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is valid (step S23). A disparity determined to be valid is maintained.

[0053] As described above, in the image processing device 100 of the second embodiment, the reliability calculation unit 104 calculates the correlation value between the luminance array in one image and the luminance array in the other image within the search range as the array similarity, and calculates the reliability based on the difference between the maximum and minimum values ​​of the calculated correlation values.

[0054] As a result, even if the displacement amount (parallax) of corresponding points between a pair of images is erroneously calculated due to low texture of the subject despite the high degree of alignment similarity between the pair of images, the image processing device 100 of the second embodiment can appropriately and simply screen out the erroneously calculated displacement amount (parallax). Thus, the image processing device 100 of the second embodiment can more appropriately and simply screen out the erroneously calculated displacement amount (parallax) due to a mismatch in the shooting environment between the pair of images or low texture of the subject.

[0055] Third Embodiment An image processing device 100 according to a third embodiment will be described with reference to Fig. 7. In the image processing device 100 according to the third embodiment, the description of the same configuration and operation as those of the first or second embodiment will be omitted.

[0056] The reliability calculation unit 104 of the third embodiment calculates reliability based on both a correlation value between the luminance array of one image and the luminance array of the other image and a difference value between the maximum and minimum values ​​of the correlation value. Specifically, the reliability calculation unit 104 of the third embodiment calculates a first reliability Ryc based on the correlation value and calculates a second reliability Rtex based on the difference value.

[0057] The invalid parallax determination unit 105 of the third embodiment performs a first determination to determine whether the first reliability Ryc is lower than the first threshold value THyc, and a second determination to determine whether the second reliability Rtex is lower than the second threshold value THtex.

[0058] The correction unit 106 of the third embodiment invalidates or corrects the disparity when the first reliability Ryc is lower than the first threshold THyc or when the second reliability Rtex is lower than the second threshold THtex.

[0059] FIG. 7 is a flowchart showing the flow of invalid parallax determination in the third embodiment.

[0060] The image processing device 100 of the third embodiment performs a first determination to determine whether a first reliability Ryc for a disparity is below a first threshold THyc (step S31). If the first reliability Ryc is below the first threshold THyc, the image processing device 100 determines that the disparity corresponding to the first reliability Ryc is invalid (step S32). On the other hand, if the first reliability Ryc is equal to or greater than the first threshold THyc, the image processing device 100 performs a second determination to determine whether a second reliability Rtex for the disparity is below a second threshold THtex (step S33). If the second reliability Rtex is below the second threshold THtex, the image processing device 100 determines that the disparity corresponding to the reliability Rtex is invalid (step S32). A disparity determined to be invalid is replaced with invalid information or corrected to the average value of the disparities of surrounding pixels. On the other hand, if the second reliability Rtex is equal to or greater than the second threshold THtex, the image processing device 100 determines that the disparity corresponding to the second reliability Rtex is valid (step S34). The disparity determined to be valid is maintained. In this way, in the image processing device 100 of the third embodiment, the invalid disparity determination unit 105 performs the first determination in priority to the second determination.

[0061] As described above, in the image processing device 100 of the third embodiment, the reliability calculation unit 104 calculates a correlation value between a luminance sequence in one image and a luminance sequence in the other image within a search range as the sequence similarity, and calculates a difference between the maximum and minimum values ​​of the calculated correlation values. The reliability calculation unit 104 then calculates a first reliability Ryc as the reliability based on the calculated correlation value, and calculates a second reliability Rtex as the reliability based on the calculated difference value. If the first reliability Ryc is lower than the first threshold THyc or if the second reliability Rtex is lower than the second threshold THtex, the correction unit 106 invalidates or corrects the displacement (parallax) of corresponding points between a pair of images.

[0062] As a result, the image processing device 100 of the third embodiment can appropriately and easily screen out the erroneously calculated displacement amount (parallax) between corresponding points in a pair of images, even if the displacement amount (parallax) is erroneously calculated due to a mismatch in the shooting environment between the pair of images. Furthermore, the image processing device 100 of the third embodiment can appropriately and easily screen out the erroneously calculated displacement amount (parallax), even if the displacement amount (parallax) is erroneously calculated due to a low texture of the subject, despite the high degree of sequence similarity between the pair of images. Therefore, the image processing device 100 of the third embodiment can more appropriately and easily screen out the erroneously calculated displacement amount (parallax) due to a mismatch in the shooting environment between the pair of images or a low texture of the subject.

[0063] Furthermore, the image processing device 100 of the third embodiment further includes an invalid parallax determination unit 105 that performs a first determination of whether the first reliability Ryc is lower than a first threshold THyc and a second determination of whether the second reliability Rtex is lower than a second threshold THtex. The invalid parallax determination unit 105 performs the first determination in preference to the second determination.

[0064] As a result, the image processing device 100 of the third embodiment can more efficiently screen for erroneously calculated displacement amounts than when the second determination is performed in priority over the first determination or when the first determination and the second determination are performed in parallel. Therefore, the image processing device 100 of the third embodiment can more appropriately and simply screen for erroneously calculated displacement amounts (parallax) resulting from a mismatch in the shooting environments between a pair of images or low texture of the subject.

[0065] [Fourth Embodiment] An image processing device 100 according to a fourth embodiment will be described with reference to Fig. 8. In the image processing device 100 according to the fourth embodiment, the description of the configuration and operation similar to those of the first to third embodiments will be omitted.

[0066] Fig. 8 is a diagram showing the configuration of a camera system 1 including an image processing device 100 according to the fourth embodiment. In Fig. 8, a displacement amount calculation unit corresponding to the parallax calculation unit 103 in the first to third embodiments, an invalid displacement amount determination unit corresponding to the invalid parallax determination unit 105, a correction unit corresponding to the correction unit 106, and an ECU 107 are not shown.

[0067] The camera system 1 of the fourth embodiment includes a monocular camera 801 instead of the stereo camera 101. The current frame image captured by the monocular camera 801 is input to a search unit 802 and also input to and recorded in memory 803. The search unit 802 searches for corresponding points between the current frame image output from the monocular camera 801 and the previous frame image recorded in memory 803. That is, in the image processing device 100 of the fourth embodiment, a set of images are images captured at different times by the same camera (monocular camera 801), one image of the set of images is the current frame image, and the other image of the set of images is the previous frame image.

[0068] The search unit 802 performs a matching process to search for corresponding points between a set of images, similar to the search unit 102 of the first to third embodiments. The search unit 802 may output similarity information, similar to the search unit 102 of the first to third embodiments, or may output corresponding position information indicating the positions of corresponding points between a set of images, as shown in FIG. 8. The similarity information or corresponding position information output from the search unit 802 is input to a displacement amount calculation unit and used to calculate the displacement amount of corresponding points between a set of images. This displacement amount is the time-series displacement amount between the set of images. Furthermore, the search unit 802 outputs luminance array information to a reliability calculation unit 804, similar to the search unit 102 of the first to third embodiments.

[0069] The reliability calculation unit 804 of the fourth embodiment, like the reliability calculation unit 104 of the first to third embodiments, calculates the reliability of the displacement calculated by the displacement calculation unit based on the luminance array information and outputs the reliability information. At this time, like the first to third reliability calculation units 104, the reliability calculation unit 804 of the fourth embodiment calculates the reliability Ryc or the reliability Rtex, or both. The reliability information output from the reliability calculation unit 804 of the fourth embodiment is input to an invalid displacement determination unit and used to determine whether or not the displacement calculated by the displacement calculation unit needs to be invalidated or corrected.

[0070] As described above, in the image processing device 100 of the fourth embodiment, a set of images are images taken at different times by the same camera (monocular camera 801), and the amount of displacement of corresponding points between the set of images is the amount of displacement over time between the set of images.

[0071] As a result, the image processing device 100 of the fourth embodiment can be applied to a set of images captured by a monocular camera 801, and even when calculating the amount of time-series displacement between a set of images, such as optical flow, it can appropriately screen for the amount of displacement that has been erroneously calculated due to inconsistencies in the shooting environments between the set of images or low texture of the subject.

[0072] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0073] Furthermore, the above-described configurations, functions, processing units, or processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations or functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, or file that implements each function can be stored in a storage device such as a memory, a hard disk, or an SSD (solid state drive), or in a storage medium such as an IC card, an SD card, or a DVD.

[0074] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0075] 100... image processing device, 102... search unit, 103... parallax calculation unit, 104... reliability calculation unit, 105... invalid parallax determination unit (determination unit), 106... correction unit, 802... search unit, 804... reliability calculation unit

Claims

1. An image processing device comprising: a search unit that searches for corresponding points between a pair of images; a reliability calculation unit that calculates an array similarity indicating the similarity between the luminance array of a pixel group in one image and the luminance array of a pixel group in the other image within the search range for the corresponding points, and calculates a reliability for the amount of displacement of the corresponding points between the pair of images based on the calculated array similarity; and a correction unit that invalidates or corrects the amount of displacement if the calculated reliability is below a predetermined threshold.

2. The image processing device according to claim 1, characterized in that the reliability calculation unit calculates, as the array similarity, a correlation value between the luminance array in one of the images and the luminance array in the other image in the search range, and calculates the reliability based on the calculated correlation value.

3. The image processing device according to claim 1, characterized in that the reliability calculation unit calculates, as the array similarity, a correlation value between the luminance array in one of the images and the luminance array in the other image in the search range, and calculates the reliability based on a difference value between a maximum value and a minimum value of the calculated correlation value.

4. The image processing device according to claim 1, characterized in that the reliability calculation unit calculates, as the array similarity, a correlation value between the luminance array in one of the images and the luminance array in the other image in the search range, calculates a difference value between a maximum value and a minimum value of the calculated correlation value, calculates a first reliability as the reliability based on the calculated correlation value, and calculates a second reliability as the reliability based on the calculated difference value, and the correction unit invalidates or corrects the amount of displacement when the first reliability is below a first threshold or when the second reliability is below a second threshold.

5. An image processing device as described in claim 4, further comprising a judgment unit that performs a first judgment to judge whether the first reliability is below the first threshold value and a second judgment to judge whether the second reliability is below the second threshold value, wherein the judgment unit performs the first judgment in priority to the second judgment.

6. The image processing device according to claim 1, characterized in that the set of images are images taken by a plurality of cameras whose fields of view overlap at least partially, and the amount of displacement is the parallax between the set of images.

7. The image processing device according to claim 1, characterized in that the set of images are images taken at different times by the same camera, and the amount of displacement is a time-series amount of displacement between the set of images.

Images