Image processing device and image processing method

Abstract

Provided are an image processing device and an image processing method that enable highly accurate disparity detection using a high-resolution image and a low-resolution image in stereo vision using two cameras with different resolutions. The image processing device comprises: an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher resolution with respect to pixels than the first image; a first compression unit that generates a first compressed image by compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are arranged; a first image shift unit that shifts the first compressed image in the first direction by a predetermined number of pixels; a second compression unit that generates a second compressed image by compressing the shifted first compressed image in the first direction; and a disparity detection unit that compares the first image and the second compressed image to obtain a disparity.

Description

Image processing device and image processing method

[0001] The present invention relates to an image processing device and an image processing method.

[0002] A stereo image processing device (hereinafter referred to as a stereo camera) is known as a device for recognizing objects three-dimensionally. A stereo camera uses the differences in how images are captured by multiple cameras placed at different positions to detect parallax between multiple images based on trigonometry, and then uses the parallax to detect the depth and position of an object, thereby enabling accurate detection of the position of the target object.

[0003] Stereo cameras are used in a wide range of fields, including automotive and robotics. While conventional stereo cameras generally use the same camera, stereo vision is also performed using cameras with different lenses and sensors. Different cameras have different image resolutions relative to pixels due to differences in the image size on the sensor caused by differences in the focal length of the lens and differences in the pixel size of the sensor. In stereo vision using cameras with different resolutions, the challenge is to find a method for matching high-resolution images, which have a relatively high pixel resolution, with low-resolution images, which have a relatively low pixel resolution.

[0004] Patent Documents 1 and 2 propose methods for matching high-resolution images with low-resolution images. Patent Document 1 describes the problem as "an image capturing device having a plurality of cameras capturing images with different spatial resolutions, and forming a distance image by measuring the distance to a subject, to improve the spatial resolution of the distance image," and the solution as "an image capturing device according to one aspect of the disclosed technology is an image capturing device including: a camera capturing a first image having a first spatial resolution in a predetermined field of view; and a camera capturing a second image having a second spatial resolution lower than the first spatial resolution, the camera being arranged at a predetermined interval in a predetermined array direction; the image capturing device forms a distance image from the distance to the subject measured based on the first and second images, the distance image having the first spatial resolution in a direction intersecting the array direction." Furthermore, Patent Document 2 states that the problem to be solved is to provide a three-dimensional structure estimation device that can estimate the three-dimensional structure of an object within the field of view in greater detail while providing a wide-field image and a high-resolution image for part of the field of view. The solution is described as follows: "The camera 3, 4 has a wide-field cone 1 and a narrow-field cone 2 of different fields of view, and generates images with different resolutions. The camera 3, 4 outputs wide-field image 5 and narrow-field image 6, and these images are output from the camera 3, 4. The wide-field cone 1 and narrow-field cone 2 have different fields of view, and generate images with different resolutions. The conversion unit converts the images to match the wide-field image 5 and narrow-field image 6 for stereo calculations. The conversion unit includes a thinned-out sample unit 7 that thins out the pixels of the narrow-field image 6 to generate and output a pixel-by-pixel matched image 8 in which the pixels are matched, and a depth image creation unit 9 that inputs and directly compares the matched wide-field image 5 and pixel-by-pixel matched image 8, calculates the depth of the image, and creates and outputs a depth image 11. Furthermore, the thinned-out sample unit generates multiple images with matched pixels for each small movement, enabling more precise depth estimation."

[0005] JP2019-158776A JP10-134187A

[0006] In Patent Literature 1, disparity between high-resolution and low-resolution images is detected by converting a low-resolution image into a high-resolution image. On the other hand, because increasing the resolution of a low-resolution image with low information content does not increase the amount of information, the accuracy of disparity estimation follows the low-resolution image. Patent Literature 2 proposes shifting a high-resolution image and then thinning or compressively interpolating the image to match it with the low-resolution image, and then evaluating the similarity. However, because the thinned high-resolution image differs from the low-resolution image detected by the camera, highly accurate disparity detection is difficult. Furthermore, compressive interpolation can obtain a disparity image of a low-resolution image, but can also detect disparity using a high-resolution image, thereby achieving highly accurate disparity detection. However, compressive interpolation requires two-dimensional compression and interpolation for each process, which increases the processing scale.

[0007] The present invention has been made in consideration of the above-mentioned problems, and provides an image processing device and an image processing method that are capable of highly accurate disparity detection using high-resolution and low-resolution images in stereoscopic vision using two cameras with different resolutions.

[0008] The image processing device of the present invention includes an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression unit that compresses the second image in a second direction that is approximately perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift unit that shifts the first compressed image by a predetermined number of pixels in the first direction; a second compression unit that compresses the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation unit that compares the first image with the second compressed image to determine a first similarity; and a disparity detection unit that determines a disparity based on the first similarity determined by the first similarity evaluation unit.

[0009] The image processing method of the present invention is an image processing method in which a computer executes the following steps: an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression process for compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift process for shifting the first compressed image in the first direction by a predetermined number of pixels; a second compression process for compressing the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation process for comparing the first image and the second compressed image to determine a first similarity; and a disparity detection process for determining a disparity based on the first similarity determined by the first similarity evaluation process.

[0010] According to the image processing device and image processing method of the present invention, it is possible to provide an image processing device and image processing method that are capable of highly accurate parallax detection using a high-resolution image and a low-resolution image.

[0011] FIG. 1 is a block diagram illustrating a configuration of an image processing device according to a first embodiment; FIG. 2 is a block diagram illustrating a hardware configuration of the image processing device according to the first embodiment; FIG. 3 is a block diagram illustrating a parallax detection process of a stereo parallax image generation unit of a conventional image processing device that serves as a reference for the first embodiment; FIG. 4 is a diagram illustrating a parallax detection method of a conventional image processing device that serves as a reference for the first embodiment; FIG. 5 is a block diagram illustrating a parallax detection process of a stereo parallax image generation unit of the image processing device according to the first embodiment; FIG. 6 is a diagram illustrating a parallax detection method of the image processing device according to the first embodiment; FIG. 7 is a block diagram illustrating a parallax detection process of a stereo parallax image generation unit of an image processing device according to a second embodiment; FIG. 8 is a diagram illustrating a parallax detection method of the image processing device according to the second embodiment; and FIG. 9 is a block diagram illustrating another parallax detection process of a stereo parallax image generation unit of the image processing device according to the second embodiment.

[0012] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings illustrate embodiments in accordance with the principles of the present disclosure, but these drawings are for the purpose of understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0013] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0014] 1A , the configuration of a stereo image processing device 50 (hereinafter referred to as image processing device 50) according to Example 1 will be described. This image processing device 50 is mounted on a vehicle such as an automobile, and is used to detect the distance from the vehicle to three-dimensional objects (other automobiles, buildings, pedestrians, etc.) around the vehicle. In the following, a case where the image processing device 50 is mounted on a vehicle will be described as an example, but the present invention is not limited to this.

[0015] 1A is a block diagram showing an example of the configuration of an image processing device 50 according to Example 1. This image processing device 50 is configured to detect surrounding three-dimensional objects based on images obtained by cameras 1 and 2, and to issue an alarm as necessary. Cameras 1 and 2 form a stereo camera.

[0016] The image processing device 50 of this embodiment is configured to include, for example, an image processing unit 100, a stereo parallax image generating unit 200, a three-dimensional object detecting unit 600, and an alarm control unit 700.

[0017] In this image processing device 50, in an area that can be captured in common by camera 1 and camera 2 (hereinafter referred to as the "stereo viewing area"), a stereo parallax image generating unit 200 generates a parallax image using the parallax between camera 1 and camera 2. Then, a three-dimensional object detecting unit 600 measures the distance from the vehicle to the three-dimensional object according to the parallax.

[0018] The image processing unit 100 is configured to include, for example, an image acquisition unit 20, affine processing units 20a and 20b, luminance correction units 21a and 21b, pixel interpolation units 22a and 22b, and luminance information generation units 23a and 23b. The image processing unit 100 applies predetermined image processing to generate parallax images to the images P1 and P2 obtained by the cameras 1 and 2, and supplies the images to the stereo parallax image generation unit 200.

[0019] Although not shown, cameras 1 and 2 each include a lens and an image sensor. Cameras 1 and 2 each capture (take) an image of an object via the lens using the image sensor. Cameras 1 and 2 are arranged side by side in a predetermined direction so that they overlap (share) a portion of their field of view (image capture area). Image capture unit 20 captures image P1 from camera 1 and image P2 from camera 2.

[0020] The affine processing unit 20a performs affine processing on image P1 from camera 1. As a result of this affine processing, the affine processing unit 20a obtains image P3. Similarly, the affine processing unit 20b performs affine processing on image P2 from camera 2 to obtain image P4. Note that the affine processing units 20a and 20b may also perform distortion transformation processing other than affine processing. In this embodiment, f sin θ, a projection method of a fisheye lens, is projected into a coordinate system of (f tan θx, f tan θy). Here, f is the focal length of the lens, θ is the angle of view incident on the lens, and θx and θx are the horizontal and vertical components of the angle of view incident on the lens.

[0021] The luminance correction unit 21a corrects the luminance of each pixel in the image P3. For example, the luminance correction of each pixel in the image P3 is performed based on the gain of the camera 1, the difference in gain between each pixel in the image P3, etc. Similarly, the luminance correction unit 21b corrects the luminance of each pixel in the image P4.

[0022] The pixel interpolation unit 22a performs demosaicing on the image P3, for example, converting the raw image into a color image. Similarly, the pixel interpolation unit 22b performs demosaicing on the image P4.

[0023] The luminance information generator 23a generates luminance information for the image P3. For example, the luminance information generator 23a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generator 23b generates luminance information for the image P4.

[0024] The stereo parallax image generating unit 200 generates parallax images of the above-mentioned stereo viewing area (common viewing area) using the obtained images P3 and P4 (details will be explained later).

[0025] The three-dimensional object detection unit 600 detects three-dimensional objects in the stereo vision area according to the parallax images obtained by the stereo parallax image generation unit 200. Furthermore, stereo matching is applied to the detected three-dimensional objects to detect parallax and identify the type of three-dimensional object (pedestrian, bicycle, vehicle, building, etc.). By identifying pedestrians, bicycles, vehicles, etc. among the three-dimensional objects, the type to be used for preventive safety is further identified. Vehicle detection results are used for tracking control of a preceding vehicle and emergency braking control. In the case of pedestrians or bicycles, the detection is basically used for emergency braking, particularly for warnings and vehicle control of pedestrians and bicycles that suddenly appear. Compared to stationary objects, warnings and control are implemented for objects that suddenly appear within a wider angle of view. By measuring the distance to these detected objects and estimating the moving speed of the objects being tracked over time, the warning control unit 700 implements more appropriate warnings and control.

[0026] FIG. 1B is a block diagram illustrating an example of the hardware configuration of an image processing apparatus 50 according to the first embodiment.

[0027] The image processing device 50 of this embodiment includes a memory 121 and an arithmetic unit 122 configured with a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), etc. The image processing device 50 also includes a storage device 123 such as an HD (Hard Disk) or an SSD (Solid State Drive). The image processing device 50 also includes an input device 124, an output device 125, and a communication device 126 that communicates with cameras 1 and 2.

[0028] A program is stored in the storage device 123, and this program is loaded into the memory 121. The loaded program is then executed by the arithmetic unit 122. This embodies the units 100, 20, 20a to 23a, 20b to 23b, 200, 600, and 700 shown in FIG. 1A.

[0029] 2 shows the parallax detection process of a conventional stereo parallax image generating unit 200. Here, the description will be given assuming that image P3 obtained from image P1 and image P4 obtained from image P2 have the same resolution.

[0030] The stereo parallax image generating unit 200 shown in FIG. 2 includes an image shifting unit 210 , a similarity evaluation unit 220 , and a parallax detection unit 230 .

[0031] In this stereo parallax image generation unit 200, an image shifter 210 shifts image P4 by a predetermined number of pixels in the direction in which the two cameras are aligned. Then, a similarity evaluation unit 220 evaluates the similarity between image P4 after the shift and image P3. Here, a case will be described in which similarity is evaluated using SAD (Sum of Absolute Difference) of template matching as an evaluation method, but other methods such as SSD (Sum of Squared Difference) may also be used.

[0032] After the evaluation by SAD, the image shifting unit 210 shifts the pixels (for example, by one pixel from the previous processing), and the similarity evaluation unit 220 performs evaluation by SAD. This is repeated a predetermined number of times, and the disparity detection unit 230 finds the disparity Dl that minimizes the SAD.

[0033] FIG. 3 shows an example of a method for detecting the disparity Dl. Here, the explanation will be given assuming that the shift amount in the image shifting unit 210 during repeated evaluation is one pixel. The explanation will also be given assuming that the start pixel and end pixel in the image shifting unit 210 during repeated evaluation, i.e., the pixel shift range between the start pixel and the end pixel, are arbitrarily set. As shown in FIG. 3 , pixels are shifted one pixel at a time, and the minimum value is found based on the evaluation result of the SAD. Here, the four points with the smallest SAD are detected, and the intersection of the lines connecting two of these points is taken as the disparity Dl, thereby making it possible to detect sub-pixel disparity.

[0034] Next, the parallax detection process of the stereo parallax image generating unit 200 of this embodiment will be described with reference to FIG. 4. Here, the description will be made assuming that image P3 obtained from image P1 and image P4 obtained from image P2 have different resolutions. Image P3 will be described assuming that it has a lower pixel resolution than image P4, and image P4 will be described assuming that it has a higher pixel resolution than image P3. For example, image P3 is composed of images captured at a wider angle and at a shorter distance (closer) than image P4, and image P4 is composed of images captured at a narrower angle and at a longer distance (farther) than image P3.

[0035] The stereo parallax image generating unit 200 shown in FIG. 4 includes a one-dimensional compression unit 310 , an image shifting unit 210 , a one-dimensional compression unit 320 , a similarity evaluation unit 220 , and a parallax detection unit 230 .

[0036] The one-dimensional compression unit 310 compresses image P4 input from the image processing unit 100 in a direction (hereinafter referred to as the vertical direction) approximately perpendicular to the direction in which cameras 1 and 2 are aligned (hereinafter referred to as the horizontal direction), thereby obtaining image P5 as a vertically compressed image. At this time, images P5 and P3 have approximately the same pixel resolution in the vertical direction (approximately the same). Next, the image shifting unit 210 shifts image P5 by a predetermined number of pixels in the direction in which cameras 1 and 2 are aligned (the horizontal direction). The one-dimensional compression unit 320 then compresses the shifted image P5 in the horizontal direction, thereby obtaining image P6 as a horizontally and vertically compressed image. At this time, images P6 and P3 have approximately the same pixel resolution in the horizontal and vertical directions (approximately the same). The similarity evaluation unit 220 then evaluates the similarity between the shifted images P6 and P3.

[0037] After the SAD evaluation, the image shifting unit 210 shifts the pixels (e.g., by one pixel relative to the previous processing). The one-dimensional compression unit 320 then compresses the shifted image P5 in the horizontal direction, obtaining image P6 as a horizontally and vertically compressed image. The similarity evaluation unit 220 then evaluates the similarity between the shifted image P6 and image P3. This process (specifically, the image shifting unit 210 shifts image P5 horizontally by a predetermined number of pixels, the one-dimensional compression unit 320 compresses the shifted image P5 horizontally to generate image P6, and the similarity evaluation unit 220 compares image P3 and image P6 to determine the similarity) is repeated a predetermined number of times until the similarity evaluation unit 220 has determined the similarity between the shifted image P6 and image P3 a predetermined number of times, and the disparity detection unit 230 determines the disparity Dh that minimizes the SAD.

[0038] FIG. 5 shows an example of a method for detecting parallax Dh, i.e., the difference between this embodiment and the conventional method. Here, image P4 has three times the resolution of image P3. Triple resolution means that an object detected with one pixel in image P3 is detected with three pixels in image P4. The points marked with circles in FIG. 5 represent the results of evaluating SAD at one-pixel intervals, as in FIG. 3 . In this embodiment, sub-pixel SAD evaluation is possible by shifting high-resolution image P5 by one pixel in the horizontal direction, compressing it, and evaluating the SAD. If image P4 has three times the resolution of image P3, SAD can be evaluated in 1 / 3 pixel increments, calculated using low-resolution pixel conversion, as shown by the points marked with squares in FIG. 5 . This allows for highly accurate parallax.

[0039] This embodiment is characterized in that compression in a direction (vertical direction) different from the disparity detection direction (horizontal direction) is performed by the one-dimensional compression unit 310, and then repeated processing is performed by the image shift unit 210, the one-dimensional compression unit 320, and the similarity evaluation unit 220. As a result, the repeated processing is limited to compression processing in the one-dimensional direction (horizontal direction), thereby reducing processing and memory requirements.

[0040] In this embodiment, the image P4 has been described as having a resolution three times that of the image P3, but this is not limiting and the resolution may be a non-integer value such as 2.5 times, etc. In this case, interpolation processing may be performed using information from adjacent pixels.

[0041] Furthermore, in this embodiment, the shift amount of the image shifter 210 has been described as one pixel. However, this is not limited to this, and other pixel counts, such as two or three, may also be used. This is because, as the resolution difference between the two images increases, the number of repeated processes increases. Furthermore, as the shift amount in the image shifter 210 decreases, the change in SAD decreases. Therefore, optimizing the shift amount can reduce the processing load. For example, if image P4 has a resolution ten times higher than image P3 rather than three times higher, setting the shift amount to one pixel would require ten times as many image compressions in the one-dimensional direction compared to the processing shown in Figures 2 and 3 . This may improve accuracy but impair real-time performance. Therefore, setting the shift amount to five pixels, for example, can reduce the number of image compressions, which is advantageous from the perspective of speed. On the other hand, there is a concern that the accuracy of disparity detection may decrease compared to when the shift amount is one pixel. Therefore, the shift amount described above can be optimized to suit the system.

[0042] Here, in this embodiment, disparity estimation using linear approximation has been described in Figures 3 and 5, but this is not limited to this, and a nonlinear approximation curve may be used, or linear approximation using multiple points may also provide the same effect.

[0043] In this embodiment, it is assumed that the processing is performed within the image processing device 50, but the processing is not limited to this, and the processing may be performed in a different computing environment, such as an edge computer or the cloud.

[0044] The starting pixel on the horizontal axis of the graphs shown in Figures 3 and 5 may be either the small side (far side) or the large side (near side). The size of the starting point depends on the design value of the system. For example, by setting the starting point near the parallax amount of the maximum or minimum distance measured by the system, the amount of compression processing can be reduced. The size of the ending point can be set near the pixel where the SAD is minimum (for example, a pixel several points after the SAD starts to increase after decreasing), for example, to be the ending point, thereby reducing the amount of compression processing.

[0045] 6 shows the parallax detection process of the stereo parallax image generation unit 200 of the image processing device 50 according to Example 2. The difference from Example 1 is that two different stereo matching processes are performed by a stereo matching processing unit S1 and a stereo matching processing unit S2 in order to reduce the processing load.

[0046] First, the stereo matching processing unit S1 will be described. The stereo matching processing unit S1 shown in FIG.

[0047] In the stereo matching processing unit S1, the compression unit 330 compresses image P4 input from the image processing unit 100 in two dimensions, horizontally and vertically, to obtain image P7 as a horizontally and vertically compressed image. Image P7 has substantially the same resolution as image P3 in both the horizontal and vertical directions. Then, the image shifting unit 210 shifts image P7 by a predetermined number of pixels in the direction in which camera 1 and camera 2 are aligned (horizontally). Thereafter, the similarity evaluation unit 220 evaluates the similarity between image P7 after the shift and image P3.

[0048] After the SAD evaluation, the image shifting unit 210 shifts the pixels (e.g., by one pixel relative to the previous processing), and the similarity evaluation unit 220 performs SAD evaluation. This process (specifically, the image shifting unit 210 shifts image P7 horizontally by a predetermined number of pixels, and the similarity evaluation unit 220 compares the shifted image P7 with image P3 to determine the similarity) is repeated a predetermined number of times until the similarity evaluation unit 220 has determined the similarity between the shifted image P7 and image P3 a predetermined number of times. Then, the pixel shift range determination unit 240 detects two points with small SAD and determines the pixel shift range (also referred to as the search range or search area) of the stereo matching processing unit S2 (of the image shifting unit 210). The pixel (parallax) with the smallest SAD is included between these two detected points. Therefore, the pixel shift range determination unit 240 determines and outputs, for example, the area between the two detected points as the pixel shift range of the stereo matching processing unit S2 (of the image shift unit 210).

[0049] Next, the stereo matching processing unit S2 will be described. Similar to FIG. 4 of the first embodiment, the stereo matching processing unit S2 shown in FIG. 6 is configured to include a one-dimensional compression unit 310, an image shifting unit 210, a one-dimensional compression unit 320, a similarity evaluation unit 220, and a disparity detection unit 230. The difference from the first embodiment is that the pixel shift range determined by the pixel shift range determination unit 240 of the stereo matching processing unit S1 described above is input to the image shifting unit 210.

[0050] In the stereo matching processing unit S2, the one-dimensional compression unit 310 vertically compresses image P4 input from the image processing unit 100 to obtain image P5 as a vertically compressed image. At this time, images P5 and P3 have approximately the same vertical pixel resolution (approximately the same). Next, the image shifting unit 210 shifts image P5 horizontally by a predetermined number of pixels based on the information obtained by the pixel shift range determination unit 240 (i.e., according to the pixel shift range determined by the pixel shift range determination unit 240). Then, the one-dimensional compression unit 320 compresses the shifted image P5 horizontally to obtain image P6 as a horizontally and vertically compressed image. At this time, images P6 and P3 have approximately the same horizontal and vertical pixel resolution (approximately the same). Then, the similarity evaluation unit 220 evaluates the similarity between the shifted image P6 and image P3.

[0051] After the evaluation by SAD, the image shifting unit 210 generates image P5 with a different shift amount (e.g., shifting one pixel from the previous processing) based on the information obtained by the pixel shift range determining unit 240 (i.e., according to the pixel shift range determined by the pixel shift range determining unit 240).Then, the one-dimensional compression unit 320 compresses the shifted image P5 in the horizontal direction to obtain image P6 as a horizontally and vertically compressed image.Then, the similarity evaluation unit 220 evaluates the similarity between the shifted image P6 and image P3. This process (more specifically, the process of having the image shifter 210 shift image P5 horizontally by a predetermined number of pixels in accordance with the pixel shift range, having the one-dimensional compressor 320 compress the shifted image P5 horizontally to generate image P6, and having the similarity evaluation unit 220 compare image P3 and image P6 to determine the similarity) is repeated a predetermined number of times until the similarity evaluation unit 220 has determined the similarity between shifted image P6 and image P3 a predetermined number of times, and the disparity detection unit 230 determines the disparity Dh that minimizes the SAD.

[0052] 7 shows an example of the disparity detection method of this embodiment. The SAD evaluated by the stereo matching processing unit S1 is indicated by a circle, and the SAD evaluated by the stereo matching processing unit S2 is indicated by a square. Here, it is assumed that image P4 has three times the resolution of image P3.

[0053] The stereo matching processing unit S1 obtains the SAD for every other pixel. Two points with the smallest SAD are detected, and the stereo matching processing unit S2 performs the process between them. In other words, the area between the two detected points is used as the pixel shift range of the stereo matching processing unit S2 (the image shift unit 210), and the stereo matching processing is performed. By performing such stereo matching processing, the number of iterations of the stereo matching processing unit S2, indicated by squares, can be reduced compared to the processing method of Example 1, and the disparity at which the SAD is minimized can be efficiently determined.

[0054] In this embodiment, the stereo matching processing unit S1 is performed first, followed by the stereo matching processing unit S2, but it is not necessary to perform the same processing repeatedly. If the distance to the target object does not deviate significantly over time, this processing may be performed once, and thereafter, only the stereo matching processing unit S2 may be performed. For example, in the field of in-vehicle sensing, this processing can be used in adaptive cruise control (ACC). ACC is a function that automatically operates the accelerator and brake while driving to maintain a constant distance from a vehicle ahead. With this function, the distance from the vehicle ahead does not change significantly over a short period of time. Therefore, by performing this processing once, an approximate distance to the vehicle ahead can be obtained, and thereafter, only the stereo matching processing unit S2 can be performed to measure the accurate distance to the vehicle ahead.

[0055] The stereo matching processing unit S2 of this embodiment uses the processing of Example 1, but may also use other processing such as that shown in FIG. 8 . In this case, the stereo matching processing unit S2 shifts the high-resolution image P4 horizontally by a predetermined number of pixels based on information obtained by the pixel shift range determination unit 240 (i.e., according to the pixel shift range determined by the pixel shift range determination unit 240) using the image shift unit 210. The compression unit 300 then compresses the shifted image horizontally and vertically to obtain image P8 as a horizontally and vertically compressed image. The similarity evaluation unit 220 then evaluates the SAD, which is the similarity between image P8 and image P3. At this time, image P8 and image P3 have approximately the same pixel resolution in the horizontal and vertical directions. Furthermore, this processing is repeated until the similarity evaluation unit 220 has determined the similarity between image P8 and image P3 a predetermined number of times, and the disparity detection unit 230 determines the disparity that minimizes the SAD.

[0056] In this embodiment, the stereo matching processing unit is divided into two, the stereo matching processing unit S1 and the stereo matching processing unit S2, but this does not necessarily have to be divided. For example, the pixel shift amount may be increased in the stereo parallax image generation unit 200 shown in Fig. 4 of the first embodiment, and the pixel shift amount may be reduced based on the result. In other words, for example, the pixel shift amount may be variable in the stereo parallax image generation unit 200 shown in Fig. 4 of the first embodiment, and the stereo matching processing may be performed.

[0057] The starting pixel on the horizontal axis of the graph shown in FIG. 7 may be either the small side (far side) or the large side (near side). The size of the starting point depends on the design value of the system. For example, by setting the starting point near the parallax amount of the maximum or minimum distance measured by the system, the amount of compression processing can be reduced. The size of the ending point can be set near the pixel where the SAD is minimum (for example, a pixel several points after the SAD starts to increase after decreasing), for example, to be the ending point, thereby reducing the amount of compression processing.

[0058] [Summary] As described above, the image processing device 50 of this embodiment includes an image acquisition unit 20 that acquires a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image; a first compression unit (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a first compressed image (image P5); a second compression unit (one-dimensional compression unit 320) that compresses the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6); a first similarity evaluation unit (similarity evaluation unit 220) that compares the first image (image P3) and the second compressed image (image P6) to determine a first similarity; and a disparity detection unit 230 that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the first similarity determined by the first similarity evaluation unit (similarity evaluation unit 220) (Examples 1 and 2).

[0059] Furthermore, the image processing device 50 of this embodiment repeatedly executes the following process until the first similarity evaluation unit (similarity evaluation unit 220) has determined the first similarity a predetermined number of times: the first image shift unit (image shift unit 210) shifts the first compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels; the second compression unit (one-dimensional compression unit 320) compresses the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6); and the first similarity evaluation unit (similarity evaluation unit 220) compares the first image (image P3) with the second compressed image (image P6) to determine the first similarity (Examples 1 and 2).

[0060] The image processing device 50 of this embodiment also includes a third compression unit (compression unit 330) that compresses the second image (image P4) in the first direction (horizontal direction) and the second direction (vertical direction) to generate a third compressed image (image P7), a second image shifting unit (image shifting unit 210) that shifts the third compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, a second similarity evaluation unit (similarity evaluation unit 220) that compares the shifted third compressed image (image P7) with the first image (image P3) to obtain a second similarity, and a third compression unit (compression unit 330) that compresses the second image (image P4) in the first direction (horizontal direction) and the second direction (vertical direction) to generate a third compressed image (image P7). and a pixel shift range determination unit 240 that determines a pixel shift range (in the first direction (horizontal direction)) in the first image shift unit (image shift unit 210) based on the second similarity determined by the pixel shift range determination unit 220, and the first image shift unit (image shift unit 210) shifts the first compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels in accordance with the pixel shift range determined by the pixel shift range determination unit 240 (within the pixel shift range determined by the pixel shift range determination unit 240) (Example 2).

[0061] The image processing device 50 of this embodiment includes an image acquisition unit 20 that acquires a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image, and first and second stereo matching processing units (S1 and S2). The first stereo matching processing unit (stereo matching processing unit S1) includes a first compression unit (compression unit 330) that compresses the second image (image P4) in a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned and in a second direction (vertical direction) that is substantially perpendicular to the first direction (horizontal direction) to generate a first compressed image (image P7), a first image shift unit (image shift unit 210) that shifts the first compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, and a second image shift unit (image shift unit 210) that shifts the first compressed image (image P7) by a predetermined number of pixels in the first direction (horizontal direction). The stereo matching processing apparatus includes a first similarity evaluation unit (similarity evaluation unit 220) that compares the shifted first compressed image (image P7) with the first image (image P3) to determine a first similarity, and a pixel shift range determination unit 240 that determines a pixel shift range (in the first direction (horizontal direction)) in the second stereo matching processing unit (stereo matching processing unit S2) based on the first similarity determined by the first similarity evaluation unit (similarity evaluation unit 220), and the second stereo matching processing unit (stereo matching processing unit S2) shifts the second image (image P4) (by a predetermined number of pixels in the first direction (horizontal direction)) in accordance with the pixel shift range determined by the pixel shift range determination unit 240 (within the pixel shift range determined by the pixel shift range determination unit 240) to determine a disparity (between images P3 and P4 or between images P1 and P2) (Example 2).

[0062] In the image processing device 50 of this embodiment, the second stereo matching processing unit (stereo matching processing unit S2) includes a second compression unit (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a second compressed image (image P5), and a second image shift unit (image a first image shifting unit 210), a third compression unit (one-dimensional compression unit 320) that compresses the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6), a second similarity evaluation unit (similarity evaluation unit 220) that compares the first image (image P3) and the third compressed image (image P6) to determine a second similarity, and a disparity detection unit 230 that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the second similarity determined by the second similarity evaluation unit (similarity evaluation unit 220) (Example 2).

[0063] Furthermore, the image processing device 50 of this embodiment repeatedly executes the following process until the second similarity evaluation unit (similarity evaluation unit 220) has determined the second similarity a predetermined number of times: causing the second image shift unit (image shift unit 210) to shift the second compressed image (image P5) in the first direction (horizontal direction) in accordance with the pixel shift range (within the pixel shift range); causing the third compression unit (one-dimensional compression unit 320) to compress the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6); and causing the second similarity evaluation unit (similarity evaluation unit 220) to compare the first image (image P3) and the third compressed image (image P6) to determine the second similarity (Example 2).

[0064] The image processing method of this embodiment includes an image acquisition process (image acquisition unit 20) for acquiring a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image; a first compression process (one-dimensional compression unit 310) for compressing the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned to generate a first compressed image (image P5); and a second compression process (one-dimensional compression unit 310) for shifting the first compressed image (image P5) by a predetermined number of pixels in the first direction (horizontal direction). This is an image processing method in which a computer executes the following steps (Examples 1 and 2): a first image shifting process (image shifting unit 210); a second compression process (one-dimensional compression unit 320) that compresses the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6); a first similarity evaluation process (similarity evaluation unit 220) that compares the first image (image P3) and the second compressed image (image P6) to determine a first similarity; and a disparity detection process (disparity detection unit 230) that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the first similarity determined by the first similarity evaluation process (similarity evaluation unit 220).

[0065] The image processing method of this embodiment is an image processing method in which a computer executes an image acquisition process (image acquisition unit 20) for acquiring a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image, and first and second stereo matching processes (stereo matching processing units S1 and S2). The first stereo matching process (stereo matching processing unit S1) includes a first compression process (compression unit 330) for compressing the second image (image P4) in a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned and in a second direction (vertical direction) substantially perpendicular to the first direction (horizontal direction) to generate a first compressed image (image P7), a first image shift process (image shift unit 210) for shifting the first compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, and a second image shift process (image shift unit 210) for shifting the shifted image. and a pixel shift range determination process (pixel shift range determination unit 240) that determines a pixel shift range (in the first direction (horizontal direction)) in the second stereo matching process (stereo matching processing unit S2) based on the first similarity determined by the first similarity evaluation process (similarity evaluation unit 220). In the second stereo matching process (stereo matching processing unit S2), the second image (image P4) is shifted (by a predetermined number of pixels in the first direction (horizontal direction)) in accordance with the pixel shift range determined by the pixel shift range determination process (pixel shift range determination unit 240) (within the pixel shift range determined by the pixel shift range determination process (pixel shift range determination unit 240)), thereby determining a disparity (between images P3 and P4 or between images P1 and P2) (Example 2).

[0066] Furthermore, in the image processing method of this embodiment, the second stereo matching process (stereo matching processing unit S2) includes a second compression process (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a second compressed image (image P5); and a second image shift process (image shift unit 210) that shifts the second compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels in accordance with the pixel shift range (within the range of the pixel shift range). This image processing method (Example 2) includes a third compression process (one-dimensional compression unit 320) that compresses the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6), a second similarity evaluation process (similarity evaluation unit 220) that compares the first image (image P3) and the third compressed image (image P6) to determine a second similarity, and a disparity detection process (disparity detection unit 230) that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the second similarity determined by the second similarity evaluation process (similarity evaluation unit 220).

[0067] According to the image processing device and image processing method of this embodiment, in stereo vision using two cameras with different resolutions (different-resolution stereo cameras), by repeatedly shifting and compressing a high-resolution image and deriving the similarity with a low-resolution image, it is possible to provide an image processing device and image processing method that can perform highly accurate disparity detection using a high-resolution image and a low-resolution image.

[0068] <Modifications> The present invention is not limited to the above-described embodiments, and includes various other modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0069] Furthermore, the above-described configurations, functions, and units 100, 20, 20a-23a, 20b-23b, 200, 600, 700, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. As shown in FIG. 1B , the above-described configurations, functions, etc. may be implemented in software by a computing device 122, such as a CPU, interpreting and executing programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions can be stored on a hard disk or on a recording device, such as memory 121, an SSD, or a recording medium, such as an IC (Integrated Circuit) card, an SD (Secure Digital) card, or a DVD (Digital Versatile Disc).

[0070] In addition, in each embodiment, the control lines and information lines shown are those that are considered necessary for explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected.

[0071] DESCRIPTION OF SYMBOLS 1 Camera 2 Camera 50 Image processing device 100 Image processing unit 200 Stereo parallax image generation unit 210 Image shift unit 220 Similarity evaluation unit 230 Parallax detection unit 240 Pixel shift range determination unit 300 Compression unit 310 One-dimensional compression unit 320 One-dimensional compression unit 330 Compression unit 600 Three-dimensional object detection unit 700 Alarm control unit

Claims

1. An image processing device comprising: an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression unit that compresses the second image in a second direction that is approximately perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift unit that shifts the first compressed image in the first direction by a predetermined number of pixels; a second compression unit that compresses the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation unit that compares the first image and the second compressed image to determine a first similarity; and a disparity detection unit that determines a disparity based on the first similarity determined by the first similarity evaluation unit.

2. An image processing device according to claim 1, which repeatedly executes the following process until the first similarity evaluation unit has determined the first similarity a predetermined number of times: causing the first image shift unit to shift the first compressed image in the first direction by a predetermined number of pixels; causing the second compression unit to compress the shifted first compressed image in the first direction to generate a second compressed image; and causing the first similarity evaluation unit to compare the first image with the second compressed image to determine the first similarity.

3. An image processing device according to claim 1, wherein the first compression unit compresses the second image in the second direction so that the pixel resolutions of the first image and the first compressed image in the second direction are approximately the same.

4. An image processing device according to claim 1, wherein the second compression unit compresses the shifted first compressed image in the first direction so that the resolutions of the pixels in the first direction of the second compressed image and the first image are approximately the same.

5. An image processing device according to claim 1, further comprising: a third compression unit that compresses the second image in the first direction and the second direction to generate a third compressed image; a second image shifting unit that shifts the third compressed image in the first direction by a predetermined number of pixels; a second similarity evaluation unit that compares the shifted third compressed image with the first image to determine a second similarity; and a pixel shift range determination unit that determines a pixel shift range in the first image shifting unit based on the second similarity determined by the second similarity evaluation unit, wherein the first image shifting unit shifts the first compressed image in the first direction by the predetermined number of pixels in accordance with the pixel shift range determined by the pixel shift range determination unit.

6. An image processing apparatus comprising: an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; and first and second stereo matching processing units, wherein the first stereo matching processing unit has: a first compression unit that compresses the second image in a first direction in which the first camera and the second camera are aligned and in a second direction substantially perpendicular to the first direction to generate a first compressed image; a first image shift unit that shifts the first compressed image by a predetermined number of pixels in the first direction; a first similarity evaluation unit that compares the shifted first compressed image with the first image to obtain a first similarity; and a pixel shift range determination unit that determines a pixel shift range in the second stereo matching processing unit based on the first similarity obtained by the first similarity evaluation unit, and the second stereo matching processing unit obtains parallax by shifting the second image in accordance with the pixel shift range determined by the pixel shift range determination unit.

7. An image processing device according to claim 6, wherein the second stereo matching processing unit comprises: a second compression unit that compresses the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned, to generate a second compressed image; a second image shift unit that shifts the second compressed image in the first direction by a predetermined number of pixels according to the pixel shift range; a third compression unit that compresses the shifted second compressed image in the first direction to generate a third compressed image; a second similarity evaluation unit that compares the first image with the third compressed image to determine a second similarity; and a disparity detection unit that determines a disparity based on the second similarity determined by the second similarity evaluation unit.

8. An image processing device according to claim 7, which repeatedly executes the following processes until the second similarity evaluation unit has determined the second similarity a predetermined number of times: causing the second image shift unit to shift the second compressed image in the first direction by a predetermined number of pixels in accordance with the pixel shift range; causing the third compression unit to compress the shifted second compressed image in the first direction to generate a third compressed image; and causing the second similarity evaluation unit to compare the first image with the third compressed image to determine the second similarity.

9. An image processing method in which a computer executes the following steps: an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression process for compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift process for shifting the first compressed image in the first direction by a predetermined number of pixels; a second compression process for compressing the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation process for comparing the first image and the second compressed image to determine a first similarity; and a disparity detection process for determining a disparity based on the first similarity determined by the first similarity evaluation process.

10. An image processing method in which a computer executes an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image, and first and second stereo matching processes, wherein the first stereo matching process includes: a first compression process for compressing the second image in a first direction in which the first camera and the second camera are aligned and in a second direction substantially perpendicular to the first direction to generate a first compressed image; a first image shift process for shifting the first compressed image by a predetermined number of pixels in the first direction; a first similarity evaluation process for comparing the shifted first compressed image with the first image to obtain a first similarity; and a pixel shift range determination process for determining a pixel shift range in the second stereo matching process based on the first similarity obtained by the first similarity evaluation process, wherein the second stereo matching process obtains parallax by shifting the second image in accordance with the pixel shift range determined by the pixel shift range determination process.

11. An image processing method according to claim 10, wherein the second stereo matching process comprises: a second compression process that compresses the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned, to generate a second compressed image; a second image shift process that shifts the second compressed image in the first direction by a predetermined number of pixels according to the pixel shift range; a third compression process that compresses the shifted second compressed image in the first direction, to generate a third compressed image; a second similarity evaluation process that compares the first image with the third compressed image to determine a second similarity; and a disparity detection process that determines a disparity based on the second similarity determined by the second similarity evaluation process.

Images