Image processing apparatus and image processing method

Abstract

To provide an image processor and an image processing method capable of highly accurately detecting parallax by a high resolution image and a low resolution image in stereoscopic vision using two cameras different in resolution.SOLUTION: The image processing apparatus includes an image acquisition unit configured to acquire a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image, a first compression unit configured to compress the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are arranged to generate a first compressed image, a first image shift unit configured to shift the first compressed image by a predetermined number of pixels in the first direction, a second compression unit configured to compress the shifted first compressed image in the first direction to generate a second compressed image, and a parallax detection unit configured to compare the first image and the second compressed image to obtain a parallax.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] A stereo image processing device (hereafter referred to as a stereo camera) is known as a device for recognizing objects three-dimensionally. A stereo camera uses the differences in how images are captured by multiple cameras placed at different positions to detect the parallax between multiple images based on trigonometry, and then uses this parallax to detect the depth and position of an object, thereby enabling accurate detection of the position of the target object.

[0003] Stereo cameras are used in a wide range of fields, including automotive and robotics. While conventional stereo cameras generally use the same camera, stereo vision is also performed using cameras with different lenses and sensors. Different cameras have different image resolutions relative to pixels due to differences in the image size on the sensor caused by differences in the focal length of the lens and differences in the pixel size of the sensor. In stereo vision using cameras with different resolutions, the challenge is to find a method for matching high-resolution images, which have a relatively high pixel resolution, with low-resolution images, which have a relatively low pixel resolution.

[0004] Patent Documents 1 and 2 propose methods for matching high-resolution images with low-resolution images. Patent Document 1 describes the problem as "an image capturing device having a plurality of cameras capturing images with different spatial resolutions and forming a distance image by measuring the distance to a subject, to improve the spatial resolution of the distance image," and the solution as "an image capturing device according to one aspect of the disclosed technology is an image capturing device in which cameras capturing first images with a first spatial resolution in a predetermined field of view and cameras capturing second images with a second spatial resolution lower than the first spatial resolution are arranged at predetermined intervals in a predetermined array direction, and the image capturing device forms a distance image from the distance to the subject measured based on the first images and the second images, wherein the distance image has the first spatial resolution in a direction intersecting the array direction." Furthermore, Patent Document 2 states that the problem is to provide a three-dimensional structure estimation device that can estimate the three-dimensional structure of an object within the field of view in greater detail while providing a wide-field image and a high-resolution image for part of the field of view. The solution is described as follows: "The camera 3, 4 has a wide-field-of-view cone 1 and a narrow-field-of-view cone 2 of different fields of view, and generates images with different resolutions. The camera 3, 4 outputs wide-field-of-view image 5 and narrow-field-of-view image 6, and these images are output from the camera 3, 4. The wide-field-of-view cone 1 and narrow-field-of-view cone 2 have different fields of view, and generate images with different resolutions. The conversion unit converts the images to match the wide-field-of-view image 5 and narrow-field-of-view image 6 for stereo calculations. The conversion unit includes a thinned-out sample unit 7 that thins out the pixels of the narrow-field-of-view image 6 to generate and output a pixel-by-pixel matched image 8 in which the pixels are matched, and a depth image creation unit 9 that inputs and directly compares the matched wide-field-of-view image 5 and pixel-by-pixel matched image 8, calculates the depth of the image, and creates and outputs a depth image 11. Furthermore, the thinned-out sample unit generates multiple images with matched pixels for each small movement, enabling more precise depth estimation." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-158776 [Patent Document 2] Japanese Patent Application Publication No. 10-134187 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, disparity between high-resolution and low-resolution images is detected by converting the low-resolution image into a high-resolution image. On the other hand, because increasing the resolution of a low-resolution image with little information does not increase the amount of information, the accuracy of disparity estimation follows the low-resolution image. Patent Document 2 proposes shifting a high-resolution image and then thinning or compressively interpolating it to match it with the low-resolution image, and then evaluating the similarity. However, because the thinned high-resolution image differs from the low-resolution image detected by the camera, highly accurate disparity detection is difficult. Furthermore, compressive interpolation can obtain a disparity image of a low-resolution image, but can also detect disparity using a high-resolution image, thereby achieving highly accurate disparity detection. However, compressive interpolation requires two-dimensional compression and interpolation for each process, which increases the processing scale.

[0007] The present invention has been made in consideration of the above-mentioned problems, and provides an image processing device and an image processing method that are capable of highly accurate disparity detection using high-resolution and low-resolution images in stereoscopic vision using two cameras with different resolutions. [Means for solving the problem]

[0008] The image processing device of the present invention includes an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression unit that compresses the second image in a second direction that is approximately perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift unit that shifts the first compressed image by a predetermined number of pixels in the first direction; a second compression unit that compresses the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation unit that compares the first image with the second compressed image to determine a first similarity; and a disparity detection unit that determines a disparity based on the first similarity determined by the first similarity evaluation unit.

[0009] The image processing method of the present invention is an image processing method in which a computer executes the following steps: an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression process for compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shift process for shifting the first compressed image in the first direction by a predetermined number of pixels; a second compression process for compressing the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation process for comparing the first image and the second compressed image to determine a first similarity; and a disparity detection process for determining a disparity based on the first similarity determined by the first similarity evaluation process. [Effects of the Invention]

[0010] According to the image processing device and image processing method of the present invention, it is possible to provide an image processing device and image processing method that are capable of highly accurate parallax detection using a high-resolution image and a low-resolution image. [Brief explanation of the drawings]

[0011] [Figure 1A] 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. [Figure 1B] 1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 10 is a block diagram illustrating a parallax detection process of a stereo parallax image generating unit of a conventional image processing device serving as a reference for the first embodiment. [Figure 3] 1A and 1B are diagrams illustrating a parallax detection method of a conventional image processing device that serves as a reference for the first embodiment. [Figure 4] 3 is a block diagram illustrating a parallax detection process of a stereo parallax image generating unit of the image processing device according to the first embodiment. FIG. [Figure 5] 1A to 1C are diagrams illustrating a parallax detection method of the image processing device according to the first embodiment. [Figure 6] FIG. 10 is a block diagram illustrating a parallax detection process of a stereo parallax image generating unit of an image processing device according to a second embodiment. [Figure 7] 10A and 10B are diagrams illustrating a parallax detection method of the image processing device according to the second embodiment. [Figure 8] FIG. 10 is a block diagram illustrating another parallax detection process of a stereo parallax image generating unit of the image processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the present embodiment will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings illustrate embodiments in accordance with the principles of the present disclosure, but these drawings are for the purpose of understanding the present disclosure and are not to be used to interpret the present disclosure in a limiting manner. The descriptions in this specification are merely typical examples and are not intended to limit the scope or application of the present disclosure in any way.

[0013] Although the present embodiment has been described in sufficient detail to enable those skilled in the art to practice the present disclosure, it should be understood that other embodiments are possible, and that changes in configuration and structure and substitutions of various elements are possible without departing from the scope and spirit of the technical ideas of the present disclosure. Therefore, the following description should not be interpreted as being limited thereto.

[0014] [Example 1] 1A, a configuration of a stereo image processing device 50 (hereinafter referred to as image processing device 50) according to a first embodiment will be described. The image processing device 50 is mounted on a vehicle such as an automobile, and is used to detect the distance from the vehicle to three-dimensional objects (other automobiles, buildings, pedestrians, etc.) around the vehicle. In the following, a case where the image processing device 50 is mounted on a vehicle will be described as an example, but the present invention is not limited to this.

[0015] 1A is a block diagram showing an example of the configuration of an image processing device 50 according to a first embodiment. The image processing device 50 is configured to detect surrounding three-dimensional objects based on images acquired by a camera 1 and a camera 2, and to issue an alarm as necessary. The camera 1 and the camera 2 form a stereo camera.

[0016] The image processing device 50 of this embodiment is configured to include an image processing unit 100, a stereo parallax image generating unit 200, a three-dimensional object detecting unit 600, and an alarm control unit 700, as an example.

[0017] In this image processing device 50, in an area that can be captured in common by camera 1 and camera 2 (hereinafter referred to as the "stereo viewing area"), a stereo parallax image generating unit 200 generates a parallax image using the parallax between camera 1 and camera 2. Then, a three-dimensional object detecting unit 600 measures the distance from the vehicle to the three-dimensional object according to the parallax.

[0018] The image processing unit 100 is configured to include, for example, an image acquisition unit 20, affine processing units 20a and 20b, luminance correction units 21a and 21b, pixel interpolation units 22a and 22b, and luminance information generation units 23a and 23b. The image processing unit 100 applies predetermined image processing to generate parallax images to the images P1 and P2 acquired by the cameras 1 and 2, and supplies the images to the stereo parallax image generation unit 200.

[0019] Although not shown, camera 1 and camera 2 each include a lens and an image sensor. Camera 1 and camera 2 each capture (take) an image of an object via the lens using the image sensor. Camera 1 and camera 2 are arranged side by side in a predetermined direction so that they overlap (share) a portion of their field of view (image capture area). Image capture unit 20 captures image P1 from camera 1 and image P2 from camera 2.

[0020] The affine processing unit 20a performs affine processing on image P1 from camera 1. As a result of this affine processing, the affine processing unit 20a obtains image P3. Similarly, the affine processing unit 20b performs affine processing on image P2 from camera 2 to obtain image P4. Note that the affine processing units 20a and 20b may also perform distortion transformation processing other than affine processing. In this embodiment, f sin θ, the projection method of the fisheye lens, is projected into a coordinate system of (f tan θx, f tan θy). Here, f is the focal length of the lens, θ is the angle of view incident on the lens, and θx and θx are the horizontal and vertical components of the angle of view incident on the lens.

[0021] The luminance correction unit 21a corrects the luminance of each pixel in the image P3. For example, the luminance of each pixel in the image P3 is corrected based on the gain of the camera 1, the difference in gain between each pixel in the image P3, etc. Similarly, the luminance correction unit 21b corrects the luminance of each pixel in the image P4.

[0022] The pixel interpolation unit 22a performs demosaicing on the image P3 (e.g., converting the RAW image into a color image). Similarly, the pixel interpolation unit 22b performs demosaicing on the image P4.

[0023] The luminance information generator 23a generates luminance information for the image P3. For example, the luminance information generator 23a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generator 23b generates luminance information for the image P4.

[0024] The stereo parallax image generating unit 200 generates parallax images of the above-mentioned stereo viewing area (common viewing area) using the obtained images P3 and P4 (details will be explained later).

[0025] The three-dimensional object detection unit 600 detects three-dimensional objects in the stereo vision area according to the parallax images obtained by the stereo parallax image generation unit 200. It also applies stereo matching to the detected three-dimensional objects to detect parallax and identify the type of three-dimensional object (pedestrian, bicycle, vehicle, building, etc.). By identifying pedestrians, bicycles, vehicles, etc. among the three-dimensional objects, it further identifies the type to be used for preventive safety. Vehicle detection results are used for tracking control of the preceding vehicle and emergency braking control. Pedestrians and bicycles are basically used for emergency braking, especially for warnings and vehicle control for pedestrians and bicycles that suddenly appear. Compared to stationary objects, warnings and control are implemented for objects that suddenly appear within a wider angle of view. By measuring the distance to these detected objects and estimating the moving speed of the objects being tracked over time, the warning control unit 700 implements more appropriate warnings and control.

[0026] FIG. 1B is a block diagram illustrating an example of the hardware configuration of the image processing device 50 according to the first embodiment.

[0027] The image processing device 50 of this embodiment includes a memory 121 and an arithmetic device 122 configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The image processing device 50 also includes a storage device 123 such as an HD (Hard Disk) or an SSD (Solid State Drive). The image processing device 50 also includes an input device 124, an output device 125, and a communication device 126 that communicates with cameras 1 and 2.

[0028] A program is stored in storage device 123, and this program is loaded into memory 121. The loaded program is then executed by arithmetic device 122. This embodies the units 100, 20, 20a to 23a, 20b to 23b, 200, 600, and 700 shown in FIG. 1A.

[0029] 2 shows the parallax detection process of a conventional stereo parallax image generator 200. Here, the description will be given assuming that image P3 obtained from image P1 and image P4 obtained from image P2 have the same resolution.

[0030] The stereo parallax image generating unit 200 shown in FIG. 2 includes an image shifting unit 210, a similarity evaluating unit 220, and a parallax detecting unit 230.

[0031] In this stereo parallax image generation unit 200, an image shifter 210 shifts image P4 by a predetermined number of pixels in the direction in which the two cameras are aligned. Then, a similarity evaluation unit 220 evaluates the similarity between image P4 after the shift and image P3. Here, a case will be described in which similarity is evaluated using SAD (Sum of Absolute Difference) of template matching as an evaluation method, but other methods such as SSD (Sum of Squared Difference) may also be used.

[0032] After the evaluation by SAD, the image shifter 210 shifts the pixels (for example, by one pixel from the previous processing), and the similarity evaluator 220 performs evaluation by SAD. This is repeated a predetermined number of times, and the disparity detector 230 finds the disparity Dl that minimizes the SAD.

[0033] FIG. 3 shows an example of a method for detecting disparity Dl. In this description, the shift amount in image shift unit 210 during repeated evaluation is assumed to be one pixel. In addition, the description will be made assuming that the start pixel and end pixel in image shift unit 210 during repeated evaluation, i.e., the pixel shift range between the start pixel and the end pixel, are arbitrarily set. As shown in FIG. 3, pixels are shifted one pixel at a time, and the minimum value is found based on the evaluation result of SAD. Here, four points with the smallest SAD are detected, and the intersection of the lines connecting two points is taken as the disparity Dl, thereby making it possible to detect sub-pixel disparity.

[0034] Next, the parallax detection process of the stereo parallax image generating unit 200 of this embodiment will be described with reference to Fig. 4. Here, the description will be made assuming that image P3 obtained from image P1 and image P4 obtained from image P2 have different resolutions. Image P3 will be described assuming that it has a lower pixel resolution than image P4, and image P4 will be described assuming that it has a higher pixel resolution than image P3. For example, image P3 is composed of images captured at a wider angle and shorter distance (nearby) than image P4, and image P4 is composed of images captured at a narrower angle and longer distance (farther) than image P3.

[0035] The stereo parallax image generating unit 200 shown in FIG. 4 includes a one-dimensional compression unit 310, an image shifting unit 210, a one-dimensional compression unit 320, a similarity evaluation unit 220, and a parallax detection unit 230.

[0036] The one-dimensional compression unit 310 compresses image P4 input from the image processing unit 100 in a direction (hereinafter referred to as the vertical direction) approximately perpendicular to the direction in which camera 1 and camera 2 are aligned (hereinafter referred to as the horizontal direction), thereby obtaining image P5 as a vertically compressed image. At this time, images P5 and P3 have approximately the same pixel resolution in the vertical direction (approximately the same). Next, the image shift unit 210 shifts image P5 by a predetermined number of pixels in the direction in which camera 1 and camera 2 are aligned (the horizontal direction). Then, the one-dimensional compression unit 320 compresses the shifted image P5 in the horizontal direction, thereby obtaining image P6 as a horizontally and vertically compressed image. At this time, images P6 and P3 have approximately the same pixel resolution in the horizontal and vertical directions (approximately the same). Thereafter, the similarity evaluation unit 220 evaluates the similarity between the shifted image P6 and image P3.

[0037] After the SAD evaluation, the image shifter 210 shifts the pixels (for example, by one pixel relative to the previous processing). Then, the one-dimensional compressor 320 compresses the shifted image P5 in the horizontal direction to obtain image P6 as a horizontally and vertically compressed image. The similarity evaluator 220 then evaluates the similarity between the shifted image P6 and image P3. This process (more specifically, the image shifter 210 shifts image P5 horizontally by a predetermined number of pixels, the one-dimensional compressor 320 compresses the shifted image P5 horizontally to generate image P6, and the similarity evaluator 220 compares image P3 and image P6 to determine the similarity) is repeated a predetermined number of times until the similarity evaluator 220 has determined the similarity between the shifted image P6 and image P3 a predetermined number of times, and the disparity detector 230 determines the disparity Dh that minimizes the SAD.

[0038] FIG. 5 shows an example of a method for detecting disparity Dh, i.e., the difference between this embodiment and the conventional method. Here, image P4 has three times the resolution of image P3. Triple resolution means that an object detected with one pixel in image P3 is detected with three pixels in image P4. The dots marked with circles in FIG. 5 represent the results of SAD evaluation performed at one-pixel intervals, as in FIG. 3. In this embodiment, sub-pixel SAD evaluation is possible by shifting high-resolution image P5 by one pixel in the horizontal direction, compressing it, and evaluating the SAD. If image P4 has three times the resolution of image P3, SAD can be evaluated in 1 / 3 pixel increments, calculated as low-resolution pixels, as shown by the dots marked with squares in FIG. 5. This allows for highly accurate parallax.

[0039] This embodiment is characterized in that compression in a direction (vertical direction) different from the disparity detection direction (horizontal direction) is performed by the one-dimensional compression unit 310, and then repeated processing is performed by the image shift unit 210, the one-dimensional compression unit 320, and the similarity evaluation unit 220. As a result, the repeated processing is limited to compression processing in the one-dimensional direction (horizontal direction), thereby reducing processing and memory consumption.

[0040] In this embodiment, the image P4 has been described as having a resolution three times that of the image P3, but this is not limited to this, and it is also acceptable if the resolution is not an integer value, such as 2.5 times. In this case, interpolation processing can be performed using information from adjacent pixels.

[0041] Furthermore, in this embodiment, the shift amount of the image shifter 210 is described as one pixel. However, this is not limited to this, and other pixel counts, such as two or three, may also be used. This is because, as the resolution difference between the two images increases, the number of iterations increases. Furthermore, as the shift amount in the image shifter 210 decreases, the SAD change decreases. Therefore, optimizing the shift amount can reduce the processing load. For example, if image P4 has a resolution ten times higher than image P3 rather than three times higher, setting the shift amount to one pixel would require ten times as many image compressions in the one-dimensional direction compared to the processing shown in Figures 2 and 3 . This may improve accuracy but impair real-time performance. Therefore, setting the shift amount to five pixels, for example, can reduce the number of image compressions, which is advantageous from the perspective of speed. On the other hand, there is a concern that the accuracy of disparity detection may decrease compared to when the shift amount is one pixel. Therefore, the shift amount described above can be optimized to suit the system.

[0042] Here, in this embodiment, disparity estimation using linear approximation has been described in Figures 3 and 5, but this is not limited to this, and a nonlinear approximation curve may be used, or linear approximation using multiple points may also provide the same effect.

[0043] In this embodiment, it is assumed that the processing is performed within the image processing device 50, but the processing is not limited to this, and the processing may be performed in a different computing environment, such as an edge computer or the cloud.

[0044] The starting pixel on the horizontal axis of the graphs shown in Figures 3 and 5 can be either the small side (far side) or the large side (near side). The size of the starting point depends on the design values ​​of the system. For example, by setting the starting point near the parallax amount of the maximum or minimum distance measured by the system, the amount of compression processing can be reduced. The size of the ending point can be set near the pixel where the SAD is minimum (for example, a few pixels after the SAD starts to increase after decreasing), for example, to be the ending point, thereby reducing the amount of compression processing.

[0045] [Example 2] 6 shows the parallax detection process of the stereo parallax image generation unit 200 of the image processing device 50 according to the second embodiment. The difference from the first embodiment is that two different stereo matching processes are performed by a stereo matching processing unit S1 and a stereo matching processing unit S2 in order to reduce the processing load.

[0046] First, the stereo matching processing unit S1 will be described. The stereo matching processing unit S1 shown in Fig. 6 includes a compression unit 330, an image shifting unit 210, a similarity evaluation unit 220, and a pixel shift range determination unit 240.

[0047] In the stereo matching processing unit S1, the compression unit 330 compresses image P4 input from the image processing unit 100 in two dimensions, horizontally and vertically, to obtain image P7 as a horizontally and vertically compressed image. Image P7 has substantially the same resolution as image P3 in both the horizontal and vertical directions. Then, the image shifting unit 210 shifts image P7 by a predetermined number of pixels in the direction in which camera 1 and camera 2 are lined up (horizontally). Thereafter, the similarity evaluation unit 220 evaluates the similarity between image P7 after the shift and image P3.

[0048] After the SAD evaluation, the image shifting unit 210 shifts the pixels (e.g., by one pixel relative to the previous processing), and the similarity evaluation unit 220 performs SAD evaluation. This process (specifically, the image shifting unit 210 shifts image P7 horizontally by a predetermined number of pixels, and the similarity evaluation unit 220 compares the shifted image P7 with image P3 to determine the similarity) is repeated a predetermined number of times until the similarity evaluation unit 220 has determined the similarity between the shifted image P7 and image P3 a predetermined number of times. Then, the pixel shift range determination unit 240 detects two points with small SAD and determines the pixel shift range (also referred to as the search range or search area) of the stereo matching processing unit S2 (the image shifting unit 210). The pixel (parallax) with the smallest SAD is included between these two detected points. Therefore, the pixel shift range determination unit 240 determines and outputs, for example, the area between the two detected points as the pixel shift range of the stereo matching processing unit S2 (the image shifting unit 210).

[0049] Next, the stereo matching processing unit S2 will be described. Similar to FIG. 4 of the first embodiment, the stereo matching processing unit S2 shown in Fig. 6 is configured to include a one-dimensional compression unit 310, an image shifting unit 210, a one-dimensional compression unit 320, a similarity evaluation unit 220, and a disparity detection unit 230. The difference from the first embodiment is that the pixel shift range determined by the pixel shift range determination unit 240 of the stereo matching processing unit S1 described above is input to the image shifting unit 210.

[0050] In the stereo matching processing unit S2, the one-dimensional compression unit 310 vertically compresses image P4 input from the image processing unit 100 to obtain image P5 as a vertically compressed image. At this time, images P5 and P3 have approximately the same vertical pixel resolution (approximately the same). Next, the image shifting unit 210 shifts image P5 horizontally by a predetermined number of pixels based on the information obtained by the pixel shift range determination unit 240 (i.e., according to the pixel shift range determined by the pixel shift range determination unit 240). Then, the one-dimensional compression unit 320 compresses the shifted image P5 horizontally to obtain image P6 as a horizontally and vertically compressed image. At this time, images P6 and P3 have approximately the same horizontal and vertical pixel resolution (approximately the same). Thereafter, the similarity evaluation unit 220 evaluates the similarity between the shifted images P6 and P3.

[0051] After the evaluation by SAD, the image shifter 210 generates image P5 with a different shift amount (for example, shifting it by one pixel from the previous processing) based on the information obtained by the pixel shift range determiner 240 (i.e., according to the pixel shift range determined by the pixel shift range determiner 240).Then, the one-dimensional compressor 320 compresses the shifted image P5 in the horizontal direction to obtain image P6 as a horizontally and vertically compressed image.Then, the similarity evaluator 220 evaluates the similarity between the shifted image P6 and image P3. The process is repeated a predetermined number of times until the similarity evaluation unit 220 has determined the similarity between the shifted image P6 and image P3 a predetermined number of times (more specifically, the process of having the image shift unit 210 shift image P5 horizontally by a predetermined number of pixels according to the pixel shift range, having the one-dimensional compression unit 320 compress the shifted image P5 horizontally to generate image P6, and having the similarity evaluation unit 220 compare image P3 and image P6 to determine the similarity), and the disparity detection unit 230 determines the disparity Dh that minimizes the SAD.

[0052] An example of the disparity detection method of this embodiment is shown in Figure 7. The SAD evaluated by the stereo matching processing unit S1 is indicated by a circle, and the SAD evaluated by the stereo matching processing unit S2 is indicated by a square. Here, it is assumed that image P4 has three times the resolution of image P3.

[0053] The stereo matching processing unit S1 obtains the SAD for every other pixel. Two points with the smallest SAD are detected, and the stereo matching processing unit S2 performs the processing between them. In other words, the area between the two detected points is used as the pixel shift range of the stereo matching processing unit S2 (the image shift unit 210), and the stereo matching processing is performed. By performing such stereo matching processing, the number of iterations of the stereo matching processing unit S2, indicated by squares, can be reduced compared to the processing method of the first embodiment, and the disparity at which the SAD becomes the minimum value can be efficiently determined.

[0054] In this embodiment, the stereo matching processing section S1 is performed first, followed by the stereo matching processing section S2, but it is not necessary to perform the same processing repeatedly. If the distance to the target object does not vary significantly over time, this processing can be performed once, and thereafter, only the stereo matching processing section S2 can be performed. For example, in the field of in-vehicle sensing, this processing can be used in adaptive cruise control (ACC). ACC is a function that automatically operates the accelerator and brake while driving to maintain a constant distance from the vehicle ahead. With this function, the distance from the vehicle ahead does not change significantly over a short period of time. Therefore, by performing this processing once, an approximate distance to the vehicle ahead can be obtained, and thereafter, only the stereo matching processing section S2 can be performed to measure the accurate distance to the vehicle ahead.

[0055] The stereo matching processing unit S2 of this embodiment uses the processing of the first embodiment, but may also use other processing such as that shown in FIG. 8. In this case, the stereo matching processing unit S2 shifts the high-resolution image P4 horizontally by a predetermined number of pixels based on the information obtained by the pixel shift range determination unit 240 (i.e., according to the pixel shift range determined by the pixel shift range determination unit 240) using the image shift unit 210. The compression unit 300 then compresses the shifted image horizontally and vertically to obtain image P8 as a horizontally and vertically compressed image. The similarity evaluation unit 220 then evaluates the SAD, which is the similarity between image P8 and image P3. At this time, image P8 and image P3 have approximately the same pixel resolution in the horizontal and vertical directions. Furthermore, this processing is repeated until the similarity evaluation unit 220 has determined the similarity between image P8 and image P3 a predetermined number of times, and the disparity detection unit 230 determines the disparity that minimizes the SAD.

[0056] In this embodiment, the stereo matching processing unit is divided into two, the stereo matching processing unit S1 and the stereo matching processing unit S2, but this does not necessarily have to be divided. For example, the pixel shift amount may be increased in the stereo parallax image generation unit 200 shown in Fig. 4 of the first embodiment, and the pixel shift amount may be reduced based on the result. In other words, for example, the pixel shift amount may be variable in the stereo parallax image generation unit 200 shown in Fig. 4 of the first embodiment, and the stereo matching processing may be performed.

[0057] The starting pixel on the horizontal axis of the graph shown in FIG. 7 may be either the small side (far side) or the large side (near side). The size of the starting point depends on the design value of the system. For example, by setting the starting point near the parallax amount of the maximum or minimum distance measured by the system, the amount of compression processing can be reduced. The size of the ending point can be set near the pixel where the SAD is minimum (for example, a few pixels after the SAD starts to increase after decreasing), for example, to be the ending point, thereby reducing the amount of compression processing.

[0058] [summary] As described above, the image processing device 50 of this embodiment includes an image acquisition unit 20 that acquires a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image; a first compression unit (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a first compressed image (image P5); and a one-dimensional compression unit 311 that compresses the first compressed image (image P5) in the first direction (horizontal direction). a second compression unit (one-dimensional compression unit 320) that compresses the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6); a first similarity evaluation unit (similarity evaluation unit 220) that compares the first image (image P3) and the second compressed image (image P6) to determine a first similarity; and a disparity detection unit 230 that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the first similarity determined by the first similarity evaluation unit (similarity evaluation unit 220) (Examples 1 and 2).

[0059] Furthermore, the image processing device 50 of this embodiment repeatedly executes the following process until the first similarity evaluation unit (similarity evaluation unit 220) has determined the first similarity a predetermined number of times: causing the first image shift unit (image shift unit 210) to shift the first compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels; causing the second compression unit (one-dimensional compression unit 320) to compress the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6); and causing the first similarity evaluation unit (similarity evaluation unit 220) to compare the first image (image P3) and the second compressed image (image P6) to determine the first similarity (Examples 1 and 2).

[0060] The image processing device 50 of this embodiment also includes a third compression unit (compression unit 330) that compresses the second image (image P4) in the first direction (horizontal direction) and the second direction (vertical direction) to generate a third compressed image (image P7), a second image shift unit (image shift unit 210) that shifts the third compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, a second similarity evaluation unit (similarity evaluation unit 220) that compares the shifted third compressed image (image P7) with the first image (image P3) to obtain a second similarity, and a second similarity evaluation unit (similarity evaluation unit 220) that compares the shifted third compressed image (image P7) with the first image (image P3). and a pixel shift range determination unit 240 that determines a pixel shift range (in the first direction (horizontal direction)) in the first image shift unit (image shift unit 210) based on the second similarity determined by the pixel shift range determination unit 220, and the first image shift unit (image shift unit 210) shifts the first compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels in accordance with the pixel shift range determined by the pixel shift range determination unit 240 (within the range of the pixel shift range determined by the pixel shift range determination unit 240) (Example 2).

[0061] The image processing device 50 of this embodiment includes an image acquisition unit 20 that acquires a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image, and first and second stereo matching processing units (S1 and S2). The first stereo matching processing unit (stereo matching processing unit S1) includes a first compression unit (compression unit 330) that compresses the second image (image P4) in a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned and in a second direction (vertical direction) that is substantially perpendicular to the first direction (horizontal direction) to generate a first compressed image (image P7), a first image shift unit (image shift unit 210) that shifts the first compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, and a second image shift unit (image shift unit 210) that shifts the first compressed image (image P7) by a predetermined number of pixels in the first direction (horizontal direction). The stereo matching processing system includes a first similarity evaluation unit (similarity evaluation unit 220) that compares the shifted first compressed image (image P7) with the first image (image P3) to determine a first similarity, and a pixel shift range determination unit 240 that determines a pixel shift range (in the first direction (horizontal direction)) in the second stereo matching processing unit (stereo matching processing unit S2) based on the first similarity determined by the first similarity evaluation unit (similarity evaluation unit 220). The second stereo matching processing unit (stereo matching processing unit S2) shifts the second image (image P4) (by a predetermined number of pixels in the first direction (horizontal direction)) in accordance with the pixel shift range determined by the pixel shift range determination unit 240 (within the pixel shift range determined by the pixel shift range determination unit 240) to determine a disparity (between images P3 and P4 or between images P1 and P2) (Example 2).

[0062] In the image processing device 50 of this embodiment, the second stereo matching processing unit (stereo matching processing unit S2) includes a second compression unit (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a second compressed image (image P5), and a second image shift unit (one-dimensional compression unit 310) that shifts the second compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels in accordance with the pixel shift range (within the range of the pixel shift range). a first image shifting unit 210), a third compression unit (one-dimensional compression unit 320) that compresses the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6), a second similarity evaluation unit (similarity evaluation unit 220) that compares the first image (image P3) and the third compressed image (image P6) to determine a second similarity, and a disparity detection unit 230 that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the second similarity determined by the second similarity evaluation unit (similarity evaluation unit 220) (Example 2).

[0063] Furthermore, the image processing device 50 of this embodiment repeatedly executes the following process until the second similarity evaluation unit (similarity evaluation unit 220) has determined the second similarity a predetermined number of times: causing the second image shift unit (image shift unit 210) to shift the second compressed image (image P5) in the first direction (horizontal direction) in accordance with the pixel shift range (within the pixel shift range); causing the third compression unit (one-dimensional compression unit 320) to compress the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6); and causing the second similarity evaluation unit (similarity evaluation unit 220) to compare the first image (image P3) and the third compressed image (image P6) to determine the second similarity (Example 2).

[0064] The image processing method of this embodiment includes an image acquisition process (image acquisition unit 20) for acquiring a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image; a first compression process (one-dimensional compression unit 310) for compressing the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned to generate a first compressed image (image P5); and a second compression process (one-dimensional compression unit 310) for shifting the first compressed image (image P5) by a predetermined number of pixels in the first direction (horizontal direction). This is an image processing method in which a computer executes a first image shifting process (image shifting unit 210), a second compression process (one-dimensional compression unit 320) that compresses the shifted first compressed image (image P5) in the first direction (horizontal direction) to generate a second compressed image (image P6), a first similarity evaluation process (similarity evaluation unit 220) that compares the first image (image P3) and the second compressed image (image P6) to determine a first similarity, and a disparity detection process (disparity detection unit 230) that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the first similarity determined by the first similarity evaluation process (similarity evaluation unit 220).

[0065] The image processing method of this embodiment is an image processing method in which a computer executes an image acquisition process (image acquisition unit 20) for acquiring a first image (image P1) captured by a first camera (camera 1) and a second image (image P2) captured by a second camera (camera 2), the second image (image P2) having a higher pixel resolution than the first image, and first and second stereo matching processes (stereo matching processing units S1 and S2). The first stereo matching process (stereo matching processing unit S1) includes a first compression process (compression unit 330) for compressing the second image (image P4) in a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned and in a second direction (vertical direction) substantially perpendicular to the first direction (horizontal direction) to generate a first compressed image (image P7), a first image shift process (image shift unit 210) for shifting the first compressed image (image P7) in the first direction (horizontal direction) by a predetermined number of pixels, and a second image shift process (image shift unit 210) for shifting the shifted image. a first similarity evaluation process (similarity evaluation unit 220) for comparing the first compressed image (image P7) with the first image (image P3) to obtain a first similarity; and a pixel shift range determination process (pixel shift range determination unit 240) for determining a pixel shift range (in the first direction (horizontal direction)) in the second stereo matching process (stereo matching processing unit S2) based on the first similarity obtained by the first similarity evaluation process (similarity evaluation unit 220). In the stereo matching process (stereo matching processing unit S2), the second image (image P4) is shifted (by a predetermined number of pixels in the first direction (horizontal direction)) in accordance with the pixel shift range determined by the pixel shift range determination process (pixel shift range determination unit 240) (within the pixel shift range determined by the pixel shift range determination process (pixel shift range determination unit 240)), thereby obtaining disparity (between image P3 and image P4 or between image P1 and image P2) (Example 2).

[0066] Furthermore, in the image processing method of this embodiment, the second stereo matching process (stereo matching processing unit S2) includes a second compression process (one-dimensional compression unit 310) that compresses the second image (image P4) in a second direction (vertical direction) that is substantially perpendicular to a first direction (horizontal direction) in which the first camera (camera 1) and the second camera (camera 2) are aligned, to generate a second compressed image (image P5); and a second image shift process (image shift unit 210) that shifts the second compressed image (image P5) in the first direction (horizontal direction) by a predetermined number of pixels in accordance with the pixel shift range (within the range of the pixel shift range). This is an image processing method (Example 2) that includes a third compression process (one-dimensional compression unit 320) that compresses the shifted second compressed image (image P5) in the first direction (horizontal direction) to generate a third compressed image (image P6), a second similarity evaluation process (similarity evaluation unit 220) that compares the first image (image P3) and the third compressed image (image P6) to determine a second similarity, and a disparity detection process (disparity detection unit 230) that determines the disparity (between image P3 and image P4 or between image P1 and image P2) based on the second similarity determined by the second similarity evaluation process (similarity evaluation unit 220).

[0067] According to the image processing device and image processing method of this embodiment, in stereo vision using two cameras with different resolutions (different resolution stereo cameras), by repeatedly shifting and compressing a high-resolution image and deriving the similarity with a low-resolution image, it is possible to provide an image processing device and image processing method that can perform highly accurate disparity detection using a high-resolution image and a low-resolution image.

[0068] <Modification> The present invention is not limited to the above-described embodiments, and includes various other modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0069] Furthermore, the above-described configurations, functions, and units 100, 20, 20a-23a, 20b-23b, 200, 600, 700, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. As shown in FIG. 1B , the above-described configurations, functions, etc. may be implemented in software by a processor 122, such as a CPU, interpreting and executing programs that implement the respective functions. Information such as programs, tables, and files that implement the respective functions can be stored in a hard disk, or in a memory device such as memory 121, an SSD, or a recording medium such as an integrated circuit (IC) card, a secure digital (SD) card, or a digital versatile disc (DVD).

[0070] In addition, in each embodiment, the control lines and information lines shown are those that are considered necessary for explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0071] 1. Camera 2. Camera 50...Image processing device 100...Image processing unit 200...Stereo parallax image generation unit 210...Image shift unit 220...Similarity evaluation unit 230... Parallax detection unit 240...Pixel shift range determination unit 300...Compression section 310...One-dimensional compression section 320...One-dimensional compression section 330...Compression section 600...3D object detection unit 700...Alarm control unit

Claims

1. an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression unit that compresses the second image in a second direction that is substantially perpendicular to a first direction in which the first camera and the second camera are aligned, to generate a first compressed image; a first image shifting unit that shifts the first compressed image by a predetermined number of pixels in the first direction; a second compression unit that compresses the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation unit that compares the first image with the second compressed image to obtain a first similarity; a disparity detection unit that determines a disparity based on the first similarity determined by the first similarity evaluation unit.

2. 2. The image processing device according to claim 1, until the first similarity evaluation unit calculates the first similarity a predetermined number of times. causing the first image shifting unit to shift the first compressed image by a predetermined number of pixels in the first direction; causing the second compression unit to compress the shifted first compressed image in the first direction to generate a second compressed image; an image processing device that repeatedly executes a process of causing the first similarity evaluation unit to compare the first image with the second compressed image to obtain a first similarity;

3. 2. The image processing device according to claim 1, The image processing device wherein the first compression unit compresses the second image in the second direction so that the resolutions of the first image and the first compressed image with respect to pixels in the second direction are approximately the same.

4. 2. The image processing device according to claim 1, the second compression unit compresses the shifted first compressed image in the first direction so that the resolutions of the second compressed image and the first image with respect to pixels in the first direction are approximately the same.

5. 2. The image processing device according to claim 1, a third compression unit that compresses the second image in the first direction and the second direction to generate a third compressed image; a second image shifting unit that shifts the third compressed image in the first direction by a predetermined number of pixels; a second similarity evaluation unit that compares the shifted third compressed image with the first image to obtain a second similarity; a pixel shift range determination unit that determines a pixel shift range in the first image shift unit based on the second similarity calculated by the second similarity evaluation unit, The first image shifting unit shifts the first compressed image in the first direction by a predetermined number of pixels in accordance with the pixel shift range determined by the pixel shift range determining unit.

6. an image acquisition unit that acquires a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; first and second stereo matching processors; The first stereo matching processing unit a first compression unit that compresses the second image in a first direction in which the first camera and the second camera are aligned and in a second direction that is substantially perpendicular to the first direction, to generate a first compressed image; a first image shifting unit that shifts the first compressed image by a predetermined number of pixels in the first direction; a first similarity evaluation unit that compares the shifted first compressed image with the first image to obtain a first similarity; a pixel shift range determination unit that determines a pixel shift range in the second stereo matching processing unit based on the first similarity calculated by the first similarity evaluation unit, The image processing device, wherein the second stereo matching processing unit obtains parallax by shifting the second image in accordance with the pixel shift range determined by the pixel shift range determination unit.

7. 7. The image processing device according to claim 6, The second stereo matching processing unit a second compression unit that compresses the second image in a second direction that is substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a second compressed image; a second image shifting unit that shifts the second compressed image in the first direction by a predetermined number of pixels in accordance with the pixel shift range; a third compression unit that compresses the shifted second compressed image in the first direction to generate a third compressed image; a second similarity evaluation unit that compares the first image with the third compressed image to obtain a second similarity; a disparity detection unit that determines a disparity based on the second similarity determined by the second similarity evaluation unit.

8. 8. The image processing device according to claim 7, until the second similarity evaluation unit calculates the second similarity a predetermined number of times. causing the second image shifter to shift the second compressed image in the first direction by a predetermined number of pixels according to the pixel shift range; causing the third compression unit to compress the shifted second compressed image in the first direction to generate a third compressed image; an image processing device that repeatedly executes a process of causing the second similarity evaluation unit to compare the first image with the third compressed image to obtain a second similarity;

9. an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first compression process for compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a first compressed image; a first image shifting process for shifting the first compressed image by a predetermined number of pixels in the first direction; a second compression process compressing the shifted first compressed image in the first direction to generate a second compressed image; a first similarity evaluation process for comparing the first image with the second compressed image to obtain a first similarity; a disparity detection process for determining a disparity based on the first similarity determined by the first similarity evaluation process.

10. an image acquisition process for acquiring a first image captured by a first camera and a second image captured by a second camera, the second image having a higher pixel resolution than the first image; a first stereo matching process and a second stereo matching process, The first stereo matching process includes: a first compression process for compressing the second image in a first direction in which the first camera and the second camera are aligned and in a second direction substantially perpendicular to the first direction to generate a first compressed image; a first image shifting process for shifting the first compressed image by a predetermined number of pixels in the first direction; a first similarity evaluation process for comparing the shifted first compressed image with the first image to obtain a first similarity; a pixel shift range determination process for determining a pixel shift range in the second stereo matching process based on the first similarity calculated by the first similarity evaluation process, In the second stereo matching process, the second image is shifted in accordance with the pixel shift range determined in the pixel shift range determination process, thereby obtaining a parallax.

11. 11. An image processing method according to claim 10, The second stereo matching process a second compression process for compressing the second image in a second direction substantially perpendicular to a first direction in which the first camera and the second camera are aligned to generate a second compressed image; a second image shifting process for shifting the second compressed image in the first direction by a predetermined number of pixels according to the pixel shift range; a third compression process for compressing the shifted second compressed image in the first direction to generate a third compressed image; a second similarity evaluation process for comparing the first image with the third compressed image to obtain a second similarity; a parallax detection process for determining a parallax based on the second similarity determined by the second similarity evaluation process.

Images