Image processing apparatus

Abstract

An arithmetic device of this image processing apparatus: performs, on the basis of a parameter stored in a storage unit, parallelization on a stereo image composed of a first image and a second image to generate a third image and a fourth image; sets a reference region, including a distant subject image, in the third image; sets a search region, corresponding to the reference region, in the fourth image; calculates a positional deviation amount in the vertical direction between the third image and the fourth image by using the reference region in the third image and the search region in the fourth image; and corrects the parameter stored in the storage unit on the basis of the positional deviation amount.

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] Conventionally, there are known techniques for correcting vertical misalignment of stereo images captured by a stereo camera. Patent Document 1 describes a stereo image misalignment adjustment device that sets a search range in a comparison image using distance data corresponding to each of multiple reference areas randomly set on a reference image plane, calculates a matching point from the search range that has the highest correlation with the reference area, and calculates the vertical misalignment amount using the matching points. Here, the distance data corresponding to the reference area is calculated based on the parallax in the stereo images, i.e., the horizontal misalignment amount between the reference image and the comparison image.

[0003] Japanese Patent Application Laid-Open No. 2003-256875

[0004] The stereo image positional deviation adjustment device described in Patent Document 1 corrects vertical positional deviation using distance data calculated based on left-right parallax, but has the problem that if the vertical positional deviation is large, it is not possible to calculate left-right parallax, and therefore it is not possible to correct vertical positional deviation.

[0005] An object of the present invention is to provide an image processing device that can reliably correct positional deviation in the vertical direction even when the positional deviation in the vertical direction is large.

[0006] An image processing device according to one aspect of the present invention includes an arithmetic unit that, based on parameters stored in a memory unit, parallelizes a stereo image consisting of a first image and a second image to generate a third image and a fourth image, sets a reference area including an image of a distant subject in the third image, sets a search area corresponding to the reference area in the fourth image, calculates the amount of vertical positional shift between the third image and the fourth image using the reference area of ​​the third image and the search area of ​​the fourth image, and corrects the parameters stored in the memory unit based on the amount of positional shift.

[0007] According to the present invention, even if the positional deviation in the vertical direction is large, the positional deviation in the vertical direction can be reliably corrected.

[0008] FIG. 1 is a schematic diagram of a vehicle equipped with an image processing device according to a first embodiment. FIG. 2 is a schematic diagram showing the hardware configuration of the image processing device according to the first embodiment. FIG. 3 is a schematic diagram showing the functional configuration of the image processing device according to the first embodiment. FIG. 4 is a schematic diagram showing an example of a candidate area. FIG. 5 is a schematic diagram showing an example of a reference area. FIG. 6 is a schematic diagram showing an example of a search area. FIG. 7 is a flowchart showing an example of calculation processing executed by the image processing device according to the first embodiment. FIG. 8 is a flowchart showing an example of calculation processing executed by the image processing device according to a second embodiment.

[0009] First Embodiment An image processing apparatus according to an embodiment of the present invention will be described with reference to FIGS.

[0010] FIG. 1 is a schematic diagram of a vehicle equipped with an image processing device according to a first embodiment. The vehicle 1 is equipped with a first imaging device 2a, a second imaging device 2b, and an image processing device 3. In the following description, the first imaging device 2a and the second imaging device 2b are collectively referred to as the imaging device 2. The first imaging device 2a and the second imaging device 2b function as a so-called stereo camera. The imaging device 2 captures images of the area ahead of the vehicle 1 and outputs the captured images to the image processing device 3 in the form of imaging signals. The first imaging device 2a and the second imaging device 2b are installed so that the positions of their optical axes in the vertical direction (perpendicular direction) are approximately equal. The optical axis 4a of the first imaging device 2a is located slightly left of the center of the vehicle 1 in the horizontal direction. The optical axis 4b of the second imaging device 2b is located slightly right of the center of the vehicle 1 in the horizontal direction. In the following description, the distance between the optical axes 4a and 4b in the horizontal direction is referred to as the baseline length. The first imaging device 2a captures an image of a range 5a slightly to the left in front of the vehicle 1. The second imaging device 2b captures an image of a range 5b slightly to the right in front of the vehicle 1.

[0011] 2 is a schematic diagram showing the hardware configuration of an image processing device according to the first embodiment. The image capture device 2 includes an image sensor, such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and an imaging optical system. As described above, the first image capture device 2a and the second image capture device 2b are disposed horizontally spaced apart by the base line length, and therefore, parallax occurs in the captured images resulting from each image capture, depending on the distance from the image capture device 2 to the subject and the base line length.

[0012] The image processing device 3 is composed of a computer equipped with an arithmetic unit 11 such as a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP), a non-volatile memory 12 such as a read-only memory (ROM), a flash memory, or a hard disk drive, a volatile memory 13 called a random access memory (RAM), an input / output interface 14, and other peripheral circuits. These hardware components work together to run software and realize multiple functions. The image processing device 3 may be composed of a single computer or multiple computers. The arithmetic unit 11 may be an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0013] The nonvolatile memory 12 stores programs capable of executing various calculations. In other words, the nonvolatile memory 12 is a storage medium (storage device) from which programs realizing the functions of this embodiment can be read. The volatile memory 13 is a storage medium (storage device) that temporarily stores the results of calculations by the calculation device 11 and signals input from the input / output interface 14. The calculation device 11 is a device that loads the programs stored in the nonvolatile memory 12 into the volatile memory 13 and executes calculations, and performs predetermined calculations on data taken in from the input / output interface 14, the nonvolatile memory 12, and the volatile memory 13 in accordance with the programs. The nonvolatile memory 12 also stores camera parameters, which will be described later.

[0014] The input / output interface 14 is connected to the imaging device 2. The input section of the input / output interface 14 converts signals input from various devices (such as the imaging device 2) into data that can be calculated by the arithmetic device 11. The output section of the input / output interface 14 generates an output signal according to the calculation result of the arithmetic device 11, and outputs the signal to various devices (such as the imaging device 2).

[0015] 3 is a schematic diagram showing the functional configuration of the image processing device according to the first embodiment. The image processing device 3 includes a parallelization unit 31, a reference region setting unit 32, a search region setting unit 33, a matching unit 34, a deviation amount calculation unit 35, and a correction unit 36. Each of these functional units is functionally realized by the calculation device 11 executing the above-mentioned program.

[0016] The rectification unit 31 rectifies a stereo image consisting of a right image and a left image based on the camera parameters stored in the non-volatile memory 12 to generate a rectified right image and a rectified left image. The reference area setting unit 32 sets a reference area including an image of a distant subject in the rectified right image. The reference area setting unit 32 of this embodiment sets multiple reference areas. The search area setting unit 33 sets a search area corresponding to the reference area in the rectified left image. The search area setting unit 33 of this embodiment sets search areas corresponding to each of the multiple reference areas.

[0017] The matching unit 34 performs matching with the corresponding reference area within the search area and identifies the position where the reference area best matches. In other words, the matching unit 34 identifies the position within the search area where the correlation with the reference area is highest. The deviation amount calculation unit 35 calculates the amount of vertical positional deviation between the rectified right image and the rectified left image using the matching result by the matching unit 34. The correction unit 36 ​​corrects the camera parameters stored in the non-volatile memory 12 based on the amount of positional deviation calculated by the deviation amount calculation unit 35.

[0018] The rectification process performed by the rectification unit 31 will now be described. The rectification process is what is known as stereo rectification. That is, it is a process of converting images so that the same point in the world coordinate system is aligned at the same height in both the right image and the left image. The rectification process is achieved by performing projective transformation from a state in which geometric distortion occurs due to the imaging optical system onto a preset projection plane, for example, a perspective projection plane. As described above, the non-volatile memory 12 stores camera parameters of the imaging optical system. The rectification unit 31 uses these camera parameters to perform projective transformation of the right image to generate a rectified right image. Similarly, the rectification unit 31 uses these camera parameters to perform projective transformation of the left image to generate a rectified left image.

[0019] The following describes how the reference region setting unit 32 sets a plurality of candidate regions in the parallelized right image. The reference region setting unit 32 selects one of the candidate regions and sets a plurality of reference regions inside the selected candidate region.

[0020] 4 is a schematic diagram showing an example of a candidate region. Each candidate region 42 has a rectangular shape with its long side oriented horizontally. The multiple candidate regions 42 are set along the vertical direction near a vanishing point 41 in the rectified right image 40.

[0021] The reference area setting unit 32 identifies one or more candidate areas that include an image of a distant subject from the multiple candidate areas 42. The reference area setting unit 32 calculates the amount of change in luminance value for each pixel between frames for each of the multiple candidate areas 42. That is, the reference area setting unit 32 calculates the amount of change in luminance value for each pixel for each candidate area 42 between a rectified right image based on stereo images captured at a certain time t1 and a rectified right image based on stereo images captured at a time t2 that is later than time t1. The amount of change in luminance value is expressed using a measure such as SAD (sum of absolute difference).

[0022] As shown in FIG. 4 , the angle of view of the imaging device 2 indicates that the image of a distant subject is likely to be near the vanishing point 41. Furthermore, since the distance between the vehicle 1 and the subject does not change significantly between frames, the brightness value of each pixel of the distant subject image is likely to not change significantly between frames. Therefore, the reference area setting unit 32 identifies candidate areas where the SAD is equal to or less than a predetermined threshold value to identify candidate areas that include distant subject images. A candidate area where the SAD is equal to or less than a predetermined threshold value is an area that includes a subject image in which the brightness value of each pixel does not change significantly between frames even when the vehicle 1 is moving, and is therefore likely to include a distant subject image. The reference area setting unit 32 estimates such a candidate area as including a distant subject image. In this embodiment, the reference area setting unit 32 identifies one candidate area with the smallest SAD from among the candidate areas narrowed down in this manner. Then, the reference area is set inside that single candidate area. Since it is expected that the left-right parallax of a distant subject image will be small (for example, almost zero), even if there is a large vertical positional deviation, it is expected that this will not interfere with matching with the search area described below.

[0023] FIG. 5 is a schematic diagram showing an example of a reference region. The reference region 51 is a region of a predetermined size, such as 16 pixels horizontally and 16 pixels vertically. The reference region setting unit 32 of this embodiment randomly sets a predetermined number of reference regions 51 inside the candidate region 52 in the rectified right image 50. The number of reference regions 51 may be a fixed number, or may be varied depending on the situation. Furthermore, rather than setting the position of the reference region 51 randomly, it may be set at a fixed position, or may be set using the brightness value of each pixel in the candidate region 52.

[0024] The search area setting process performed by the search area setting unit 33 will be described.

[0025] 6 is a schematic diagram showing an example of a search region. The search region setting unit 33 checks the position of each reference region 62 in the rectified right image 61. Then, it sets search regions 63 at the same positions (corresponding positions) in the rectified left image 60. In this case, the size of each search region 63 is set to be larger than the size of the reference region 62, and the center coordinates are set to coincide with those of the reference region 62. For example, if the reference region 62 is 16 pixels wide and 16 pixels high, the search region 63 is set to be 32 pixels wide and 32 pixels high.

[0026] The matching process performed by the matching unit 34 will now be described. The matching unit 34 finds the sequence of luminance values ​​within the search area 63 that is most similar to the sequence of luminance values ​​within the reference area 62. For example, the matching unit 34 calculates the SAD with the luminance values ​​of the reference area 62 while shifting the search area 63 vertically and horizontally by one pixel starting from the upper left corner, and identifies the position where the SAD is smallest. In this way, the matching unit 34 matches the corresponding reference area 62 within the search area 63, and identifies the position where the reference area 62 best matches, or the position where the correlation with the reference area 62 is highest. In the following description, the best-matched position (coordinates) for each reference area 62 is referred to as the matching point of that reference area 62. The matching unit 34 stores the matching point for each reference area 62 in the volatile memory 13. Note that no matching point is stored for a search area where the SAD is not below a predetermined threshold, i.e., for a search area that does not match well with the reference area.

[0027] The displacement calculation process performed by the displacement calculation unit 35 will now be described. The displacement calculation unit 35 performs an approximation calculation using a linear function using the multiple coincidence points stored by the matching unit 34, and the value of the intercept is used as the vertical displacement amount. For example, if the position of the imaging device 2 and the camera parameters are perfectly matched, the vertical displacement amount between the parallelized right image and the parallelized left image will be zero. In this case, the y coordinates of the coincidence points will match, and the straight line derived by the approximation calculation will be a horizontal line.

[0028] The correction unit 36 ​​reflects the calculated vertical positional deviation in the camera parameters. Specifically, the correction unit 36 ​​corrects the camera parameters so that the vertical positional deviation between the coincident points is small (for example, to zero). This allows the parallelization unit 31 to accurately perform parallelization even if the vertical positions of the first image capture device 2 a and the second image capture device 2 b are misaligned due to, for example, poor installation accuracy when the image capture device 2 is attached to the vehicle 1 or distortion of the housing of the image capture device 2 or the vehicle 1 due to aging or temperature changes.

[0029] 7 is a flowchart showing an example of the calculation process executed by the image processing apparatus according to the first embodiment. The calculation device 11 periodically executes the calculation process shown in FIG.

[0030] In step S100, the arithmetic unit 11 receives stereo images, i.e., right and left images, captured by the imaging device 2 at time t1 from the imaging device 2. In step S110, the arithmetic unit 11 receives stereo images, i.e., right and left images, captured by the imaging device 2 at time t2, which is later than time t1, from the imaging device 2.

[0031] In step S120, the rectification unit 31 performs the above-described rectification process on each of the stereo images received in step S100 and step S110, thereby generating a rectified right image and a rectified left image corresponding to time t1, and a rectified right image and a rectified left image corresponding to time t2, respectively.

[0032] In step S130, the reference region setting unit 32 calculates the SAD for each candidate region using the rectified right image corresponding to time t1 and the rectified right image corresponding to time t2. In step S140, the reference region setting unit 32 identifies one candidate region with the smallest SAD. That is, the reference region setting unit 32 identifies one candidate region that includes an image of a distant subject. In step S150, the reference region setting unit 32 sets multiple reference regions within one candidate region identified in step S140. For example, the reference region setting unit 32 randomly arranges the reference regions so that they are evenly distributed throughout the entire candidate region.

[0033] In step S160, the search area setting unit 33 sets a search area corresponding to each reference area in the rectified left image. In step S170, the matching unit 34 matches each reference area with the reference area inside the corresponding search area, and identifies the position where the reference area best matches. The matching unit 34 stores the position of the matched reference area as a matching point in the volatile memory 13. In step S180, the deviation amount calculation unit 35 calculates the vertical positional deviation amount between the rectified right image and the rectified left image using the matching point stored in the volatile memory 13. In step S190, the correction unit 36 ​​corrects the camera parameters stored in the non-volatile memory 12 using the vertical positional deviation amount calculated in step S180.

[0034] According to the above-described first embodiment, the following advantageous effects are achieved.

[0035] (1) The arithmetic unit 11 rectifies a stereo image consisting of a right image (first image) and a left image (second image) based on camera parameters stored in the nonvolatile memory 12 (storage unit) to generate a rectified right image (third image) and a rectified left image (fourth image). The arithmetic unit 11 sets a reference area including a distant subject image in the rectified right image (third image) and sets a search area corresponding to the reference area in the rectified left image (fourth image). The arithmetic unit 11 calculates the amount of vertical positional deviation between the rectified right image (third image) and the rectified left image (fourth image) using the reference area in the rectified right image (third image) and the search area in the rectified left image (fourth image), and corrects the camera parameters stored in the nonvolatile memory 12 (storage unit) based on the amount of positional deviation. This allows for reliable correction of the vertical positional deviation even when the vertical positional deviation is large.

[0036] (2) The calculation device 11 compares a collimated right image (third image) based on a right image (first image) captured at a first time with a collimated right image (third image) based on a right image (first image) captured at a second time after the first time, and estimates that the image of a distant subject is included in an area where the amount of change between the first time and the second time is equal to or less than a predetermined threshold. This makes it possible to reliably capture an area that includes the image of a distant subject.

[0037] (3) The calculation device 11 sets a plurality of candidate areas near the vanishing point and sets the reference area inside one of the candidate areas, thereby ensuring that the reference area includes the image of a distant subject.

[0038] (4) The calculation device 11 sets multiple rectangular candidate areas with the long sides aligned horizontally. This allows the reference areas to be set aligned horizontally (x-axis direction), making it easier to calculate the vertical positional deviation amount by linear approximation.

[0039] (5) The calculation unit 11 calculates the amount of change between the first time and the second time for each candidate area, and sets the reference area inside the candidate area with the smallest amount of change. This allows the reference area as a whole to include as much of the distant subject image as possible.

[0040] (6) The calculation device 11 sets the position of the reference area in the rectified right image (third image) as the position of the search area in the rectified left image (fourth image). In this way, the parallax obtained from the stereo images is not used to position the search area, so that the reference area and the search area can be reliably matched even when the amount of positional deviation is large and the parallax cannot be accurately obtained.

[0041] Second Embodiment An image processing device according to a second embodiment of the present invention will be described with reference to Fig. 8. Note that components that are the same as or equivalent to those described in the first embodiment are given the same reference symbols, and differences will be mainly described.

[0042] In the first embodiment, the reference area setting unit 32 identifies candidate areas where the SAD is equal to or less than a predetermined threshold, and if there are multiple such candidate areas, selects the candidate area with the smallest SAD and sets the reference area inside that candidate area. In the present embodiment, the reference area setting unit 32 identifies candidate areas where the SAD is equal to or less than a predetermined threshold, and if there are multiple such candidate areas, selects the lowest candidate area and sets the reference area inside that candidate area.

[0043] Fig. 8 is a flowchart similar to Fig. 7, showing an example of calculation processing executed by the image processing apparatus according to the second embodiment. In the flowchart of Fig. 8, processing in steps S241 to S242 is executed instead of processing in step S140 in the flowchart of Fig. 7.

[0044] In step S241, the reference area setting unit 32 identifies one or more candidate areas whose SAD is equal to or less than a predetermined threshold. That is, the reference area setting unit 32 identifies one or more candidate areas that include an image of a distant subject. In step S242, the reference area setting unit 32 identifies one candidate area at the bottom of the screen from among the one or more candidate areas. Then, in step S150, the reference area setting unit 32 sets multiple reference areas within the one candidate area identified in step S242.

[0045] According to the second embodiment described above, the following advantageous effects are achieved.

[0046] (1) The calculation device 11 calculates the amount of change between the first time and the second time for each candidate area, and sets the reference area inside the candidate area located at the bottom among the candidate areas for which the amount of change is equal to or less than a predetermined threshold. This allows the candidate area to include images with many features, such as a distant road surface, rather than images with few features, such as the sky, thereby improving the accuracy of calculating the amount of positional deviation.

[0047] The following modified examples are also within the scope of the present invention, and it is possible to combine the configuration shown in the modified example with the configuration described in the above embodiment, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0048] <Variation 1> The roles of the right image and the left image may be reversed. For example, the candidate region and the reference region may be set in the rectified left image instead of the rectified right image, and the search region may be set and the matching with the candidate region may be performed in the rectified right image instead of the rectified left image.

[0049] <Modification 2> The measure representing the amount of change in luminance value between frames may be a measure other than SAD, such as SSD (Sum of Squared Difference).

[0050] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0051] REFERENCE SIGNS LIST 1 vehicle, 2 imaging device, 2a first imaging device, 2b second imaging device, 3 image processing device, 11 arithmetic unit, 12 non-volatile memory, 13 volatile memory, 14 input / output interface, 31 parallelization unit, 32 reference area setting unit, 33 search area setting unit, 34 matching unit, 35 deviation amount calculation unit, 36 correction unit

Claims

1. An image processing apparatus including an arithmetic unit, wherein the arithmetic unit: generates a third image and a fourth image by parallelizing a stereo image composed of a first image and a second image based on parameters stored in a storage unit; sets a reference region including a distant subject image in the third image; sets a search region corresponding to the reference region in the fourth image; calculates a vertical displacement amount between the third image and the fourth image using the reference region of the third image and the search region of the fourth image; and corrects the parameters stored in the storage unit based on the displacement amount.

2. The image processing apparatus according to claim 1, wherein the arithmetic unit compares the third image based on the first image captured at a first time with the third image based on the first image captured at a second time after the first time, and estimates that the distant subject image is included in a region where a change amount between the first time and the second time is equal to or less than a predetermined threshold value.

3. The image processing apparatus according to claim 2, wherein the arithmetic unit sets a plurality of candidate regions near a vanishing point and sets the reference region inside any one of the plurality of candidate regions.

4. The image processing apparatus according to claim 3, wherein the arithmetic unit sets a plurality of rectangular candidate regions having a horizontal direction as a long side direction.

5. The image processing apparatus according to claim 4, wherein the arithmetic unit calculates the change amount between the first time and the second time for each of the candidate regions and sets the reference region inside the candidate region having the smallest change amount.

6. The image processing apparatus according to claim 4, wherein the arithmetic unit calculates the change amount between the first time and the second time for each of the candidate regions and sets the reference region inside the candidate region that is located most downward among the candidate regions where the change amount is equal to or less than the predetermined threshold value.

7. The image processing apparatus according to claim 1, wherein the arithmetic unit sets the position of the reference region in the third image as the position of the search region in the fourth image.

Images