Image processing device

Abstract

Provided is an image processing device capable of achieving a different-angle-of-view stereo with high disparity performance in overlapping image capturing ranges of two image capturing devices having different optical specifications (angle of view, focal length). Magnification adjustment for matching the size of an image of a narrow-angle camera 101 to that of a wide-angle camera 102 and resolution adjustment for matching the degrees of sharpness (blur degrees) of the images of the two cameras are performed, and magnification adjustment is performed in consideration of all the pixel data in the range of an image P1 of the narrow-angle camera 101 in which disparity calculation is performed. In other words, along with the magnification adjustment for matching the size of the image of the narrow-angle camera 101 to that of the wide-angle camera 102, all the pixel data to be used for the disparity calculation of a target object is used for the magnification adjustment (no decimation is performed).

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] In recent years, in-vehicle camera devices that use multiple cameras to detect pedestrians, vehicles, etc. Stereo cameras, one type of in-vehicle camera device, are devices that calculate the positional deviation (parallax) of the same object in a pair of images captured simultaneously by left and right cameras using a method such as template matching, and then calculate the position of the object in real space based on the parallax using a well-known transformation formula.

[0003] Conventional stereo cameras perform image processing under the assumption that the optical specifications (angle of view, focal length) of the two cameras are the same. Therefore, if the optical specifications (angle of view, focal length) of the two cameras are different (hereinafter, such camera devices may be referred to as different-angle-of-view stereo cameras), and differences in image size or image sharpness (blurring) occur, there is a problem that matching processing cannot be performed or the desired stereo performance (parallax performance) cannot be obtained. In other words, if the optical specifications of the two cameras are different, differences in image size and image sharpness (blurring) occur, and conventional image processing cannot achieve high parallax performance in different-angle-of-view stereo. Regarding technology for matching such different-angle-of-view stereo cameras, for example, Patent Document 1 describes a technology for matching the size of images from two cameras.

[0004] The abstract of Patent Document 1 describes the problem as "improving the accuracy of parallax calculation when capturing images using lenses with different angles of view (imaging magnifications) for each camera constituting a stereo camera." The abstract also describes the solution as "forming a stereo camera using left and right cameras with different angles of view, and making the sensor pitch of the sensor of the right camera on the narrower angle side wider than the sensor pitch of the sensor of the left camera on the wide-angle side. In this way, when the captured image of the sensor of the left camera on the wide-angle side, which has the narrower sensor pitch, is enlarged in post-processing, it is possible to obtain an captured image in which the sampling intervals of the left and right subjects are aligned. By performing parallax calculation using this captured image, it is possible to improve the accuracy of parallax calculation."

[0005] Furthermore, paragraph 0048 of Patent Document 1 states, "For example, if the left camera 4 is the wide-angle camera and the right camera 6 is the narrow-angle camera, the sensor pitch of the narrow-angle camera 6 is increased to approach the angle of view per unit pixel of the wide-angle camera 4. In other words, the sampling pitch of sensor 10b of the narrow-angle camera 6 is increased to approach the sampling pitch of sensor 10a of the wide-angle camera 4."

[0006] Japanese Patent Application Laid-Open No. 2022-101302

[0007] In Patent Document 1, the sensor pitch of the sensor (imaging element) of the narrow-angle camera is changed to align the angle of view per unit pixel (the size of the image in the captured image), but changing the sensor pitch of the sensors (imaging elements) of the two cameras complicates product specifications and increases product costs, making it impractical. Furthermore, in Patent Document 1, the sampling pitch of the sensor (imaging element) of the narrow-angle camera is changed to align the angle of view per unit pixel (the size of the image in the captured image), but in principle, the sampling of the sensor (imaging element) of the high-resolution narrow-angle camera is thinned out to align the image size to the image size of the wide-angle camera, which means that pixel data used to calculate the parallax of the object is thinned out, and there is a risk that the stereo performance (parallax performance) will deteriorate as the object becomes farther away, for example.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device that can achieve different angle-of-view stereo with high parallax performance in the overlapping imaging ranges of two imaging devices with different optical specifications (angle of view, focal length).

[0009] In order to solve the above problem, the image processing device of the present invention comprises a narrow-angle imaging device with a narrow-angle imaging range and a wide-angle imaging device with a wider-angle imaging range than the narrow-angle imaging device, the narrow-angle imaging device and the wide-angle imaging device imaging imaging ranges that overlap at least partially, processing image information obtained from the overlapping imaging ranges, and performing stereo processing including parallax calculation of target objects in the overlapping imaging ranges, and is characterized in that the first image information obtained from the narrow-angle imaging device is subjected to magnification adjustment corresponding to the second image information obtained from the wide-angle imaging device, and the magnification adjustment is performed taking into account all pixel data of the first image information within the range where the parallax calculation is performed.

[0010] According to the present invention, it is possible to realize different-angle-of-view stereo with high parallax performance in the overlapping imaging ranges of two imaging devices with different optical specifications (angle of view, focal length). Note that problems, configurations, and effects other than those described above will become clear from the description of the following embodiments.

[0011] A top view of a vehicle equipped with a stereo camera device as an image processing device according to an embodiment of the present invention. A functional block diagram of the stereo camera device as an image processing device according to an embodiment of the present invention. A diagram for schematically explaining the processing content of a magnification adjustment unit. A diagram for explaining the effect of magnification adjustment on parallax variation. A diagram for explaining the effect of resolution adjustment on parallax variation.

[0012] Hereinafter, an embodiment of the image processing apparatus of the present invention will be described with reference to the drawings.

[0013] FIG. 1 is a top view of a vehicle V equipped with a stereo camera device 100 as an image processing device according to this embodiment. Two cameras (a narrow-angle camera 101 and a wide-angle camera 102) are attached as follows: The narrow-angle camera (also referred to as a telephoto camera) 101 is a monocular camera with a narrow-angle imaging range, a focal length f1 (>f2), and is installed to capture an image of a range of a field of view α1 (<α2) in front of the vehicle. The wide-angle camera 102 is a monocular camera with a wider-angle imaging range than the narrow-angle camera (telephoto camera) 101, a focal length f2 (<f1), and is installed to capture an image of a range of a field of view α2 (>α1) in front of the vehicle. The imaging ranges of the narrow-angle camera 101 and the wide-angle camera 102 partially overlap (share). In other words, the narrow-angle camera 101 and the wide-angle camera 102 capture imaging ranges that at least partially overlap (hereinafter also referred to as overlapping imaging ranges). Therefore, stereo camera device 100 processes image information obtained from the overlapping imaging ranges of narrow-angle camera 101 and wide-angle camera 102, and performs stereo processing including calculating the parallax of the target object in the overlapping imaging range, thereby being able to detect target objects such as pedestrians and vehicles from the parallax calculated in the overlapping imaging range. Note that the number, installation positions, and installation directions of the cameras are not limited to the example in Figure 1, and any installation conditions may be used as long as at least a portion of the imaging ranges of a pair of cameras overlap.

[0014] As described above, the narrow-angle camera (telephoto camera) 101 captures a narrower angle of view than the wide-angle camera 102, and the image P1 captured by the narrow-angle camera 101 has a higher resolution than the image P2 captured by the wide-angle camera 102; in other words, the image P2 captured by the wide-angle camera 102 has a lower resolution than the image P1 captured by the narrow-angle camera 101.

[0015] 2 is a functional block diagram of a stereo camera device 100 as an image processing device according to this embodiment. The stereo camera device 100 of this embodiment includes two cameras (a narrow-angle camera 101 and a wide-angle camera 102), a computing device 1, a memory (not shown), and the like, and can calculate the parallax of a target object based on a pair of images captured simultaneously by the pair of cameras.

[0016] [Camera] The cameras (narrow-angle camera 101, wide-angle camera 102) are devices for capturing images and are primarily composed of an image sensor and a lens. The specifications of the image sensor and lens constituting each camera can be determined arbitrarily, but in this embodiment, as described above, the optical specifications (angle of view, focal length) of each camera are different. Then, a narrow-angle, high-resolution image P1 captured by the narrow-angle camera 101 and a wide-angle, low-resolution image P2 captured by the wide-angle camera 102 are each output to the image acquisition unit 10.

[0017] [Calculation Device] The calculation device 1 is an FPGA (Field Programmable Gate Array) or the like programmed to execute predetermined functions, and has an image acquisition unit 10, a magnification adjustment unit 12, a pre-processing unit 14, a resolution adjustment unit 16, and a stereo processing unit 18.

[0018] The image acquisition unit 10 receives (acquires) an image P1 from the narrow-angle camera 101 and outputs the image P1 to the magnification adjustment unit 12. The image acquisition unit 10 also receives (acquires) an image P2 from the wide-angle camera 102 and outputs the image P2 to the pre-processing unit 14.

[0019] Magnification adjustment unit 12 adjusts the magnification of image P1 obtained from narrow-angle camera 101 to correspond to image P2 obtained from wide-angle camera 102. That is, as shown in FIG. 3 , magnification adjustment unit 12 reduces image P1 from narrow-angle camera 101 at a determined magnification in order to match the size of the image from narrow-angle camera 101 to the size of the image from wide-angle camera 102. Magnification adjustment unit 12 reduces image P1 from narrow-angle camera 101 using, for example, a common pixel interpolation processing method (bilinear, bicubic, etc.). Here, the reduction magnification is determined by the focal length ratio (f2 / f1 [times]) of each camera when the same image sensor is used in narrow-angle camera 101 and wide-angle camera 102. The reduction ratio can be determined, for example, by using (1) the design value of the focal length ratio (f2 / f1), (2) the actual measured value of the focal length ratio (f2 / f1) measured on a factory line or the like and stored in memory, or (3) a table of temperature and focal length ratio (f2 / f1) (i.e., focal length ratio according to temperature) that stores the relationship between temperature and lens focal length ratio in a table format, since the lens focal length changes with temperature.

[0020] By adjusting the magnification to match the size of the images from the two cameras (narrow-angle camera 101 and wide-angle camera 102), it becomes possible to estimate parallax by stereo matching, which will be described later. Furthermore, by matching the size of the images from the two cameras (narrow-angle camera 101 and wide-angle camera 102) to the magnification of the wide-angle camera 102, high parallax performance can be obtained (described later).

[0021] Furthermore, in this embodiment, the magnification adjustment is performed before performing all of the pre-processing described below. By adjusting the magnification, the wide-angle camera 102 only needs to perform the pre-processing described below on the image size of the overlapping imaging range, and the narrow-angle camera 101 only needs to perform the pre-processing described below on the image size reduced to match the wide-angle camera 102 (see also FIG. 3 ). As a result, by adjusting the magnification before pre-processing, the image size after pre-processing is smaller than the full-size image size, thereby reducing the subsequent processing load.

[0022] The image P1 after the magnification adjustment is output to the preprocessing unit 14.

[0023] Preprocessing unit 14 has lens distortion correction units 141 and 142, color restoration units (also called demosaic units) 143 and 144, and RGB Y conversion units 145 and 146, and performs at least one of the preprocessing steps of lens distortion correction, color restoration, and color information brightness information conversion (RGB Y conversion) on images P1 and P2 input from narrow-angle camera 101 and wide-angle camera 102. Note that the processing order of lens distortion correction, color restoration, and RGB Y conversion is not limited to the example in FIG. 2 , and it is sufficient that at least one process can be performed as needed.

[0024] The lens distortion correction unit 141, color restoration unit (demosaic unit) 143, and RGB Y conversion unit 145 each perform preprocessing such as well-known lens distortion correction, color restoration, and color information brightness information conversion (RGB Y conversion) on the image P1 (after magnification adjustment) input from the narrow-angle camera 101.

[0025] The lens distortion correction unit 142, color restoration unit (demosaic unit) 144, and RGB Y conversion unit 146 each perform preprocessing such as well-known lens distortion correction, color restoration, and color information brightness information conversion (RGB Y conversion) on the image P2 input from the wide-angle camera 102 (without magnification adjustment).

[0026] The pre-processed images P1 and P2 are each output to the resolution adjustment unit 16.

[0027] After at least one of the above preprocessing steps has been performed, the resolution adjustment unit 16 adjusts the resolution of the images P1 and P2. That is, the resolution adjustment unit 16 adjusts the sharpness (blurring) of the image P1 obtained from the narrow-angle camera 101 and the image P2 obtained from the wide-angle camera 102. The resolution adjustment unit 16 adjusts the sharpness (blurring) of the images using, for example, common filter processing (Gaussian filter, unsharp mask). A Gaussian filter is a filter used to convert an image from high resolution to low resolution (blur the image). When using this Gaussian filter, the high-resolution image P1 is adjusted to the sharpness of the low-resolution image P2. An unsharp mask is a filter used to convert an image from low resolution to high resolution (sharpen the image). When using this unsharp mask, the low-resolution image P2 is adjusted to the sharpness of the high-resolution image P1. For example, in the case of a fixed-focus camera, individual differences in image sharpness occur due to focus adjustments performed on the factory line. Also, heat treatment changes the focus. Therefore, the camera's sharpness is measured in the final inspection process after camera assembly, and the filter coefficients are calculated and stored in memory. Also, for example, image sharpness changes with temperature. Therefore, the relationship between temperature and image sharpness is measured, and the temperature and filter coefficients are tabulated and stored in memory. In other words, a filter coefficient according to temperature is used. It is conceivable that which sharpness to match could be chosen depending on, for example, the driving scene.

[0028] By adjusting the resolution to match the sharpness (degree of blur) of the two camera images (P1, P2), it becomes possible to estimate the parallax by stereo matching, which will be described later.

[0029] Furthermore, the sharpness (blurring) of the image changes during preprocessing. Therefore, resolution adjustment is performed after preprocessing. By adjusting the resolution and aligning the sharpness (blurring) of the two camera images (P1, P2), the matching accuracy of the stereo matching described below is improved (explained later).

[0030] The images P1 and P2 after the resolution adjustment are output to the stereo processing unit 18.

[0031] The stereo processing unit 18 processes image information in the overlapping imaging range of the narrow-angle camera 101 and the wide-angle camera 102 using the image P1 and the image P2, performs stereo processing including calculating the parallax of the target object in the overlapping imaging range, and calculates a stereo parallax image and parallax information. The stereo processing unit 18 calculates the stereo parallax image and parallax information using a known stereo matching method such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference).

[0032] As described above, in this embodiment, the size of the images from the two cameras is matched by adjusting the magnification. Specifically, the size of the image from the narrow-angle camera 101 is matched to that of the wide-angle camera 102, and the sharpness (degree of blur) of the images from the two cameras is matched by adjusting the resolution. Furthermore, in this embodiment, the magnification is adjusted without thinning, taking into consideration all pixel data within the range of the image P1 from the narrow-angle camera 101 where the parallax calculation is performed. In other words, in addition to the magnification adjustment to match the size of the image from the narrow-angle camera 101 to that of the wide-angle camera 102, all pixel data used in the parallax calculation of the target object is also used for the magnification adjustment (no thinning). Therefore, the stereo processing unit 18 can achieve high parallax performance (different-angle stereo can be realized with high parallax performance), and for example, degradation of parallax performance at long distances can be suppressed.

[0033] [Effect of Magnification Adjustment] The effect of the magnification adjustment in this embodiment configured as described above will be specifically described with reference to Fig. 4. The inventors acquired images of the target object while changing the distance to the target object (target distance), and compared the performance of the acquired images when the image size was adjusted to the magnification of the narrow-angle camera and the wide-angle camera, respectively.

[0034] Figure 4 shows the relationship between the target distance and the ratio of the parallax variation when the magnification of the narrow-angle camera is adjusted to the parallax variation when the magnification of the wide-angle camera is adjusted to 1x (reference) (= parallax variation when the magnification of the narrow-angle camera is adjusted / parallax variation when the magnification of the wide-angle camera is adjusted). As shown in Figure 4, the results are greater than 1x for all target distances, and the parallax variation when the magnification of the narrow-angle camera is adjusted is greater than the parallax variation when the magnification of the wide-angle camera is adjusted. In other words, the parallax variation when the magnification of the wide-angle camera is adjusted is superior to the parallax variation when the magnification of the narrow-angle camera is adjusted.

[0035] The reason why the disparity variation is smaller when matching to the magnification of a wide-angle camera is that, for the same matching block size, the reference range and amount of information are wider and larger when matching to the magnification of a wide-angle camera, resulting in higher matching accuracy and smaller disparity variation.If the matching block size is increased to have the same reference range, the disparity variation will be the same, but the calculation load will increase significantly.

[0036] [Effect of Resolution Adjustment] Next, the effect of the resolution adjustment of this embodiment configured as described above will be specifically described with reference to Fig. 5. The inventors acquired images of the target object while changing the distance to the target object (target distance) on a test course, and compared the performance of the acquired images in three ways: no resolution adjustment, adjustment to high resolution (unsharp mask), and adjustment to low resolution (Gaussian filter).

[0037] Figure 5 shows the relationship between the target distance and the ratio of disparity variation with resolution matching (unsharp mask and Gaussian filter) when the disparity variation without resolution matching is set to 1x (reference). As shown in Figure 5, the results are smaller than 1x for all target distances, meaning that the disparity variation when the resolution is matched is smaller than when the resolution is not matched. In other words, the disparity variation was improved by matching the resolution. However, there was no significant difference between the unsharp mask and Gaussian filter resolution matching methods.

[0038] Effect of the Present Embodiment As described above, the stereo camera device (image processing device) 100 according to the present embodiment includes a narrow-angle imaging device (narrow-angle camera 101) having a narrow-angle imaging range and a wide-angle imaging device (wide-angle camera 102) having a wider-angle imaging range than the narrow-angle imaging device (narrow-angle camera 101). The narrow-angle imaging device and the wide-angle imaging device capture images of imaging ranges that at least partially overlap. The image processing device processes image information obtained from the overlapping imaging ranges and performs stereo processing including parallax calculation of target objects in the overlapping imaging ranges. In this image processing device, first image information (image P1) obtained from the narrow-angle imaging device is magnified (reduced) corresponding to second image information (image P2) obtained from the wide-angle imaging device, and the magnification adjustment is performed taking into consideration all pixel data of the first image information (image P1) within the range where the parallax calculation is performed.

[0039] As described above, by adjusting the magnification, the sizes of the images from the wide-angle imaging device and the narrow-angle imaging device become the same, enabling stereo matching. Furthermore, matching the size of the image from the narrow-angle imaging device to the size of the image from the wide-angle imaging device increases the target reference range within the matching block and the amount of information, resulting in higher matching accuracy and reduced in-plane variation in parallax.

[0040] In addition, the stereo camera device (image processing device) 100 according to this embodiment performs at least one of preprocessing steps of lens distortion correction, color restoration, and color information brightness information conversion (RGB Y conversion) on the first image information (image P1), and then adjusts the resolution of the first image information (image P1) and the second image information (image P2), and then performs the stereo processing.

[0041] Pre-processing changes the sharpness (blur) of the images. Therefore, as described above, resolution adjustment is performed after pre-processing. By performing this resolution adjustment and aligning the sharpness (blur) of the images from the two imaging devices, matching accuracy is improved.

[0042] Furthermore, in the stereo camera device (image processing device) 100 according to this embodiment, the magnification adjustment is performed before all of the pre-processing is performed.

[0043] By adjusting the magnification, the wide-angle imaging device only needs to perform preprocessing on the image size of the overlapping imaging range, and the narrow-angle imaging device only needs to perform preprocessing on the image size reduced to match the wide-angle imaging device. As a result, by adjusting the magnification before preprocessing as described above, the image size after preprocessing is smaller than the full-size image size, thereby reducing the subsequent processing load.

[0044] That is, in the stereo camera device (image processing device) 100 according to this embodiment, after a magnification adjustment (magnification adjustment for matching the size of the image of the narrow-angle imaging device to the size (magnification) of the image of the wide-angle imaging device) is performed on the first image information (image P1) corresponding to the second image information (image P2), at least one of preprocessing steps including lens distortion correction, color restoration, and color information brightness information conversion (RGB to Y conversion) is performed on the first image information (image P1) and the second image information (image P2), and then resolution adjustment is performed on the first image information (image P1) and the second image information (image P2), and the stereo processing is performed.

[0045] With the stereo camera device (image processing device) 100 according to this embodiment configured as described above, magnification adjustment is performed to match the size of the image from the narrow-angle camera 101 to that of the wide-angle camera 102, and resolution adjustment is performed to match the sharpness (degree of blur) of the images from the two cameras. The magnification adjustment is performed taking into consideration all pixel data within the range of image P1 from the narrow-angle camera 101 where parallax calculation is performed, thereby achieving different-angle-of-view stereo with high parallax performance in the overlapping imaging ranges of two imaging devices with different optical specifications (angle of view, focal length).

[0046] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0047] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0048] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0049] REFERENCE SIGNS LIST 1 Calculation device 10 Image acquisition unit 12 Magnification adjustment unit 14 Preprocessing unit 16 Resolution adjustment unit 18 Stereo processing unit 100 Stereo camera device (image processing device) 101 Narrow-angle camera (narrow-angle imaging device) 102 Wide-angle camera (wide-angle imaging device) P1 Image from narrow-angle camera P2 Image from wide-angle camera V Vehicle

Claims

1. An image processing device comprising a narrow-angle imaging device with a narrow-angle imaging range and a wide-angle imaging device with a wider-angle imaging range than the narrow-angle imaging device, wherein the narrow-angle imaging device and the wide-angle imaging device capture imaging ranges that at least partially overlap, process image information obtained from the overlapping imaging ranges, and perform stereo processing including parallax calculation of target objects in the overlapping imaging ranges, wherein first image information obtained from the narrow-angle imaging device is subjected to magnification adjustment corresponding to second image information obtained from the wide-angle imaging device, and the magnification adjustment is performed taking into account all pixel data of the first image information within the range where the parallax calculation is performed.

2. An image processing device according to claim 1, wherein the first image information is subjected to at least one of preprocessing steps of lens distortion correction, color restoration, and color information brightness information conversion, and then resolution adjustment is performed on the first image information and the second image information, and the stereo processing is performed.

3. An image processing device according to claim 2, wherein the magnification adjustment is performed before all of the pre-processing is performed.

4. An image processing device according to claim 1, wherein after the magnification adjustment of the first image information corresponding to the second image information is performed, at least one of lens distortion correction, color restoration, and preprocessing of color information to brightness information conversion is performed on the first image information and the second image information, and then resolution adjustment of the first image information and the second image information is performed, and the stereo processing is performed.

Images