Image processing device and image processing method

Abstract

To provide an image processing device which can enhance accuracy of a parallax image.SOLUTION: An image processing device according to an embodiment of the present disclosure comprises: a first distance calculation unit which calculates a first distance being a distance to an imaging object on the basis of a plurality of first images generated by a first camera included in a plurality of stereo images generated by sequentially performing an imaging operation by a stereo camera; a second distance calculation unit which calculates a second distance being a distance to the imaging object on the basis of one of the plurality of stereo images; a correction amount calculation unit which calculates a correction amount of a pixel position corresponding to the imaging object in the image generated by a second camera on the basis of the first distance and second distance when a difference in the pixel position corresponding to the imaging object in each of the plurality of first images is equal to or less than a prescribed amount; and an image correction unit which moves the position of a pixel value included in the image generated by the second camera in the lateral direction on the basis of the correction amount.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device for obtaining a parallax image based on a stereo image, and an image processing method used in such an image processing device. [Background technology]

[0002] Some vehicles, such as automobiles, detect the distance and direction to surrounding objects by performing stereo matching processing based on stereo images acquired by a stereo camera. This technology detects the distance to an object based on the parallax between a left image and a right image. For example, Patent Document 1 discloses a technology for calibrating parallax using an image captured with no windshield and an image captured with the windshield present. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-165968 Summary of the Invention [Problem to be solved by the invention]

[0004] In image processing devices, it is desirable that the accuracy of the parallax images generated by performing stereo matching processing is high, and further improvements in accuracy are expected.

[0005] It is desirable to provide an image processing device that can improve the accuracy of the parallax image. [Means for solving the problem]

[0006] An image processing device according to an embodiment of the present disclosure includes a first distance calculation unit, a second distance calculation unit, a correction amount calculation unit, and an image correction unit. The first distance calculation unit calculates a first distance, which is the distance to an imaging target, based on multiple first images generated by the first camera and included in multiple stereo images generated by a stereo camera having a first camera and a second camera sequentially performing imaging operations. The second distance calculation unit calculates a second distance, which is the distance to the imaging target, based on any one of the multiple stereo images. The correction amount calculation unit determines whether a difference between pixel positions corresponding to the imaging target in each of the multiple first images is within a predetermined amount, and if the difference between the pixel positions is within the predetermined amount, calculates a correction amount for the pixel position corresponding to the imaging target in the image generated by the second camera based on the first distance and the second distance. The image correction unit shifts the position of a pixel value included in the image generated by the second camera in the horizontal direction based on the correction amount.

[0007] An image processing method according to one embodiment of the present disclosure includes calculating a first distance, which is the distance to an imaging target, based on a plurality of first images generated by the first camera and included in a plurality of stereo images generated by a stereo camera having a first camera and a second camera sequentially performing imaging operations; calculating a second distance, which is the distance to the imaging target, based on any one of the plurality of stereo images; determining whether a difference in pixel position corresponding to the imaging target in each of the plurality of first images is within a predetermined amount; and if the difference in pixel position is within the predetermined amount, calculating a correction amount for the pixel position corresponding to the imaging target in the image generated by the second camera based on the first distance and the second distance; and moving the position of the pixel value included in the image generated by the second camera laterally based on the correction amount. [Effects of the Invention]

[0008] According to the image processing device and image processing method according to an embodiment of the present disclosure, the accuracy of the parallax image can be improved. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of the arrangement of the stereo cameras shown in FIG. [Figure 3] 2 is an explanatory diagram illustrating the operation principle of the correction amount calculation unit shown in FIG. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of an image region R. [Figure 5] 2 is a flowchart illustrating an example of an operation of the processing unit illustrated in FIG. 1. [Figure 6] FIG. 10 is a block diagram illustrating an example of the configuration of an image processing device according to a modified example. [Figure 7] 7 is a flowchart illustrating an example of an operation of the processing unit illustrated in FIG. 6. [Figure 8] FIG. 10 is a block diagram illustrating an example of the configuration of an image processing device according to another modified example. [Figure 9] 9 is a flowchart illustrating an example of an operation of the processing unit illustrated in FIG. 8. [Figure 10] FIG. 10 is a block diagram illustrating an example of the configuration of an image processing device according to another modified example. [Figure 11] 11 is a flowchart illustrating an example of the operation of the processing unit illustrated in FIG. [Figure 12] FIG. 10 is a block diagram illustrating an example of the configuration of an image processing device according to another modified example. [Figure 13] 13 is a flowchart illustrating an example of an operation of the processing unit illustrated in FIG. 12. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0011] <Embodiment> [Configuration example] 1 shows an example of the configuration of an image processing device (image processing device 1) according to an embodiment. The image processing device 1 includes a stereo camera 11 and a processing unit 20. In this example, the image processing device 1 is mounted on a vehicle 10 such as an automobile.

[0012] The stereo camera 11 is configured to capture an image of the area ahead of the vehicle 10, thereby generating a pair of images (a left image PL and a right image PR) having a parallax therebetween. The stereo camera 11 includes a left camera 11L and a right camera 11R. Each of the left camera 11L and the right camera 11R includes a lens and an image sensor.

[0013] FIG. 2 shows an example of the arrangement of the stereo camera 11 in a vehicle 10. In this example, the left camera 11L and the right camera 11R are arranged inside the vehicle 10 near the top of the windshield 9 of the vehicle 10, spaced a predetermined distance apart in the width direction of the vehicle 10. The left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL and the right image PR form a stereo image PIC. The stereo camera 11 performs an imaging operation at a predetermined frame rate (e.g., 60 fps) to generate a series of stereo images PIC and supply the generated stereo images PIC to the processing unit 20.

[0014] The processing unit 20 (FIG. 1) is configured to recognize an object ahead of the vehicle 10 based on the stereo image PIC supplied from the stereo camera 11. The vehicle 10, for example, can perform driving control of the vehicle 10 based on information about the object recognized by the processing unit 20, or display information about the recognized object on a console monitor. The processing unit 20 is configured, for example, with a CPU (Central Processing Unit) that executes a program, a RAM (Random Access Memory) that temporarily stores processed data, and a ROM (Read Only Memory) that stores the program. The processing unit 20 has an image correction unit 21, a parallax image generation unit 22, a SLAM (Simultaneous Localization and Mapping) processing unit 24, a correction amount calculation unit 30, a control unit 25, and an object recognition unit 26.

[0015] The image correction unit 21 is configured to generate a left image PL1 by performing image correction processing on the left image PL using the correction map data MAP. The image correction unit 21 performs image correction processing according to image distortion caused by the windshield 9. That is, although it is desirable that the optical characteristics of the glass portion of the windshield 9 in front of the imaging plane of the left camera 11L and the glass portion in front of the imaging plane of the right camera 11R are uniform, in reality, they may not be uniform. In this case, for example, the left image PL and the right image PR may be distorted. In particular, if the image distortions in the horizontal direction differ between the left image PL and the right image PR, the parallax obtained by the stereo matching processing is affected by this difference in image distortion, resulting in a decrease in the accuracy of the parallax image PD. Therefore, the image processing device 1 shifts pixel values ​​of the left image PL in the horizontal direction by the amount of this difference in image distortion to reduce the effect of this difference in image distortion on the parallax image PD.

[0016] The correction map data MAP is map data of correction amounts for correcting horizontal image distortion. The image correction unit 21 generates the left image PL1 by, for example, shifting the positions of pixel values ​​in the left image PL in the horizontal direction by the correction amounts corresponding to the positions of those pixel values ​​in the correction map data MAP.

[0017] The parallax image generator 22 is configured to generate a parallax image PD by performing predetermined image processing, including stereo matching, on the left image PL1 and the right image PR. The parallax image PD has a plurality of pixel values. Each of the pixel values ​​indicates a value for the parallax at the corresponding pixel. In other words, each of the pixel values ​​corresponds to a distance to a point corresponding to the corresponding pixel in three-dimensional real space.

[0018] The parallax image generation unit 22 includes a corresponding point detection unit 23. The corresponding point detection unit 23 is configured to detect corresponding points, including image points in the left image PL1 and image points in the right image PR, that correspond to each other by performing stereo matching based on the left image PL1 and the right image PR. The corresponding point detection unit 23 may detect corresponding points, for example, by template matching or feature matching based on local features. The corresponding point detection unit 23 then calculates the difference between the horizontal position of an image point in the left image PL1 and the horizontal position of an image point in the right image PR as the parallax DISP2 of the corresponding point. The unit of the parallax DISP2 is, for example, pixels. The corresponding point detection unit 23 supplies the parallax DISP2 of each of the detected corresponding points to the correction amount calculation unit 30, along with data on the pixel positions of the corresponding points (pixel position data). The parallax image generation unit 22 generates a parallax image PD based on the detection result of the corresponding point detection unit 23.

[0019] The SLAM processing unit 24 is configured to calculate the distance to a stationary object included in a right image PR by performing SLAM processing based on multiple right images PR captured at different times. The multiple right images PR may be, for example, two consecutive right images PR or three consecutive right images PR. The SLAM processing unit 24 then supplies data on the distance (distance data) and data on the pixel position of the object in each of the multiple right images PR (pixel position data) to the correction amount calculation unit 30.

[0020] The correction amount calculation unit 30 is configured to generate correction map data MAP based on the distance data and pixel position data supplied from the SLAM processing unit 24 and the parallax DISP2 and pixel position data supplied from the corresponding point detection unit 23.

[0021] 3 shows the operating principle of the operation of generating the correction amounts in the correction map data MAP by the correction amount calculation unit 30. In this example, the SLAM process is performed using two consecutive right images PR.

[0022] In this example, at timing t1, the left camera 11L and the right camera 11R capture an image of a stationary object. Then, a stereo matching process is performed based on the left image PL generated by the left camera 11L and the right image PR generated by the right camera 11R, and the distance to the object is calculated. In the left image PL, the lateral position of the object is xl1, and in the right image PR, the lateral position of the object is xr1. Therefore, the position POS1 of the object in three-dimensional space is calculated based on these positions xl1 and xr1. If the optical characteristics of the glass portion in front of the imaging plane of the left camera 11L and the glass portion in front of the imaging plane of the right camera 11R on the windshield 9 are not uniform, the left image PL and the right image PR will be distorted. Therefore, the position POS1 of the object obtained in this manner may differ from the actual position of the object. Therefore, the distance from the stereo camera 11 to the position POS1 may also differ from the actual distance.

[0023] Next, for example, at timing t2, which is the next imaging timing after timing t1, the right camera 11R captures an image of the object. In this example, SLAM processing is performed based on the two right images PR generated at timings t1 and t2, and the distance to the object is calculated. In the right image PR at timing t1, the lateral position of the object is xr1, and in the right image PR at timing t2, the lateral position of the object is xr2. Based on these positions xr1 and xr2, the position POS2 of the object in three-dimensional space is calculated. If the optical characteristics of the glass portion in front of the imaging surface of the right camera 11R are not uniform, the right image PR will be distorted. Therefore, the position POS2 of the object obtained in this manner may differ from the actual position.

[0024] In this example, position xr1 at time t1 and position xr2 at time t2 are approximately the same. In this case, the distortion of the image of the object in the right image PR obtained at time t1 is approximately the same as the distortion of the image of the object in the right image PR obtained at time t2. Therefore, the distance from the stereo camera 11 to this position POS2 is not affected by image distortion due to the windshield 9 and is approximately the same as the actual distance.

[0025] In this way, when the position xr1 at time t1 and the position xr2 at time t2 are substantially the same, the distance obtained by the SLAM processing is substantially the same as the actual distance. In such a case, the correction amount calculation unit 30 performs calibration using the distance obtained by the SLAM processing. Specifically, the correction amount calculation unit 30 adjusts the amount of horizontal shift of the left image PL so that the distance obtained by the stereo matching processing is the same as the distance obtained by the SLAM processing. In this way, the correction amount calculation unit 30 calculates the correction amount in the correction map data MAP.

[0026] The correction amount calculation unit 30 (FIG. 1) includes a determination unit 31, a parallax value storage unit 32, a parameter calculation unit 33, a correction coefficient calculation unit , and a correction map storage unit .

[0027] The determination unit 31 is configured to determine, based on the distance data and pixel position data supplied from the SLAM processing unit 24, whether the distance data is to be used to calculate the correction amount. Specifically, the determination unit 31 determines, based on the pixel position data supplied from the SLAM processing unit 24, whether pixel positions of objects in multiple right images PR are close to each other. For example, in the example of FIG. 3 , the pixel position data includes a position xr1 at timing t1 and a position xr2 at timing t2. The determination unit 31 determines whether the difference between the position xr1 at timing t1 and the position xr2 at timing t2 is within a predetermined amount. If the difference between the position xr1 at timing t1 and the position xr2 at timing t2 is within the predetermined amount, the determination unit 31 determines that the distance data supplied from the SLAM processing unit 24 is to be used to calculate the correction amount, and supplies the distance data and pixel position data supplied from the SLAM processing unit 24 to the disparity value storage unit 32.

[0028] The disparity value storage unit 32 is configured to convert the distance to the object indicated by the distance data and pixel position data supplied from the determination unit 31 into a disparity DISP1, and store this disparity DISP1 in association with the pixel position indicated by the pixel position data. The unit of disparity DISP1 is, for example, pixels. The disparity value storage unit 32 is also configured to store the disparity DISP2 related to the object supplied from the corresponding point detection unit 23 in association with the pixel position of the corresponding point. In this example, as shown in FIG. 4, the entire image area of ​​the right image PR is divided into multiple (10 in this example) image areas R. The disparity value storage unit 32 stores the disparity DISP1 in association with the image area R to which the pixel position indicated by the pixel position data belongs, and stores the disparity DISP2 in association with the image area R to which the pixel position of the corresponding point belongs. Note that, although 10 image areas R are used in this example, the number of image areas R is not limited to this, and more image areas R may be used.

[0029] The parameter calculation unit 33 is configured to calculate parameters α and β indicating the relationship between the parallaxes DISP1 and DISP2 in each of the multiple image regions R. Specifically, the parameter calculation unit 33 calculates the parameters α and β in each of the multiple image regions R using the following equation so that the value err is minimized. err = Σ( DISP1 - DISP2×α+β ) α indicates the gain of parallax, and β indicates the displacement of parallax. When the image correction unit 21 performs appropriate image correction, the parameter α is expected to be “1” and the parameter β is expected to be “0.”

[0030] The correction coefficient calculation unit 34 is configured to calculate correction coefficients A and B based on the parameters α and β for each of the multiple image regions R. For example, if the parameter α is greater than 1, the correction coefficient calculation unit 34 increases the correction coefficient A, and if the parameter α is less than 1, the correction coefficient calculation unit 34 decreases the correction coefficient A. Furthermore, if the parameter β is greater than 0, the correction coefficient calculation unit 34 increases the correction coefficient B, and if the parameter β is less than 0, the correction coefficient B decreases. In an ideal case where there is no image distortion due to the windshield 9, the correction coefficient A is expected to be "1" and the correction coefficient B is expected to be "0."

[0031] The correction map storage unit 35 is configured to update the correction amount in each of the plurality of image regions R in the correction map data MAP based on the correction coefficients A and B. For example, the correction map storage unit 35 can use the correction coefficient B in each of the plurality of image regions R as the correction amount in that image region R. Furthermore, the correction map storage unit 35 may also use the correction coefficient A, for example, to calculate the correction amount in that image region R.

[0032] The control unit 25 is configured to control the operations of the SLAM processing unit 24 and the correction amount calculation unit 30.

[0033] The object recognition unit 26 is configured to recognize an object ahead of the vehicle 10 based on the left image PL1, the right image PR, and the parallax image PD generated by the parallax image generation unit 22. The object recognition unit 26 then outputs data on the recognition result.

[0034] Here, the stereo camera 11 corresponds to a specific example of a "stereo camera" in the present disclosure. The right camera 11R corresponds to a specific example of a "first camera" in the present disclosure. The left camera 11L corresponds to a specific example of a "second camera" in the present disclosure. The SLAM processing unit 24 corresponds to a specific example of a "first distance calculation unit" in the present disclosure. The disparity DISP1 corresponds to a specific example of a "first distance" in the present disclosure. The corresponding point detection unit 23 corresponds to a specific example of a "second distance calculation unit" in the present disclosure. The disparity DISP2 corresponds to a specific example of a "second distance" in the present disclosure. The correction amount calculation unit 30 corresponds to a specific example of a "correction amount calculation unit" in the present disclosure. The image correction unit 21 corresponds to a specific example of an "image correction unit" in the present disclosure.

[0035] [Actions and Actions] Next, the operation and function of the image processing device 1 of this embodiment will be described.

[0036] (Overview of overall operation) First, an overview of the overall operation of the image processing device 1 will be described with reference to FIG. 1. The stereo camera 11 captures images of the area ahead of the vehicle 10 to generate a stereo image PIC including a left image PL and a right image PR. In the processing unit 20, the image correction unit 21 performs image correction processing on the left image PL using correction map data MAP to generate a left image PL1. The parallax image generation unit 22 performs predetermined image processing, including stereo matching processing, based on the left image PL1 and the right image PR to generate a parallax image PD. The corresponding point detection unit 23 of the parallax image generation unit 22 detects corresponding points including image points in the left image PL1 and image points in the right image PR, and calculates the difference between the lateral positions of the image points in the left image PL1 and the right image PR as the parallax DISP2 of the corresponding points. The corresponding point detection unit 23 then supplies the parallax DISP2 of each of the detected corresponding points to the correction amount calculation unit 30, along with data on the pixel positions of the corresponding points (pixel position data). The SLAM processing unit 24 calculates the distance to a stationary object included in the right image PR by performing SLAM processing based on multiple right images PR captured at different times. The SLAM processing unit 24 then supplies data on the distance (distance data) and data on the pixel position of the object in each of the multiple right images PR (pixel position data) to the correction amount calculation unit 30. The correction amount calculation unit 30 generates correction map data MAP based on the distance data and pixel position data supplied from the SLAM processing unit 24 and the parallax DISP2 and pixel position data supplied from the corresponding point detection unit 23. The control unit 25 controls the operations of the SLAM processing unit 24 and the correction amount calculation unit 30. The object recognition unit 26 recognizes objects ahead of the vehicle 10 based on the left image PL1, the right image PR, and the parallax image PD generated by the parallax image generation unit 22.

[0037] (Detailed operation) 5 shows an example of the operation of the SLAM processing unit 24 and the correction amount calculation unit 30. The SLAM processing unit 24 and the correction amount calculation unit 30 perform the following processing each time a stereo image PIC is supplied from the stereo camera 11 to the processing unit 20.

[0038] First, the SLAM processing unit 24 calculates the distance to a stationary object included in the right image PR by performing SLAM processing based on multiple right images PR captured at different times (step S101).The SLAM processing unit 24 then supplies data on this distance (distance data) and data on the pixel position of this object in each of the multiple right images PR (pixel position data) to the correction amount calculation unit 30.

[0039] Next, the determination unit 31 of the correction amount calculation unit 30 determines whether the pixel positions of the objects in the multiple right images PR are close to each other based on the pixel position data supplied from the SLAM processing unit 24 (step S102). For example, in the example of FIG. 3, the pixel position data includes a position xr1 at timing t1 and a position xr2 at timing t2. The determination unit 31 determines whether the pixel positions of the objects are close to each other by determining whether the difference between the position xr1 at timing t1 and the position xr2 at timing t2 is within a predetermined amount. If the pixel positions are not close to each other ("N" in step S103), this flow ends.

[0040] In step S103, if the pixel positions are close ("Y" in step S103), the disparity value storage unit 32 converts the distance to the object indicated by the distance data supplied from the SLAM processing unit 24 into a disparity DISP1, and stores this disparity DISP1 in association with the pixel position indicated by the pixel position data supplied from the SLAM processing unit 24 (step S104). For example, the disparity value storage unit 32 stores the disparity DISP1 in association with the image region R (FIG. 4) to which the pixel position indicated by the pixel position data belongs.

[0041] The disparity value storage unit 32 also stores the disparity DISP2 of this object obtained by the corresponding point detection unit 23 based on the stereo image PIC in association with the pixel positions of the corresponding points (step S105). For example, the disparity value storage unit 32 stores the disparity DISP2 in association with the image region R (FIG. 4) to which the pixel positions of the corresponding points belong.

[0042] Next, the parameter calculation unit 33 calculates the parameters α and β based on one or more disparities DISP1 and one or more disparities DISP2 associated with the image region R in which the disparities DISP1 and DISP2 were stored in steps S104 and S105 (step S106). Specifically, the parameter calculation unit 33 calculates the parameters α and β using the following equations so that the value err is minimized in this image region R. err = Σ( DISP1 - DISP2×α+β )

[0043] Next, the correction coefficient calculation unit 34 calculates correction coefficients A and B based on the parameters α and β for the image region R in which the parallaxes DISP1 and DISP2 were stored in steps S104 and S105, and the correction map storage unit 35 updates the correction amount for that image region R in the correction map data MAP based on the correction coefficients A and B (step S107). For example, the correction coefficient calculation unit 34 increases the correction coefficient A when the parameter α is greater than 1, and decreases the correction coefficient A when the parameter α is less than 1. Furthermore, the correction coefficient calculation unit 34 increases the correction coefficient B when the parameter β is greater than 0, and decreases the correction coefficient B when the parameter β is less than 0. For example, the correction map storage unit 35 can use the correction coefficient B for each of the multiple image regions R as the correction amount for that image region R. Furthermore, the correction map storage unit 35 may also use the correction coefficient A to calculate the correction amount for that image region R, for example.

[0044] Then, the correction map storage unit 35 checks whether the correction map data MAP has converged (step S108). If the correction map data MAP has not yet converged ("N" in step S108), this flow ends.

[0045] In step S108, if the correction map data MAP has converged ("Y" in step S108), the correction map storage unit 35 supplies this correction map data MAP to the image correction unit 21 (step S109). As a result, from this point on, the image correction unit 21 performs image correction processing based on this correction map data MAP.

[0046] This is the end of this flow.

[0047] As described above, the image processing device 1 includes a SLAM processing unit 24 that calculates a first distance to the target based on a plurality of right images PR generated by the right camera 11R and included in a plurality of stereo images PIC generated by the stereo camera 11 having the right camera 11R and the left camera 11L sequentially capturing images. A corresponding point detection unit 23 that calculates a second distance to the target based on any one of the plurality of stereo images PIC. A correction amount calculation unit 30 determines whether the difference in pixel position corresponding to the target in each of the plurality of right images PR is within a predetermined amount. If the difference in pixel position is within the predetermined amount, the correction amount calculation unit 30 calculates a correction amount for the pixel position corresponding to the target in the left image PL generated by the left camera 11L based on the first and second distances. An image correction unit 21 laterally shifts the position of the pixel value included in the image generated by the left camera 11L based on the correction amount. This allows the image processing device 1 to improve the accuracy of the parallax image PD even when there is image distortion due to the windshield 9.

[0048] That is, although it is desirable that the optical characteristics be uniform in the glass portion of the windshield 9 in front of the imaging plane of the left camera 11L and the glass portion in front of the imaging plane of the right camera 11R, in reality, they may not be uniform. In this case, for example, the left image PL and the right image PR may be distorted. In particular, if the image distortion in the horizontal direction differs between the left image PL and the right image PR, the parallax obtained by the stereo matching process is affected by the difference in image distortion, and the accuracy of the parallax image PD decreases.

[0049] In the image processing device 1, the SLAM processing unit 24 calculates a first distance, which is the distance to the imaging target, based on multiple right images PR. Then, if the difference between pixel positions corresponding to the imaging target in each of the multiple right images PR is within a predetermined amount, the correction amount calculation unit 30 calculates a correction amount for the pixel position corresponding to the imaging target in the left image PL generated by the left camera 11L based on the first distance and the second distance obtained based on the stereo image PIC. Then, the image correction unit 21 shifts the position of the pixel value included in the image generated by the left camera 11L in the horizontal direction based on the correction amount. This allows the image processing device 1 to generate a correction amount according to the coordinates even when non-uniform image distortion occurs due to the windshield 9, thereby improving the accuracy of the parallax image PD.

[0050] Another method for improving the accuracy of the parallax images PD is to provide a radar device that detects the distance and direction to surrounding objects and calibrate the parallax images PD based on the detection results of the radar device. However, this method increases costs and complicates the system. The image processing device 1 can reduce the influence of image distortion caused by the windshield 9 without providing a separate device, thereby reducing costs and improving the accuracy of the parallax images PD in a simple manner.

[0051] [effect] As described above, this embodiment includes a SLAM processing unit that calculates a first distance to the target based on multiple right images generated by the right camera, the multiple right images being included in multiple stereo images generated by a stereo camera having a right camera and a left camera sequentially capturing images; a corresponding point detection unit that calculates a second distance to the target based on one of the multiple stereo images; a correction amount calculation unit that determines whether a difference in pixel position corresponding to the target in each of the multiple right images is within a predetermined amount and, if the difference in pixel position is within the predetermined amount, calculates a correction amount for the pixel position corresponding to the target in the image generated by the second camera based on the first distance and the second distance; and an image correction unit that shifts the position of a pixel value included in the left image generated by the left camera horizontally based on the correction amount. This allows for improved accuracy of the parallax image even when there is image distortion due to a windshield.

[0052] [Variation 1] In the above embodiment, the control unit 25 controls the operations of the SLAM processing unit 24 and the correction amount calculation unit 30. However, for example, the control unit 25 may control the operations of the SLAM processing unit 24 and the correction amount calculation unit 30 based on the traveling speed or yaw rate of the vehicle 10. The image processing device 1A according to this modification will be described in detail below.

[0053] 6 shows an example configuration of an image processing device 1A. The image processing device 1A includes a processing unit 20A. The processing unit 20A includes a control unit 25A. The control unit 25A is configured to control the operations of the SLAM processing unit 24 and the correction amount calculation unit 30 based on driving information INF of the vehicle 10 supplied from an ECU (not shown) provided in the vehicle 10. This driving information INF includes, for example, information about the driving speed of the vehicle 10 and information about the yaw rate.

[0054] 7 shows an example of the operation of the SLAM processing unit 24 and the correction amount calculation unit 30 according to this modification. First, the control unit 25A checks whether the value of the traveling speed of the vehicle 10 is within a predetermined range and whether the value of the yaw rate of the vehicle 10 is within a predetermined range, based on the traveling information INF of the vehicle 10 supplied from the ECU (step S121). If the value of the traveling speed of the vehicle 10 is within the predetermined range and the value of the yaw rate of the vehicle 10 is within the predetermined range ("Y" in step S121), the process proceeds to step S101. If these conditions are not met ("N" in step S121), the flow ends.

[0055] That is, for example, when the vehicle 10 is almost stopped or when the traveling speed of the vehicle 10 is very high, it is difficult for the SLAM processing unit 24 to perform SLAM processing with high accuracy. Also, for example, when the yaw rate of the vehicle 10 is high, it is difficult for the SLAM processing unit 24 to perform SLAM processing with high accuracy. Therefore, the control unit 25A controls the SLAM processing unit 24 and the correction amount calculation unit 30 to perform the processes of steps S101 to S109 when, for example, the traveling speed and yaw rate of the vehicle 10 are values ​​that allow the SLAM processing unit 24 to perform SLAM processing with high accuracy, and not to perform the processes in other cases.

[0056] The processing from step S102 onwards is the same as in the above embodiment (FIG. 5).

[0057] As a result, in the image processing device 1A, the SLAM processing unit 24 can perform the SLAM processing with high accuracy, and therefore the accuracy of the parallax image PD can be improved.

[0058] [Variation 2] In the above embodiment, the SLAM processing unit 24 calculates the distance to a stationary object included in the right image PR by performing SLAM processing based on multiple right images PR. In this case, the SLAM processing unit may estimate the pixel position of the object in each of the multiple right images PR based on, for example, the traveling speed and yaw rate of the vehicle 10. The image processing device 1B according to this modification will be described in detail below.

[0059] 8 shows an example of the configuration of an image processing device 1B. The image processing device 1B includes a processing unit 20B. The processing unit 20B includes a control unit 25B and a SLAM processing unit 24B.

[0060] The control unit 25B is configured to control the operations of the SLAM processing unit 24B and the correction amount calculation unit 30. The control unit 25B receives driving information INF of the vehicle 10 from an ECU (not shown) provided in the vehicle 10. This driving information INF includes, for example, information about the driving speed and yaw rate of the vehicle 10. The control unit 25B supplies the information about the driving speed and yaw rate to the SLAM processing unit 24B.

[0061] Similar to the SLAM processing unit 24 according to the above embodiment, the SLAM processing unit 24B performs SLAM processing based on multiple right images PR to calculate the distance to a stationary object included in the right image PR. The SLAM processing unit 24B also has a function of estimating the pixel position of an object in the right image PR based on the traveling speed and yaw rate of the vehicle 10. Specifically, the SLAM processing unit 24B estimates the pixel position of an object in the next right image PR based on, for example, the pixel position of the object in a previously obtained right image PR and the traveling speed and yaw rate of the vehicle 10. The SLAM processing unit 24B then determines whether the pixel position of the object obtained by the SLAM processing is close to the estimated pixel position.

[0062] FIG. 9 shows an example of the operation of the SLAM processing unit 24B and the correction amount calculation unit 30 according to this modification.

[0063] First, the SLAM processing unit 24B estimates the pixel position of an object in each of the right images PR based on the traveling speed and yaw rate of the vehicle 10 (step S131).

[0064] Next, the SLAM processing unit 24B performs SLAM processing based on the multiple right images PR, as in the above embodiment, to calculate the distance to a stationary object included in the right images PR (step S101). In this SLAM processing, the SLAM processing unit 24B calculates the pixel position of the object in the right images PR.

[0065] Next, the SLAM processing unit 24B checks whether the pixel position of the object in the right image PR obtained in step S101 is close to the pixel position of the object estimated in step S131 (step S132). Specifically, the SLAM processing unit 24B determines whether the distance between the pixel position of the object in the right image PR obtained in step S101 and the pixel position of the object estimated in step S131 is within a predetermined amount, thereby determining whether the pixel position of the object obtained in step S101 is close to the pixel position of the object estimated in step S131. If this condition is met, the process proceeds to step S102; if this condition is not met, the flow ends. In other words, if this condition is not met, there is a possibility that the pixel position of the object was erroneously detected in step S101, and therefore the SLAM processing unit 24B discards the processing result of step S101.

[0066] The processing from step S102 onwards is the same as in the above embodiment (FIG. 5).

[0067] As a result, the image processing device 1B calculates the correction amount without using the processing result in step S101 when the pixel position of the object is erroneously detected, and therefore the accuracy of the parallax image PD can be improved.

[0068] [Variation 3] In the above embodiment, the correction amount calculation unit 30 supplies the correction map data MAP to the image correction unit 21. However, this is not limited to this, and instead, for example, a function of the correction amount may be supplied. An image processing device 1C according to this modification will be described in detail below.

[0069] 10 shows an example of the configuration of an image processing device 1C. The image processing device 1C includes a processing unit 20C. The processing unit 20C includes an image correction unit 21C and a correction amount calculation unit 30C.

[0070] The image correction unit 21C is configured to generate a left image PL1 by performing image correction processing on the left image PL using a correction function FUNC. The correction function FUNC receives coordinate values ​​of pixel positions in the left image PL and outputs a correction amount. The image correction unit 21C, for example, calculates the correction amount for all pixel positions in the left image PL based on this correction function FUNC. Then, the image correction unit 21C generates the left image PL1 by shifting the positions of pixel values ​​in the left image PL in the horizontal direction by the calculated correction amount.

[0071] The correction amount calculation unit 30C has a correction function generation unit 36C. The correction function generation unit 36C is configured to generate a correction function FUNC based on the correction map data MAP supplied from the correction map storage unit 35.

[0072] 11 shows an example of the operation of the SLAM processing unit 24 and the correction amount calculation unit 30C according to this modification. The processes in steps S101 to S108 are the same as those in the above embodiment (FIG. 5).

[0073] In step S108, if the correction map data MAP has converged ("Y" in step S108), the correction function generation unit 36C generates the correction function FUNC based on the correction map data MAP supplied from the correction map storage unit 35 (step S141).

[0074] Then, the correction function generating unit 36C supplies this correction function FUNC to the image correcting unit 21C (step S142), whereby the image correcting unit 21C subsequently performs image correction processing based on this correction function FUNC.

[0075] This is the end of this flow.

[0076] As a result, the image processing device 1C can calculate the correction amount using the correction function FUNC, thereby improving the accuracy of the correction amount. That is, since the correction map data MAP includes, for example, 10 correction amounts for 10 image regions R, the correction amount may change suddenly and become discontinuous near the boundary of the image region R. In this modification, the correction function FUNC is used, thereby improving the continuity of the correction amount. Therefore, the image processing device 1C can improve the image quality of the left image PL1 generated by the image correction process, and can improve the accuracy of the parallax image PD.

[0077] [Variation 4] In the above embodiment, the disparity DISP2 is calculated based on the right image PR and the left image PL1 generated by the image correction unit 21, but this is not limiting and the disparity DISP2 may be calculated based on the right image PR and the left image PL. An image processing device 1D according to this modification will be described in detail below.

[0078] 12 shows an example of the configuration of an image processing device 1D. The image processing device 1D includes a processing unit 20D. The processing unit 20D includes a corresponding point detection unit 29D, a parallax image generation unit 22D, and a correction amount calculation unit 30D.

[0079] The corresponding point detection unit 29D is configured to detect corresponding points including image points in the left image PL and right image PR that correspond to each other by performing stereo matching processing based on the left image PL and the right image PR. The corresponding point detection unit 29D then calculates the difference between the horizontal position of the image point in the left image PL and the horizontal position of the image point in the right image PR as the disparity DISP2 of the corresponding point. The corresponding point detection unit 29D is configured to supply the disparity DISP2 of each of the detected corresponding points to the correction amount calculation unit 30 together with data on the pixel positions of the corresponding points (pixel position data).

[0080] Similar to the parallax image generation unit 22 according to the above embodiment, the parallax image generation unit 22D is configured to generate parallax images PD by performing predetermined image processing, including stereo matching, based on the left image PL1 and the right image PR. The parallax image generation unit 22D includes a corresponding point detection unit 23D. The corresponding point detection unit 23D is configured to detect corresponding points, including image points in the left image PL1 and image points in the right image PR, that correspond to each other by performing stereo matching based on the left image PL1 and the right image PR.

[0081] The correction amount calculation unit 30D has a determination unit 31, a disparity value storage unit 32, a parameter calculation unit 33, and a correction map storage unit 35D. The correction map storage unit 35D is configured to update the correction amount of the correction map data MAP for each of the multiple image regions R based on the parameters α and β. The correction map storage unit 35D can use, for example, the parameter β for each of the multiple image regions R as the correction amount for that image region R. Furthermore, the correction map storage unit 35D may also use, for example, the parameter α to calculate the correction amount for that image region R.

[0082] 13 shows an example of the operation of the SLAM processing unit 24 and the correction amount calculation unit 30D according to this modification. The processes in steps S101 to S104, S106, S108, and S109 are the same as those in the above embodiment (FIG. 5).

[0083] In step S104, the disparity value storage unit 32 converts the distance to the object indicated by the distance data supplied from the SLAM processing unit 24 into a disparity DISP1, and stores this disparity DISP1 in association with the pixel position indicated by the pixel position data.

[0084] Then, the parallax value storage unit 32 stores the parallax DISP2 of this object obtained by the corresponding point detection unit 29D based on the stereo image PIC in association with the pixel position of the corresponding point (step S155).

[0085] Next, in step S106, the parameter calculation unit 33 calculates parameters α and β based on one or more parallaxes DISP1 and one or more parallaxes DISP2 associated with the image region R in which the parallaxes DISP1 and DISP2 were stored in steps S104 and S155.

[0086] Next, the correction map storage unit 35D updates the correction amount for that image region R in the correction map data MAP based on the parameters α and β (step S157). For example, the correction map storage unit 35D can use the parameter β for each of the multiple image regions R as the correction amount for that image region R. Furthermore, the correction map storage unit 35D may also calculate the correction amount for that image region R by further using the parameter α, for example.

[0087] [Variation 5] In the present embodiment, the SLAM processing unit 24 performs SLAM processing based on the right image PR, and the image correction unit 21 performs image correction processing on the left image PL to generate the left image PL1, but this is not limited to this. Alternatively, for example, the SLAM processing unit 24 may perform SLAM processing based on the left image PL, and the image correction unit 21 may perform image correction processing on the right image PR to generate the right image PR1.

[0088] [Other variations] Two or more of these variations may also be combined.

[0089] The present technology has been described above by giving embodiments and some modified examples, but the present technology is not limited to these embodiments and can be modified in various ways.

[0090] For example, in the above embodiment, the stereo camera 11 is configured to capture an image in front of the vehicle 10, but this is not limitative. For example, the stereo camera 11 may capture an image of the side or rear of the vehicle 10.

[0091] The effects described in this specification are merely examples and are not limiting, and other effects may also be present. [Explanation of symbols]

[0092] 1, 1A, 1B, 1C, 1D... image processing device, 11... stereo camera, 11L... left camera, 11R... right camera, 20, 20A, 20C, 20D... processing unit, 21, 21C... image correction unit, 22, 22D... parallax image generation unit, 23, 23D... corresponding point detection unit, 24, 24B... SLAM processing unit, 25, 25A, 25B... control unit, 26... object recognition unit, 29D... corresponding point detection unit, 30, 30C, 30D... correction Quantity calculation unit, 31...determination unit, 32...parallax value storage unit, 33...parameter calculation unit, 34...correction coefficient calculation unit, 35, 35D...correction map storage unit, 36C...correction function generation unit, A, B...correction coefficient, DISP1, DISP2...parallax, FUNC...correction function, INF...driving information, MAP...correction map data, PIC...stereo image, PL, PL1...left image, PR...right image, R...image area, α, β...parameters.

Claims

1. a first distance calculation unit that calculates a first distance, which is a distance to an imaging target, based on a plurality of first images generated by the first camera and included in a plurality of stereo images generated by a stereo camera having a first camera and a second camera sequentially performing imaging operations; a second distance calculation unit that calculates a second distance to the imaging target based on any one of the plurality of stereo images; a correction amount calculation unit that determines whether a difference in pixel position corresponding to the imaging target in each of the plurality of first images is within a predetermined amount, and, if the difference in pixel position is within the predetermined amount, calculates a correction amount for the pixel position corresponding to the imaging target in the image generated by the second camera based on the first distance and the second distance; an image correction unit that moves a position of a pixel value included in an image generated by the second camera in a horizontal direction based on the correction amount; An image processing device comprising:

2. The stereo image processed by the second distance calculation unit includes an image generated by the first camera and an image corrected by the image correction unit. The image processing device according to claim 1 .

3. The image processing device further includes a control unit that controls the operations of the first distance calculation unit and the correction amount calculation unit based on whether a value of the traveling speed of the moving body on which the image processing device is provided is within a predetermined range and whether a value of the yaw rate of the moving body on which the image processing device is provided is within a predetermined range.

3. The image processing device according to claim 1.

4. the first distance calculation unit estimates a pixel position corresponding to the imaging target in the image generated by the first camera based on a traveling speed and a yaw rate of a moving body on which the image processing device is provided; The correction amount calculation unit performs processing when a difference between the pixel position corresponding to the imaging target in each of the plurality of first images and the estimated pixel position is within a predetermined amount. The image processing device according to any one of claims 1 to 3.

5. calculating a first distance, which is a distance to an imaging target, based on a plurality of first images generated by the first camera and included in a plurality of stereo images generated by a stereo camera having a first camera and a second camera sequentially performing an imaging operation; calculating a second distance to the imaging target based on one of the plurality of stereo images; determining whether a difference in pixel position corresponding to the imaging target in each of the plurality of first images is within a predetermined amount, and if the difference in pixel position is within the predetermined amount, calculating a correction amount for the pixel position corresponding to the imaging target in the image generated by the second camera based on the first distance and the second distance; shifting positions of pixel values ​​included in the image generated by the second camera in a horizontal direction based on the correction amount; An image processing method comprising:

Images