Image processing apparatus

Abstract

To provide an image processing apparatus which can suppress reduction of ranging accuracy when accuracy of stereo ranging is reduced.SOLUTION: An image processing apparatus comprises: a second ranging unit (stereo region parallax calculation unit) 200 which can measure a distance to an object by obtaining a parallax in a stereo processing region in which fields of view of at least two imaging apparatuses overlap each other among a plurality of images; an object common image capture region determination unit (object common image capture region calculation unit) 500 which determines an object common image capture region being a region of the object which can be imaged by a plurality of imaging apparatuses in common, on the basis of a detection result of an object detection unit 301, a result of ranging by a first ranging unit 302, and a geometric relation of the plurality of imaging apparatus; and a distance output unit (distance selection unit) 600 which outputs a result of ranging by the first ranging unit 302 or the second ranging unit 200 as a distance from a vehicle to the object depending on the object common image capture region.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that measures the distance to a detection target in a common imaging area of ​​a plurality of imaging devices. [Background technology]

[0002] In recent years, multi-camera systems in which multiple imaging devices (cameras) are installed on a moving object have been developed for autonomous driving and driving assistance technology, etc. In a multi-camera system, distance measurement can be realized by stereo vision between cameras that share a field of view, and accurate distance information to an object can be estimated (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-149675 Summary of the Invention [Problem to be solved by the invention]

[0004] In stereo ranging, the disparity, which is the difference in the position at which the same object is captured by two cameras (right and left cameras) that share a common field of view, is calculated. Since the disparity differs for each depth, the distance to the detected object can be calculated from the calculated disparity.

[0005] The process of searching for the position where the same object is captured is called stereo matching. Stereo matching is performed in the stereo processing area shown in FIG. 13 and FIG. 14. As shown in FIG. 13, the stereo processing area is set at a predetermined angle (θ [deg]) from the edge of the image so as to include the common field of view between the left and right cameras. In the following, the process of calculating the parallax image for the right image based on the right image is described as an example, but the process is similar when the left image is used as the reference. As shown in FIG. 14, stereo matching cuts out a template of a predetermined size from the area set in the stereo processing area in the right image, searches for a similar texture in the same row in the left image, and calculates the position with the highest similarity as the matching position. Finally, the difference between the position where the template is cut out from the right image and the matching position becomes the parallax.

[0006] In FIG. 14, a method for determining a matching position from the similarity with a template in stereo matching has been described. However, a machine learning-based method such as a Convolutional Neural Network (CNN) may also be used, or any other disparity calculation method may also be used.

[0007] Next, scenes in which the accuracy of disparity calculation decreases in stereo matching will be described with reference to Fig. 15(a) to (d). Fig. 15(a) shows an overhead view of the stereo processing area shown in Fig. 13. It shows the stereo processing area set at a specified angle (θ [deg]) from the edge of the right image. Fig. 15(b) shows a common imaging area where the fields of view of the left and right cameras overlap.

[0008] In stereo matching, disparity can be calculated correctly when the same area is captured in the left and right images, but from Fig. 15(a) and Fig. 15(b), we can see that the stereo processing area and the common imaging area are different. Fig. 15(c) shows an area that is included in the stereo processing area but not in the common imaging area (disparity unreliable area). Objects that exist in this area are subject to stereo processing, but are actually only captured in the right image, so disparity cannot be calculated correctly. Therefore, when the distance to an object detected in this area is calculated using stereo ranging, the distance may be inaccurate.

[0009] Fig. 15(d) shows an occluded area. An occluded area is an area that is in the blind spot of one of the cameras due to the difference in the installation position of the cameras. When stereo matching processing is performed on an occluded area, the disparity cannot be calculated correctly because the area is actually captured by only one of the cameras.

[0010] In order to correctly calculate the parallax by stereo matching, the same area needs to be captured by the left and right cameras. In the conventional technology disclosed in Patent Document 1, it is described that distance measurement using parallax is performed when the same subject is captured by the left and right cameras.

[0011] However, even when the same object is captured by the left and right cameras, there is a possibility that the parallax is erroneous. Examples are shown in Figs. 16(a) to 16(c). Fig. 16(a) shows a case where the left and right cameras observe the same object when the distance between the camera and the object to be measured is close, but the area of ​​the object actually captured by the left and right cameras is different. Fig. 16(b) also shows the same object captured by the left and right cameras, but since most of the object is included in the parallax unreliable area, there is a possibility that the distance is erroneous. Fig. 16(c) also shows the same object captured by the left and right cameras, but since most of the object is included in the occluded area, there is a possibility that the distance is erroneous. In such a scene, the conventional technology disclosed in Patent Document 1 may output an erroneous distance.

[0012] That is, in order to calculate the parallax, the same area needs to be captured in the images captured by the multiple imaging devices. In the conventional technology disclosed in Patent Document 1, if the same object is captured between the multiple imaging devices performing stereo ranging, the distance to the object is calculated based on the calculated parallax. However, even if the same object is captured between the multiple imaging devices, it is not necessarily the case that the same area is captured when different areas of the same object are captured, when an object exists in a parallax unreliable area, when an occluded area exists, etc. In such a scene, when stereo ranging based on parallax is performed, the distance to the object may be erroneous.

[0013] The present invention has been made in consideration of the above circumstances, and has an object to provide an image processing device that can suppress a decrease in the accuracy of stereo ranging when the accuracy of stereo ranging decreases. [Means for solving the problem]

[0014] In order to solve the above problem, the image processing device of the present invention is an image processing device that determines the distance from a vehicle to an object contained in an image based on at least one of multiple images captured by multiple imaging devices mounted on the vehicle, and includes an object detection unit that detects an object contained in the field of view of each imaging device, a first ranging unit that measures the distance from each imaging device to the object, a second ranging unit that can measure the distance to the object by calculating parallax in a stereo processing area where the field of view of at least two of the multiple images overlap, an object common imaging area determination unit that determines an object common imaging area, which is an area of ​​the object that can be commonly captured by multiple imaging devices, based on the detection result by the object detection unit, the ranging result by the first ranging unit and the geometric relationship between the multiple imaging devices, and a distance output unit that outputs the ranging result of either the first ranging unit or the second ranging unit as the distance from the vehicle to the object depending on the object common imaging area. Effect of the Invention

[0015] According to the present invention, when the accuracy of stereo ranging decreases, monocular ranging can be selected, and ranging error can be reduced. In other words, a scene in which the accuracy of stereo ranging decreases can be determined, and in that case, the result of monocular ranging can be used instead, thereby suppressing the decrease in ranging accuracy.

[0016] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0017] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a top view showing a configuration example of a sensor unit 100. [Diagram 3] 1A is a diagram illustrating the object common imaging area, FIG. 1B is an output of the object detection / ranging unit 300, FIG. 1B is a diagram illustrating the object common imaging area (bird's-eye view), and FIG. 1C is a diagram illustrating the object common imaging area (image coordinates). [Figure 4] This figure is used to explain the common imaging area of ​​an object (when different areas of the same object are imaged when the object is at a close distance), where (a) is the output of the object detection / ranging unit 300, and (b) is an explanatory diagram of the monocular detection and monocular ranging results. [Diagram 5] This figure is used to explain the object common imaging area (when most of the objects captured by the right camera are included in the parallax unreliable area), where (a) is the output of the object detection / ranging unit 300, (b) is an explanatory diagram of the object common imaging area (bird's-eye view), and (c) is an explanatory diagram of the object common imaging area (image coordinates). [Figure 6] This figure is used to explain the object common imaging area (when most of the objects captured by the right camera are located in an obstructed area), where (a) is the output of the object detection / ranging unit 300, (b) is an explanatory diagram of the object common imaging area (bird's-eye view), and (c) is an explanatory diagram of the object common imaging area (image coordinates). [Figure 7] 11A and 11B are diagrams illustrating calculation results of an object common imaging area. [Figure 8] 4 shows a processing flow of the object common imaging area calculation unit 500. [Figure 9]6A and 6B are diagrams illustrating a stereo distance correction unit 602, in which FIG. 6A illustrates the creation of an object point sequence, and FIG. 6B illustrates the correction of a three-dimensional bounding box. [Figure 10] 6 shows a processing flow of the distance selection unit 600. [Figure 11] FIG. 11 is a block diagram showing an example of the configuration of an image processing device according to a second embodiment of the present invention. [Figure 12] FIG. 4 is an explanatory diagram of a stereo distance measurement result storage unit. [Figure 13] FIG. 1 is an explanatory diagram of a stereo processing area. [Figure 14] FIG. 1 is an explanatory diagram of stereo matching. [Figure 15] 11A and 11B are diagrams for explaining regions where disparity cannot be accurately calculated, where (a) is a stereo processing region, (b) is a common imaging region, (c) is a disparity unreliable region, and (d) is an explanatory diagram of an occlusion region. [Figure 16] These figures are used to explain problematic scenes involving conventional technology. (a) is an explanatory diagram for a case in which different areas of the same object are imaged for a close-up object. (b) is an explanatory diagram for a case in which most of the object imaged by the right camera is included in the parallax unreliable area. (c) is an explanatory diagram for a case in which most of the object imaged by the right camera is in an occluded area. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In each drawing, parts having the same function or configuration are denoted by the same reference numerals, and repeated description may be omitted.

[0019] <Example 1> 1 shows a configuration diagram of an image processing device 1 according to a first embodiment of the present invention. The image processing device 1 according to the first embodiment is mounted on, for example, a vehicle and includes a sensor unit 100, a stereo area disparity calculation unit 200, an object detection / distance measurement unit 300, a sensor geometric information storage unit 400, an object common imaging area calculation unit 500, a distance selection unit 600, and a display / alarm / control unit 700.

[0020] The sensor unit 100 is composed of multiple imaging devices (cameras) that are installed so that their fields of view overlap. An example of the configuration of the sensor unit 100 is shown in Fig. 2. Fig. 2 shows two imaging devices installed on the side of the vehicle 10, with the first imaging device 101 facing the front of the vehicle 10 and the second imaging device 102 facing the rear of the vehicle 10. The two cameras have a common imaging area (Fig. 15(b)) with overlapping fields of view, and distance measurement by stereo vision can be realized in the common imaging area. By providing an area where stereo vision is possible in the side area of ​​the vehicle, the distance to a vehicle running parallel to the vehicle in the adjacent lane can be accurately calculated, and the distance information can be used for warning and avoidance control when a vehicle running parallel to the vehicle approaches or pulls over.

[0021] 2 shows an example of two cameras installed on the sides of the vehicle 10, but the present invention is not limited to this, and imaging devices installed in an orientation such that the observation range is the front or rear of the vehicle 10 may be used. Also, while imaging devices with non-parallel optical axes are shown in Fig. 2, any combination of imaging devices having a common imaging area may be set, and for example, a combination of imaging devices with parallel optical axes, cameras with different camera heights or installation angles, or imaging devices with different angles of view may be used.

[0022] In the following description, the first imaging device 101 will basically be referred to as the right camera and the second imaging device 102 as the left camera, with the image captured by the first imaging device (right camera) 101 being referred to as the right image and the image captured by the second imaging device (left camera) 102 being referred to as the left image.

[0023] The stereo area parallax calculation unit (second distance measurement unit) 200 performs stereo matching on at least two camera images (right image and left image) output by the sensor unit 100, and calculates a parallax image. In stereo matching, one of the left and right images is used as a reference, and a parallax image corresponding to the reference image is output. In the following, the process of calculating a parallax image corresponding to the right image using the right image as a reference will be described, but the same process can also be used when the left image is used as a reference.

[0024] First, optical characteristics such as lens distortion are corrected for the image output by the sensor unit 100. Furthermore, parallelization transformation is performed based on the geometric relationship between the two cameras (right and left cameras). By performing parallelization transformation on each of the right and left images, any area in the right image will always appear in the same row of the left image. A template of a predetermined size is cut out from the right image after parallelization transformation, and the same area is searched for in the same row of the left image, and the disparity is estimated from the difference in the matching positions. Similar processing is performed over the entire stereo processing area of ​​the right image (where the fields of view of the two cameras overlap) to calculate a disparity image corresponding to the right image.

[0025] Although the stereo matching process using template matching has been described above, any method capable of estimating disparity (distance) can be used, and a method such as a Convolutional Neural Network (CNN) that performs convolution integration of an input image and outputs a disparity image can also be used.

[0026] The object detection / distance measurement unit 300 detects an object and measures the distance in the image for which the stereo area disparity calculation unit 200 has calculated the disparity. That is, the object detection / distance measurement unit 300 has an object detection unit 301 that detects an object included in the field of view of each imaging device, and a distance measurement unit (first distance measurement unit) 302 that measures the distance from each imaging device to the object. Although FIG. 1 shows a configuration in which the stereo area disparity calculation unit 200 calculates the disparity for the image output by the first imaging device 101, and the object detection / distance measurement unit 300 detects and measures the object, the same processing is possible even when the image output by the second imaging device 102 is used as a reference.

[0027] The object detection / distance measurement unit 300 analyzes the input image and detects objects (vehicles, pedestrians, motorcycles, etc.). The detected object is output as a bounding box, which is a three-dimensional rectangular parallelepiped that surrounds the detected object. Each vertex of the three-dimensional bounding box stores three-dimensional coordinates in the world coordinate system, and the object is detected and its distance is estimated at the same time. The three-dimensional bounding box is inferred using a neural network model, and YOLO and the like are known as such network models. That is, the object detection / distance measurement unit 300 estimates (measures) the distance to an object included in the field of view of each imaging device using a three-dimensional bounding box, which is a three-dimensional model of the object calculated using only image information captured by each imaging device.

[0028] The sensor geometric information storage unit 400 stores geometric information of each imaging device constituting the sensor unit 100, and is used when the object common imaging area calculation unit 500 calculates the object common imaging area.

[0029] The sensor geometric information storage unit 400 stores the angle of view information and the relative positional relationship of each imaging device in order to calculate a sensing range that can be commonly captured by the first imaging device 101 and the second imaging device 102. The relative positional relationship is stored as a value obtained by calibrating the imaging devices when they are installed in a vehicle.

[0030] The object common imaging area calculation unit (object common imaging area determination unit) 500 calculates (determines) the object common imaging area, which is the area of ​​an object that can be imaged (commonly) by both the left and right cameras, based on the 3D bounding box output by the object detection / ranging unit 300 and the geometric information of each imaging device stored in the sensor geometric information storage unit 400.

[0031] A method for calculating the object common imaging area will be described with reference to Figs. 3(a) to (c). Fig. 3(a) shows the output of the object detection / distance measurement unit 300. Fig. 3(a) shows a three-dimensional bounding box surrounding the vehicle to be detected, and the coordinates in the world coordinate system are stored at each vertex of the three-dimensional bounding box. Fig. 3(b) shows an object rectangular frame obtained by projecting the three-dimensional bounding box shown in Fig. 3(a) onto a plane, the positions of the left and right cameras, and the angle of view in an overhead view. The coordinates of the four points on the bottom surface of the three-dimensional bounding box may be used for the object rectangular frame, but the four points on the top surface may also be used, or the average value of the top and bottom surfaces may also be used. From the object rectangular frame and the positions and angle of view of the left and right cameras, an object common imaging area, which is the area of ​​the object rectangular frame that can be observed from both the left and right cameras, is calculated. When the calculated object common imaging area is displayed on the right image, it becomes Fig. 3(c). Since the side of the vehicle (corresponding to the side of the 3D bounding box) is captured by both the left and right cameras in the common object imaging area, it can be determined that the parallax calculated in this area can be used safely to calculate the distance to the object. On the other hand, since the rear surface of the vehicle (corresponding to the side of the 3D bounding box) is captured only in the right image, it can be determined that the parallax calculated for this area may be incorrect, and therefore it cannot be used to calculate the distance to the object.

[0032] 4(a) and (b) explain a method for determining the object common imaging area in the problem scene described in FIG. 16(a). FIG. 4(a) shows the output of the object detection / ranging unit 300. When the 3D bounding box of the object detected in FIG. 4(a) is projected onto a bird's-eye view, it becomes as shown in FIG. 4(b). From the object rectangular frame, the positions of the left and right cameras, and the information on the angle of view shown in FIG. 4(b), it can be seen that there is no area of ​​the object rectangular frame that can be observed from both the left and right cameras. Therefore, even if the disparity value could be estimated in this scene, it is determined that there is a high possibility that an incorrect value has been calculated, and it is determined that there is no disparity value that can be used for distance measurement by stereo vision.

[0033] 5(a) to (c) explain a method for determining the object common imaging area in the problem scene described in FIG. 16(b). FIG. 5(a) shows the output of the object detection / distance measurement unit 300. When the 3D bounding box of the object detected in FIG. 5(a) is projected onto a bird's-eye view, it becomes as shown in FIG. 5(b). From the object rectangular frame and the positions and angle of view information of the left and right cameras shown in FIG. 5(b), it can be seen that the area of ​​the object rectangular frame that can be observed from both the left and right cameras is only a part of the side of the vehicle (the side of the 3D bounding box corresponding to it). When the calculated object common imaging area is displayed on the right image, it becomes as shown in FIG. 5(c). It can be determined that the parallax calculated in this area can be used without any problem to calculate the distance of the object.

[0034] 6(a) to (c) explain a method for determining the object common imaging area in the problem scene described in FIG. 16(c). FIG. 6(a) shows the output of the object detection / distance measurement unit 300. When the 3D bounding box of the object detected in FIG. 6(a) is projected onto a bird's-eye view, it becomes as shown in FIG. 6(b). From the object rectangular frame and the positions and angle of view information of the left and right cameras shown in FIG. 6(b), it can be seen that the area of ​​the object rectangular frame that can be observed from both the left and right cameras is only a part of the side of the vehicle (the side of the 3D bounding box corresponding to it). When the calculated object common imaging area is displayed on the right image, it becomes as shown in FIG. 6(c). It can be determined that the parallax calculated in this area can be used without any problem to calculate the distance of the object.

[0035] As described in Fig. 4(a) to Fig. 6(c), the object common imaging area calculation unit 500 can specify the object common imaging area for the subject scene shown in Fig. 16(a) to (c) by the above processing. Since the object common imaging area can be captured (commonly) by both the left and right cameras, there is a low possibility of erroneous calculation of parallax, and it can be used for distance measurement by stereo vision.

[0036] 7, a method of outputting information on the object common imaging area calculated by the object common imaging area calculation unit 500 to the distance selection unit 600 will be described. A straight line connecting the left end and right end of the object common imaging area calculated on the overhead view with the center of the right camera is obtained. The angles from the obtained straight line and the field angle end of the right camera are calculated (θ1, θ2), and output to the distance selection unit 600 as the calculation result of the object common imaging area.

[0037] FIG. 8 shows a process flow of the object common imaging area calculation unit 500. The object common imaging area calculation unit 500 acquires 3D bounding box information for an object detected by the object detection / distance measurement unit 300 in step S101. From the acquired 3D bounding box information, an object rectangular frame is set on the bird's-eye view in step S102. The object rectangular frame may be formed by using the coordinates of four points on the bottom surface of the 3D bounding box, but four points on the top surface may be used, or the average value of the top surface and the bottom surface may be used. In step S103, the sensor geometric information storage unit 400 acquires the angle of view information of the right camera stored therein, and in step S104, the object rectangular frame observable by the right camera is calculated. Similarly, in step S105, the sensor geometric information storage unit 400 acquires the angle of view information of the left camera stored therein, and the relative positional relationship with the right camera, and in step S106, the object rectangular frame observable by the left camera is calculated. In step S107, a common portion of the areas observable by the cameras calculated in steps S104 and S106 is calculated to identify a common object imaging area. In step S108, the angle between the common object imaging area specified in step S107 and the edge of the angle of view is calculated (θ1, θ2) and output to the distance selection unit 600.

[0038] In the above, the object common imaging area is converted into angle information, but the object common imaging area may be expressed in any other format, or may be converted into the length of each side or point sequence information and output to the distance selection unit 600. In other words, the size of the object common imaging area may be expressed using at least one or more values ​​of the angle or ratio that the object common imaging area occupies in the angle of view of the imaging device, the width of the object common imaging area on the image, and the ratio that the object common imaging area occupies in the detected object area.

[0039] The distance selection unit (distance output unit) 600 outputs to the display / warning / control unit 700 as the distance from the vehicle to the object, either the monocular ranging result output by the object detection / ranging unit 300 or the stereo ranging result measured based on the disparity calculated by the stereo area disparity calculation unit 200, depending on the object common imaging area output by the object common imaging area calculation unit 500.

[0040] The distance selection unit 600 determines whether or not stereo ranging may be performed based on the output result of the object common imaging area calculation unit 500. If it is determined that stereo ranging may be performed, the distance selection unit 600 corrects the information of the three-dimensional bounding box output by the object detection / ranging unit 300 based on the parallax calculated by the stereo area parallax calculation unit 200, and outputs the corrected information to the display / warning / control unit 700. If it is determined that stereo ranging cannot be performed, the information of the three-dimensional bounding box output by the object detection / ranging unit 300 is output to the display / warning / control unit 700 as is.

[0041] In detail, the distance selection unit 600 has a stereo ranging feasibility determination unit 601 and a stereo distance correction unit 602. When the object common imaging area calculation unit 500 does not output information on the object common imaging area, the stereo ranging feasibility determination unit 601 determines that there is no parallax available for stereo ranging and that stereo ranging is impossible. When the object common imaging area calculation unit 500 outputs information on the object common imaging area (θ1, θ2), the stereo ranging feasibility determination unit 601 determines that stereo ranging is possible since there is parallax available for stereo ranging. In addition, when the size of the object common imaging area is small, the accuracy of stereo ranging may decrease, so the feasibility of stereo ranging may be determined on the condition that the difference between θ1 and θ2 exceeds a predetermined threshold value θth according to the following formula (1). That is, when the size of the object common imaging area exceeds a predetermined value, the distance selection unit 600 may determine that stereo ranging is possible and output the stereo ranging result as the distance from the vehicle to the object to the display / warning / control unit 700 as described later, and when the size of the object common imaging area is equal to or smaller than the predetermined size, the distance selection unit 600 may determine that stereo ranging is impossible and output the monocular ranging result as the distance from the vehicle to the object to the display / warning / control unit 700 as described later. Note that, as described above, the size of the object common imaging area may be expressed using at least one or more values ​​of the angle or ratio of the object common imaging area to the angle of view of the imaging device, the width of the object common imaging area on the image, and the ratio of the object common imaging area to the detected object area. [Number 1] TIFF2025012103000002.tif657

[0042] If stereo ranging feasibility determination unit 601 determines that stereo ranging is possible, stereo distance correction unit 602 corrects the 3D bounding box information output by object detection / ranging unit 300. If stereo ranging feasibility determination unit 601 determines that stereo ranging is not possible, the 3D bounding box information output by object detection / ranging unit 300 is output to display / warning / control unit 700 as is.

[0043] When the stereo distance measurement feasibility determination unit 601 determines that stereo distance measurement is possible, the stereo distance correction unit 602 corrects the 3D bounding box based on the information (θ1, θ2) of the object common imaging area, the 3D bounding box, and the parallax image. FIGS. 9(a) and (b) show the stereo distance correction unit 602. First, the stereo distance correction unit 602 extracts an area within the range of the angle of view θ1 to θ2 of the 3D bounding box as the object common imaging area on the right image. Next, the area of ​​the parallax image corresponding to the object common imaging area obtained on the right image is set as the processing range, and the distance to the object is estimated. Specifically, the object common imaging area on the parallax image is divided into strips, and a representative parallax is calculated for each strip. When the representative parallax of each strip is plotted on a bird's-eye view, a sequence of points corresponding to the side of the object can be obtained as shown in FIG. 9(a). This sequence of points is the distance to the object calculated by stereo ranging (in other words, the object shape calculated from the parallax value of the object common imaging area), and it is assumed that the distance is calculated more accurately than the distance information of the 3D bounding box calculated by monocular vision (in other words, calculated using only image information captured by each imaging device). Next, as shown in FIG. 9(b), correction parameters are calculated to rotate and translate the 3D bounding box so that the obtained sequence of points and the 3D bounding box obtained by monocular vision overlap most closely. The 3D bounding box is moved using the obtained correction parameters, and the values ​​of each vertex of the 3D bounding box are corrected. The finally corrected value of the 3D bounding box is output to the display / alarm / control unit 700.

[0044] 10 shows a processing flow of the distance selection unit 600. In step S201, the distance selection unit 600 acquires information on the object common imaging area from the object common imaging area calculation unit 500, and in step S202, checks whether or not the object common imaging area exists to a certain extent (predetermined value). If it exists, a correction process of the three-dimensional bounding box by stereo distance measurement is performed, and if it does not exist, the information of the three-dimensional bounding box is output as it is. In other words, if the object common imaging area does not exist to a certain extent (predetermined value), the information of the three-dimensional bounding box (monocular distance measurement result) is output to the display / alarm / control unit 700 in step S203. If the object common imaging area exists to a certain extent (predetermined value), a parallax image is acquired from the stereo area parallax calculation unit 200 in step S204, and a point sequence of the object is created from the parallax information in the object common imaging area in step S205. In step S206, correction parameters that maximize the overlap between the point sequence of the created object and the 3D bounding box are calculated, and in step S207, the calculated correction parameters are used to correct the 3D bounding box (the values ​​of each vertex of the box), and the corrected information is output to the display / alarm / control unit 700 as the stereo distance measurement result.

[0045] The display / warning / control unit 700 controls a display or warning for the driver, or controls the vehicle, based on the distance to the object output by the distance selection unit 600.

[0046] According to the above-described first embodiment, when the accuracy of stereo ranging decreases, monocular ranging can be selected, and ranging errors can be reduced.

[0047] <Example 2> An image processing device according to a second embodiment of the present invention will be described with reference to Fig. 11 and Fig. 12. Fig. 11 shows a configuration diagram of an image processing device 2 according to a second embodiment of the present invention. In the image processing device 2 according to the second embodiment, a third imaging device 103 and a stereo ranging result storage unit 800 are added to the image processing device 1 in Fig. 1. Note that the same reference symbols are used for configurations that are the same as or equivalent to those described in the first embodiment, and differences will be mainly described below.

[0048] The object detection / distance measurement unit 300 in the first embodiment estimates the distance (monocular distance measurement) to the object detected by the first imaging device 101 using only the information of the first imaging device 101. However, the measurement accuracy of the monocular distance measurement may decrease depending on the scene. For example, in the monocular distance measurement that estimates the distance from the foot position of the detected object to the road surface model, the shape of the road surface is uneven, and the distance cannot be calculated accurately when the deviation from the road surface model increases. In addition, when calculating the distance to the object using a machine learning model such as CNN, the distance measurement accuracy decreases depending on the scene, object, weather, and time period that are not included in the learning image. If the distance accuracy decreases significantly, the object common imaging area calculation unit 500 may erroneously calculate the object common imaging area, and the accuracy of the stereo distance measurement may also decrease.

[0049] In the second embodiment, in order to improve the distance accuracy of the object detection / ranging unit 300, if stereo ranging has been performed on the same object in the past, the accuracy of monocular ranging can be improved by using the information from that time.

[0050] The stereo ranging result storage unit 800 will be described with reference to Fig. 12. Fig. 12 illustrates a scene in which another vehicle 30 passes the vehicle 20 in the adjacent lane. When the other vehicle 30 is behind the vehicle 20, the other vehicle 30 is in a common imaging area of ​​the second imaging device 102 installed on the side of the vehicle 20 and the third imaging device 103 installed on the rear of the vehicle 20. As the other vehicle 30 approaches the vehicle 20, the other vehicle 30 moves to a monocular area that can only be observed by the second imaging device 102, and then moves to the common imaging area of ​​the first imaging device 101 and the second imaging device 102, and then to the monocular area of ​​the first imaging device 101, in that order.

[0051] When the other vehicle 30 straddles the common imaging area of ​​the first imaging device 101 and the second imaging device 102, the distance used to calculate the object common imaging area is not the result of monocular distance measurement, but the distance calculated from the result of previous stereo distance measurement in the common imaging area of ​​the second imaging device 102 and the third imaging device 103. Specifically, when the other vehicle 30 is located in the common imaging area of ​​the second imaging device 102 and the third imaging device 103, scale information (actual size, width, height, etc.) of the other vehicle 30 is saved and used for distance measurement in the monocular area. For example, when an object that was captured at a height H [pixels] on the image approaches and its height changes to 1 / 2H [pixels], it can be calculated that the distance has approached to 1 / 2. The result of previous stereo distance measurement in the common imaging area of ​​the second imaging device 102 and the third imaging device 103 used here is stored (saved) in the stereo distance measurement result storage unit 800.

[0052] When information on past stereo ranging as described above is stored (saved) in the stereo ranging result storage unit 800, the object detection / ranging unit 300 can use that information to calculate the distance to the object.

[0053] According to the above-mentioned second embodiment, by using the results of past stereo ranging, the distance accuracy (monocular ranging accuracy) of the object detection / ranging unit 300 is improved, thereby enabling the object common imaging area calculation unit 500 to correctly calculate the object common imaging area.

[0054] The following modified examples are also within the scope of the present invention, and it is possible to combine the configurations shown in the modified examples with the configurations described in the above-mentioned embodiments, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0055] <Variation 1> In the above embodiment, the object detection / distance measurement unit 300 calculates the distance to the detected object using only the information of the monocular image. However, if the vehicle is equipped with other sensors capable of measuring distance (LiDAR, millimeter wave, sonar, etc.), the distance to the detected object may be calculated using those sensors.

[0056] Furthermore, object detection itself may be performed based on information from other sensors.

[0057] In other words, the object detection / ranging measurement unit 300 may detect an object for each of multiple images captured by multiple imaging devices mounted on the vehicle based on the images captured by each imaging device (object detection unit 301), and measure the distance to the object for each of multiple images captured by multiple imaging devices mounted on the vehicle based on the images captured by each imaging device (ranging measurement unit 302); or may detect an object included in the field of view of each imaging device based on the ranging results by a ranging sensor mounted on the vehicle (object detection unit 301), and measure the distance from each imaging device to the object based on the ranging results by the ranging sensor mounted on the vehicle (ranging measurement unit 302).

[0058] <Variation 2> In the above embodiment, the detected vehicle is approximated to a three-dimensional bounding box, which is a rectangular parallelepiped model, to calculate the object common imaging area. However, it is not necessary to be limited to the three-dimensional bounding box, and the detected vehicle may be represented by a more detailed three-dimensional model to calculate the object common imaging area.

[0059] In addition, for pedestrians, detection results may be obtained using a 2D bounding box rather than a 3D bounding box. In such cases, a cylindrical model that fits into the detected 2D bounding box can be created, and the common imaging area for the object can be calculated using the same process.

[0060] <Variation 3> The three-dimensional bounding box does not have to be expressed by the three-dimensional coordinates of each vertex. For example, it may be expressed by the three-dimensional coordinates of the center position of the three-dimensional bounding box, the size (width, depth, height) of the three-dimensional bounding box, and the orientation of the three-dimensional bounding box.

[0061] <Modification 4> The sensor geometric information storage unit 400 stores the relative positional relationship calculated by calibration when each imaging device is installed in the vehicle. However, it is expected that the relative positional relationship calculated by calibration will change due to factors such as vibrations while the vehicle is running, temperature changes due to seasons and time of day, and deterioration over time. Therefore, even after each imaging device is installed, calibration may be performed at any time to recalculate the relative positional relationship.

[0062] In this case, image feature points are calculated from the images of each imaging device output by the sensor unit 100, and translation and rotation parameters of each imaging device are calculated.

[0063] <Summary> As described above, the image processing devices 1 and 2 of the present embodiment are image processing devices that calculate a distance from a vehicle to an object included in an image based on at least one of a plurality of images captured by a plurality of imaging devices mounted on the vehicle, an object detection unit 301 that detects an object included in the field of view of each imaging device; a first distance measuring unit 302 for measuring a distance from each of the imaging devices to the object; a second distance measuring unit (stereo area disparity calculation unit) 200 capable of measuring a distance to the object by calculating disparity in a stereo processing area where the fields of view of at least two of the multiple images of the image capturing device overlap; an object common imaging area determination unit (object common imaging area calculation unit) 500 that determines an object common imaging area, which is an area of ​​the object that can be commonly imaged by the plurality of imaging devices, based on a detection result by the object detection unit 301, a distance measurement result by the first distance measurement unit 302, and a geometric relationship between the plurality of imaging devices; and a distance output unit (distance selection unit) 600 that outputs the distance measurement result of either the first distance measurement unit 302 or the second distance measurement unit 200 as the distance from the vehicle to the object depending on the object common imaging area.

[0064] The object common imaging area determination unit (object common imaging area calculation unit) 500 obtains a three-dimensional bounding box surrounding the object, and determines an object common imaging area that can be captured in common by the multiple imaging devices on at least one of the multiple faces that constitute the bounding box.

[0065] When the size of the object common imaging region exceeds a predetermined value, the distance output unit (distance selection unit) 600 outputs the distance measurement result by the second distance measurement unit 200 as the distance from the vehicle to the object.

[0066] The first ranging unit 302 further includes a stereo ranging result memory unit 800 that stores the ranging results by the second ranging unit 200, and when the ranging results for the object are stored in the stereo ranging result memory unit 800, the first ranging unit 302 measures the distance from each of the imaging devices to the object based on the ranging results for the object stored in the stereo ranging result memory unit 800.

[0067] The ranging result by the first ranging unit 302 is a three-dimensional model of the object (such as a three-dimensional bounding box or cylindrical model) calculated using only the image information captured by each imaging device, and the distance output unit (distance selection unit) 600 calculates correction parameters such that the object shape calculated from the disparity value of the object common imaging area best overlaps with the three-dimensional model, corrects the ranging result (monocular ranging result) by the first ranging unit 302 with the calculated correction parameters, and outputs it as the ranging result (stereo ranging result) by the second ranging unit 200.

[0068] In other words, the image processing devices 1 and 2 of the present embodiment are image processing devices that calculate a distance from a vehicle to an object included in at least one image among a plurality of images captured by a plurality of imaging devices mounted on the vehicle, an object detection / distance measurement unit 300 that detects an object included in a field of view of the imaging device and measures a first distance, which is the distance to the detected object; a stereo area disparity calculation unit 200 for calculating disparity in a stereo processing area where the fields of view of at least two of the plurality of images of the imaging device overlap; A sensor geometric information storage unit 400 that stores the geometric relationships of a plurality of sensors mounted on the vehicle; an object common imaging area determination unit (object common imaging area calculation unit) 500 that determines an object common imaging area, which is an area of ​​the object that can be commonly imaged by the plurality of imaging devices, based on a detection result by the object detection / distance measurement unit 300, the first distance, and a geometric relationship between the plurality of sensors; and a distance selection unit 600 that outputs, depending on the object common imaging area, either the first distance or the second distance measured based on the disparity calculated by the stereo area disparity calculation unit 200 as the distance from the vehicle to the object.

[0069] The first distance is a three-dimensional model (such as a three-dimensional bounding box or cylindrical model) of the object calculated using only the image information captured by each imaging device, and the distance selection unit 600 calculates correction parameters that maximize the overlap between the object shape calculated from the parallax value of the object common imaging area and the three-dimensional model, and outputs a second distance obtained by correcting the first distance with the calculated correction parameters.

[0070] When the size of the object common imaging area exceeds a predetermined value, the distance selection unit 600 outputs the second distance as the distance from the vehicle to the object, and when the size of the object common imaging area is equal to or smaller than a predetermined size, the distance selection unit 600 outputs the first distance as the distance from the vehicle to the object.

[0071] The object detection / ranging unit 300 further includes a stereo ranging result memory unit 800 that stores the second distance, and when the ranging result for the object is stored in the stereo ranging result memory unit 800, the object detection / ranging unit 300 measures the distance from each of the imaging devices to the object based on the ranging result for the object stored in the stereo ranging result memory unit 800.

[0072] According to this embodiment, when the accuracy of stereo ranging decreases, monocular ranging can be selected, and ranging error can be reduced. In other words, a scene in which the accuracy of stereo ranging decreases can be determined, and in that case, the result of monocular ranging can be used instead, thereby suppressing the decrease in ranging accuracy.

[0073] Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0074] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to easily explain the present invention, and the present invention is not necessarily limited to the embodiment having all of the described configurations.

[0075] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0076] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0077] 1 Image processing device (Example 1) 100 Sensor section 200 Stereo area disparity calculation unit (second distance measurement unit) 300 Object detection / distance measurement section 301 Object detection unit 302 Distance measuring unit (first distance measuring unit) 400 Sensor Geometry Information Storage Unit 500 object common imaging area calculation unit (object common imaging area determination unit) 600 Distance selection unit (distance output unit) 601 Stereo distance measurement feasibility determination unit 602 Stereo Distance Correction Unit 700 Display / Alarm / Control Unit 2. Image Processing Device (Example 2) 800 Stereo distance measurement result memory unit

Claims

1. 1. An image processing device that calculates a distance from a vehicle to an object included in at least one image captured by a plurality of image capturing devices mounted on a vehicle, the image capturing device comprising: an object detection unit that detects an object included in the field of view of each of the imaging devices; a first distance measuring unit that measures a distance from each of the imaging devices to the object; a second distance measuring unit capable of measuring a distance to the object by calculating a parallax in a stereo processing area where the fields of view of at least two of the plurality of images of the imaging device overlap; an object common imaging area determination unit that determines an object common imaging area, which is an area of ​​the object that can be commonly imaged by the plurality of imaging devices, based on a detection result by the object detection unit, a distance measurement result by the first distance measurement unit, and a geometric relationship between the plurality of imaging devices; a distance output unit that outputs a distance measurement result of either the first distance measurement unit or the second distance measurement unit according to the object common imaging area as a distance from the vehicle to the object.

2. 2. The image processing device according to claim 1, The object common imaging area determination unit obtains a three-dimensional bounding box surrounding the object, and determines an object common imaging area that can be commonly captured by the multiple imaging devices in at least one of the multiple faces that constitute the bounding box.

3. 2. The image processing device according to claim 1, The distance output unit outputs the distance measurement result by the second distance measurement unit as the distance from the vehicle to the object when the size of the object common imaging area exceeds a predetermined value.

4. 2. The image processing device according to claim 1, a stereo distance measurement result storage unit that stores a distance measurement result by the second distance measurement unit, An image processing device, wherein the first ranging unit measures the distance from each of the imaging devices to the object based on the ranging result for the object stored in the stereo ranging result memory unit when the ranging result for the object is stored in the stereo ranging result memory unit.

5. 2. The image processing device according to claim 1, The object detection unit detects the object for each of a plurality of images based on the images captured by each of the imaging devices, The first distance measuring unit measures the distance to the object for each of a plurality of images based on images captured by each imaging device.

6. 2. The image processing device according to claim 1, The object detection unit detects an object included in a field of view of each of the imaging devices based on a distance measurement result by a distance measurement sensor mounted on the vehicle, The first distance measuring unit measures the distance from each of the imaging devices to the object based on a distance measurement result by the distance measuring sensor.

7. 2. The image processing device according to claim 1, the distance measurement result by the first distance measurement unit is a three-dimensional model of the object calculated using only image information captured by each of the image capture devices, The distance output unit calculates correction parameters such that the object shape calculated from the parallax value of the object common imaging area and the three-dimensional model overlap most closely, corrects the distance measurement result by the first distance measurement unit with the calculated correction parameters, and outputs it as the distance measurement result by the second distance measurement unit.

Images