Object detection device

Abstract

This object detection device for a vehicle comprises at least one processor (105a), wherein the processor: calculates the width and height of an object from detection results of a stereo camera (101) mounted on a vehicle (201); calculates an amount of change in each of the width and height of the object obtained from the stereo camera; determines a reference feature quantity to be used to calculate the relative distance between the vehicle and the object, from among the width and height of the object calculated from detection results of a monocular camera (102) mounted on the vehicle, on the basis of the amounts of change in the width and height of the object obtained from the stereo camera; calculates the reference feature quantity from detection results of the monocular camera when the object moves from the recognition range (202-S) of the stereo camera to the recognition range (203-M) of the monocular camera; and calculates the relative distance on the basis of the reference feature quantity and the feature quantity that is among the width and height of the object obtained from the stereo camera and that corresponds to the reference feature quantity.

Description

Technical Field

[0001] The present invention relates to an object detection device for a vehicle.

Background Art

[0002] There has been active development of driving support technologies that ensure safety by mounting various sensors on automobiles and monitoring the surroundings of the vehicle. For example, there are a lane departure warning device that monitors the front of the vehicle with a camera and warns if the vehicle deviates from the driving lane by recognizing a white line, or a collision mitigation brake that monitors a front obstacle and automatically brakes if there is a possibility of a collision.

[0003] These support technologies are expanding to monitor not only the front of the vehicle but also the rear, sides, etc. Along with this, the detection means is not only a camera but also appropriately combines a radar, a sonar, etc. to monitor an object that requires attention for the vehicle. This is called FUSION technology, and the combination of sensors and the optimization of the recognition technology processing of each sensor are important. And since the monitorable directions, ranges, accuracies, and environmental conditions are different for each sensor, there may be a need to transfer the monitoring target from one sensor to another when the target (monitoring object) approaches the vehicle.

[0004] For example, in the technology described in Patent Document 1, regarding vehicle rear monitoring using a stereo camera and a monocular camera, the vehicle width (or height) of a following vehicle within the recognition range of the stereo camera is obtained and stored, and when the following vehicle moves from the recognition range of the stereo camera to the recognition range of the monocular camera, the distance to the following vehicle is obtained based on the stored vehicle width (or height) and the vehicle width (or height) of the following vehicle obtained from the recognition result of the monocular camera.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] Incidentally, if an object being detected moves from the recognition range of the first sensor to the recognition range of the second sensor in a way that changes the object's orientation relative to each sensor, the width and height of the object detected by each sensor (the apparent dimensions at each sensor) may appear to have changed significantly.

[0007] For example, while the width of a motorcycle viewed from the front is narrower than that of a car, when it moves in a way that allows its side to be seen by each sensor, such as when changing lanes, the apparent change in vehicle width observed by each sensor tends to be larger than that of a car. Under these circumstances, if the distance is calculated based on the vehicle width from the recognition results of a monocular camera using the technology described in Patent Document 1, the monocular camera may perceive the vehicle width as having changed significantly more than the vehicle width obtained from the recognition results of a stereo camera, potentially resulting in a calculated distance to the object being closer than it actually is. Conversely, if the monocular camera perceives a change in vehicle width as having changed less, the calculated distance to the object may be farther than it actually is. In other words, the distance calculation based on Patent Document 1 is prone to discrepancies in the actual distance to the object and has room for improvement. This problem stems from the characteristic of monocular cameras that they are prone to distance errors caused by changes in the dimensions of objects in images, and the same can be said not only for changes in the width (vehicle width) of an object as pointed out above, but also when the sensor detects that the height of an object has changed as if it had changed.

[0008] The object of the present invention is to provide an object detection device that can suppress the discrepancy between the calculated distance between the vehicle and the detected object and the actual distance when the distance calculation is transferred from the first sensor to the second sensor. [Means for solving the problem]

[0009] The present invention includes several means for solving the above problems, but one example is an object detection device for a vehicle equipped with at least one processor, wherein the processor calculates several feature quantities of an object from the detection result of a first sensor mounted on the vehicle, calculates the change in each of the several feature quantities obtained from the first sensor, determines a reference feature quantity to be used for calculating the relative distance between the vehicle and the object from among the feature quantities of the object calculated from the detection result of a second sensor mounted on the vehicle, based on the change in the several feature quantities, and when the object moves from the recognition range of the first sensor to the recognition range of the second sensor, calculates the reference feature quantity from the detection result of the second sensor, and calculates the relative distance based on the reference feature quantity and the feature quantity from among the several feature quantities obtained from the first sensor that corresponds to the reference feature quantity. [Effects of the Invention]

[0010] According to the present invention, it is possible to suppress the discrepancy between the calculated distance between the vehicle and the detected object and the actual distance when the distance calculation is transferred from the first sensor to the second sensor. [Brief explanation of the drawing]

[0011] [Figure 1] This is a configuration diagram including an object detection device and its peripheral devices according to a first embodiment of the present invention. [Figure 2] This figure shows an example of the installation locations for the rear-view stereo camera 101 and monocular camera 102. [Figure 3] This is a software configuration diagram for ECU105. [Figure 4] An explanatory diagram of a 2D bounding box. [Figure 5] This shows the processing flow for backward monitoring performed by the ECU 105 (processor 105a) according to the first embodiment. [Figure 6] A diagram illustrating the operation of the rearward monitoring according to the first embodiment is shown. [Figure 7] This shows the processing flow for backward monitoring performed by the ECU 105 (processor 105a) according to the second embodiment. [Figure 8] It is a diagram showing an example of a weighted map. [Figure 9] It is a diagram showing an example of a weighted map. [Figure 10] It is a diagram showing an example of a weighted map. [Figure 11] Explanatory diagram of a three-dimensional bounding box. [Figure 12] It is a diagram showing the processing flow of rear monitoring according to the second embodiment. [Figure 13] It is a diagram showing an example of the installation locations of the stereo cameras 101 and the monocular cameras 102 for front monitoring. [Figure 14] It is a diagram showing an operation explanatory diagram of front monitoring according to the fourth embodiment.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The object of this embodiment is a detection device for an object approaching the host vehicle during road travel, such as an automobile or a motorcycle. In the following, a plurality of embodiments will be described, but the same reference numerals will be given to the common parts in each embodiment, and repeated explanations may be omitted. Note that the drawings may be schematically represented compared to the actual aspect for the sake of clearer explanation, but this is merely an example and does not limit the interpretation of the present invention.

[0013] <First Embodiment> FIG. 1 is a configuration diagram including an object detection device and its peripheral devices according to the first embodiment of the present invention. The object detection device according to this embodiment includes a first sensor 101, a second sensor 102, a first processing circuit 103, a second processing circuit 104, and an electronic control unit (hereinafter sometimes referred to as ECU) 105. The ECU 105 can be connected to a vehicle control device (other ECU) 107, an alarm device (monitor, speaker, etc.) 108, etc. via a CAN interface (CAN IF) 106 The first sensor 101 is a sensor mounted on the host vehicle to monitor the rear of the host vehicle, and is any one of a stereo camera, a radar, a sonar, a LiDAR, and a monocular camera. In this embodiment, the first sensor 101 is a stereo camera. The camera image of the stereo camera 101 is processed by the first processing circuit 103 and input to the ECU 105.

[0014] The second sensor 102 is a sensor mounted on the host vehicle to monitor the rear side of the host vehicle, and is preferably a monocular camera. The camera image of the monocular camera 102 is processed by the second processing circuit 104 and input to the ECU 105.

[0015] The ECU 105 includes a memory 105b which is a data storage device, and at least one processor 105a that executes various processes according to a program stored in the memory 105b. The ECU 105 processes information on an object (for example, another vehicle) approaching the host vehicle using the processing results of the first processing circuit 103 and the second processing circuit 104 (in other words, the detection results of the first sensor 101 and the second sensor 102). The information includes relative distance data between the host vehicle and the object. Then, the information on the object approaching from the rear side of the host vehicle is sent to the CAN bus through the CAN IF 106 and output to the vehicle control device 107 and the notification device 108.

[0016] Here, a system including respective signal input parts (not shown) for inputting signals from the stereo camera 101 and the monocular camera 102, the first processing circuit 103 and the second processing circuit 104, the ECU 105, and the CAN IF 106 is shown as an example, but these are not necessarily devices contained within one unit. For example, the cameras 101, 102 and the corresponding processing circuits 103, 104 may exist as one unit each, and may be connected by signal lines to some control device including the ECU 105. Also, the first processing circuit 103 or the second processing circuit 104 may be incorporated within the ECU 105, or the first processing circuit 103, the second processing circuit 104, and the ECU 105 may be incorporated within the vehicle control device 107.

[0017] Figure 2 shows examples of installation locations for the rear-view stereo camera 101 and monocular camera 102. The stereo camera 101 can be installed, for example, on the rear bumper to monitor the area behind the vehicle 201. On the other hand, the monocular camera 102 can be installed, for example, on the door mirror to monitor the rear side of the vehicle. Region 202-S in the figure indicates the recognition area of ​​the stereo camera 101, and region 203-M indicates the recognition area of ​​the monocular camera 102. Here, the recognition area may refer to the camera's field of view or angle of view, or it may refer to the area within the camera's field of view that can be processed by the image processing unit. However, for the monocular camera 102, it is desirable to install it so that the optical axis is as far away from the vehicle 201 as possible to minimize reflections of the vehicle 201. Also, for the stereo camera 101, the range that can be viewed in stereo is the shaded region 202-S where the fields of view of the left and right cameras overlap. The stereo camera 101 may also be installed inside the vehicle 201. For example, it could be installed on the ceiling on the rear seat side, so as to monitor the area behind the vehicle through the rear window. This would prevent water droplets or mud from adhering to the stereo camera 101 during rainy weather.

[0018] Figure 3 shows the software configuration diagram of the ECU 105, with the ECU 105's internal components displayed. The ECU 105 can function as a first feature calculation unit 301, a second feature calculation unit 302, a sensor recognition area movement determination unit 303, an object distance estimation unit 304, and a collision risk determination unit 305 by executing the software stored in memory 105b using the processor 105a.

[0019] The first feature calculation unit 301 (processor 105a) assigns a recognition ID to a following vehicle traveling behind the vehicle detected in the recognition area 202-S of the stereo camera 101, calculates multiple feature quantities (width and height) of the following vehicle from the detection result (camera image) of the stereo camera 101, and calculates the amount of change in these multiple feature quantities caused by the movement of the following vehicle. The first feature calculation unit 301 can also determine a feature quantity used to calculate the relative distance between the vehicle and the following vehicle (hereinafter sometimes referred to as the "reference feature quantity") from among the feature quantities of the following vehicle calculated from the detection result of the monocular camera 102, based on the amount of change in the multiple feature quantities caused by the movement of the following vehicle. The reference feature quantity is a feature quantity obtained by the monocular camera 102 that is used to calculate the relative distance when the following vehicle is in the recognition area 203-M of the monocular camera 102.

[0020] The features of the following vehicle can be calculated, for example, using bounding box detection, a common technique for detecting objects. The first feature calculation unit 301 (processor 105a) uses deep learning or the like to assign a bounding box (called a 2D bounding box) 402 to objects in the camera images of the stereo camera 101 and monocular camera 102, representing the smallest rectangle (square) that circumscribes the object (following vehicle 401) as shown in Figure 4. The first feature calculation unit 301 (processor 105a) can then calculate the width and height (features) of the following vehicle from the width and height of the 2D bounding box. Furthermore, the first feature calculation unit 301 (processor 105a) can calculate the change in the width and height (features) of the following vehicle by monitoring the change in the width and height of the 2D bounding box caused by the movement of the following vehicle. The calculated feature data and the amount of change are used, for example, by the sensor recognition area movement determination unit 303 and the object distance estimation unit 304. Alternatively, instead of using the width and height of the bounding box, the width and height of the following vehicle may be calculated from, for example, the edges of the following vehicle in the camera image of the stereo camera 101.

[0021] The second feature calculation unit 302 (processor 105a) calculates multiple feature quantities (width and height) of a following vehicle traveling to the rear side of the vehicle, detected by the recognition area 203-M of the monocular camera 102, from the detection result (camera image) of the monocular camera 102. The feature quantities can be calculated in the same way as the first feature calculation unit 301, and can be calculated, for example, based on a two-dimensional bounding box. The calculated feature quantities will include reference feature quantities. The calculated feature quantity data is used, for example, by the sensor recognition area movement determination unit 303 and the object distance estimation unit 304.

[0022] The sensor recognition area movement determination unit 303 (processor 105a) determines whether the same following vehicle has moved from the recognition area 202-S of the stereo camera 101 to the recognition area 203-M of the monocular camera 102. If it determines that the same following vehicle has moved, it assigns the same recognition ID to the following vehicle in recognition area 203-M as it had in recognition area 202-S (inherits the recognition ID).

[0023] When the sensor recognition area movement determination unit 303 determines that the following vehicle has moved from the recognition area 202-S of the stereo camera to the recognition area 203-M of the monocular camera 102, the object distance estimation unit 304 (processor 105a) calculates the relative distance between the following vehicle and the self-vehicle based on a reference feature from the feature quantities of the following vehicle obtained from the monocular camera 102 that is used to calculate the relative distance between the following vehicle and the self-vehicle, and a feature from among the multiple feature quantities of the following vehicle obtained from the stereo camera 101 that corresponds to the reference feature.

[0024] The collision risk determination unit 305 (processor 105a) determines whether there is a possibility of a collision between the vehicle and the following vehicle based on the relative distance between the vehicle and the following vehicle calculated by the object distance estimation unit 304, and transmits whether to permit or deny collision avoidance control by the vehicle control device 107 via CAN IF 106 based on the determination result. If the collision risk determination unit 305 determines that there is a possibility of a collision based on the determination result, it may also transmit that fact to the notification device 108.

[0025] Figure 5 shows the processing flow for rearward monitoring executed by the ECU 105 (processor 105a) according to the first embodiment. First, when a vehicle following the vehicle is detected by the stereo camera 101, the processor 105a assigns a recognition ID to the following vehicle and calculates the feature quantities (width and height) of the following vehicle based on the camera image of the stereo camera 101 (S501). Then, the processor 105a calculates the amount of change in the feature quantities (width and height) caused by the movement of the following vehicle, such as changing lanes, based on the camera image of the stereo camera 101 (S502).

[0026] Next, the processor 105a determines, based on the change in the width and height of the following vehicle, a reference feature (i.e., one of the width and height of the following vehicle) to be used in the calculation of relative distance from the width and height (features) of the following vehicle obtained from the monocular camera 102 in subsequent processing (S503). In this embodiment, the processor 105a selects a feature (width or height) with a small change from the features of the following vehicle obtained from the stereo camera 101 as the reference feature (S503) up until just before the vehicle goes outside the recognition area 202-S of the stereo camera 101 (S503). Subsequently, the feature corresponding to the feature selected in S503 from the features of the following vehicle obtained from the monocular camera 102 will be used in the calculation of relative distance in S506. In other words, for example, if S503 determines that the change in height among the features obtained by the stereo camera 101 is small, then the height obtained by the monocular camera 102 will be used as a feature in the calculation of the relative distance in S506 using the image from the monocular camera 102.

[0027] The method of selecting a reference feature (a feature used when calculating relative distance) from the image of the monocular camera 102 described here is just one example, and other selection methods are acceptable as long as they allow for the reproduction of the intent of this embodiment. For example, if the change in the width of the feature obtained by the stereo camera 101 exceeds a predetermined threshold, height may be selected as the feature obtained by the monocular camera 102 and used as the reference feature.

[0028] Next, the processor 105a determines, based on the recognition ID of the following vehicle, whether the following vehicle has moved out of the recognition area 202-S of the stereo camera 101 and into the recognition area 203-M of the monocular camera 102 (S504). If it is determined that the following vehicle has moved into the recognition area 203-M of the monocular camera 102, the process proceeds to S505. For example, if it is determined that a following vehicle with a predetermined recognition ID that was detected within the recognition area 202-S of the stereo camera 101 has moved outside the recognition area 202-S, and shortly thereafter a following vehicle is detected in the recognition area 203-M of the monocular camera 102, the same recognition ID is transferred to the following vehicle, and it is considered that the following vehicle with that recognition ID has moved from the recognition area 202-S to the recognition area 203-M.

[0029] Next, after determining that the following vehicle has moved into the recognition area 203-M of the monocular camera 102, the processor 105a calculates the feature quantities (width and height) of the following vehicle based on the camera image from the monocular camera 102 (S505). Note that the feature quantities used for calculating the relative distance have already been determined in S503, so here only the determined feature quantities may be calculated from the camera image of the monocular camera 102.

[0030] Next, processor 105a calculates the relative distance to the following vehicle (object) located to the rear and side using the feature quantities calculated in S505 that correspond to the feature quantities determined in S503, and one of the following equations (1) and (2) (S506). Which of equations (1) or (2) is used to calculate the relative distance depends on the feature quantity determined in S503. If the determined feature quantity is "width", equation (1) is used, and if it is "height", equation (2) is used.

[0031] The following equation (1) is used to calculate the relative distance (first relative distance) dw when the feature quantity determined in S503 is "width". In equation (1), dw is the relative distance (first relative distance) from the vehicle to the following vehicle (object), f is the focal length of the monocular camera 102, W is the width of the following vehicle (object) calculated by the stereo camera 101, and △x represents the width of the following vehicle (object) in the image from the monocular camera 102.

[0032]

number

[0033] Equation (2) below is used to calculate the relative distance (second relative distance) dh when the feature quantity determined in S503 is "height". In equation (2), dh is the relative distance (second relative distance) from the vehicle to the following vehicle (object), f is the focal length of the monocular camera 102, H is the height of the following vehicle (object) calculated by the stereo camera 101, and △y indicates the height of the following vehicle (object) in the image from the monocular camera 102.

[0034]

number

[0035] Next, the processor 105a determines, based on the calculation results of the relative distances dw and dh between its own vehicle and the vehicle to its rear and side, whether or not the vehicle to its rear and side is likely to collide with its own vehicle (S507). If it is determined that there is a possibility of collision, it performs and transmits a decision to permit control intervention to avoid the collision (S508). If it is determined that there is no possibility of collision, it performs and transmits a decision to prohibit control intervention to avoid the collision (S509).

[0036] (operation) Figure 6 shows an explanatory diagram of the operation of rearward monitoring according to this embodiment. Reference numerals 401-404 in the figure indicate the positions of the vehicle (motorcycle) following the vehicle 201, and it is assumed that the following vehicle moves from position 401 to position 404 via a lane change.

[0037] The stereo camera 101 detects a following vehicle behind the vehicle at position 401, and the processor 105a (first feature calculation unit 301) calculates the feature quantities (width and height) and their changes from position 401 to position 402 until the following vehicle leaves the recognition area 202-S (S501, 502). At this time, the processor 105a (first feature calculation unit 301) may also calculate the relative speed and relative position between the vehicle 201 and the following vehicle. When the stereo camera 101 detects a following vehicle, a recognition ID is assigned to that following vehicle.

[0038] Next, the processor 105a (first feature calculation unit 301) determines a reference feature (width or height) for calculating the relative distance using the camera image from the monocular camera 102, in accordance with the change in the feature (width and height) of the following vehicle in the recognition region 202-S, which includes positions 401 to 402 (S503).

[0039] Next, suppose a following vehicle that was traveling at position 401 behind vehicle 201 begins to change lanes and moves to position 403 to the rear side of vehicle 201. At this time, the processor 105a (sensor recognition area movement determination unit 303) determines that the following vehicle has moved from recognition area 202-S to recognition area 203-M using the stereo camera 101 and the monocular camera 102 (S504). Whether the following vehicle detected by the stereo camera 101 and the monocular camera 102 is the same can be determined, for example, if the relative position difference of the following vehicle detected simultaneously by both cameras is within a predetermined value. If it is the same following vehicle, the recognition ID is also passed on.

[0040] Next, when a vehicle following vehicle 403 located to the rear side of vehicle 201 is detected by the monocular camera 102, the processor 105a (second feature calculation unit 302) calculates data related to the following vehicle. The data calculated at this time includes feature quantities (width and height) of the following vehicle calculated from the camera image of the monocular camera 102 (S505).

[0041] The processor 105a (object distance estimation unit 304) calculates the relative distances dw and dh between the following vehicle at position 403 and the self-vehicle 201 based on, for example, the feature quantities (width Δx and height Δy) of the following vehicle acquired by the monocular camera 102 at position 403, the feature quantities (width W or height H) of the following vehicle acquired by the stereo camera 101 selected in S503, and the above equation (1) or (2) (S506).

[0042] The system determines whether a collision has occurred based on the calculation results of the relative distances dw and dh between the following vehicle 403 and the own vehicle 201 (S507). The determination result is transmitted from the CAN IF 106 to the vehicle control device 107 and the notification device 108 (S508, S509). Even if the following vehicle moves to position 404 and completes the lane change, monitoring by the monocular camera 102 (calculation of relative distance) may continue.

[0043] (effect) (1) As described above, in this embodiment, the processor 105a calculates multiple feature quantities (width and height) of an object (following vehicle) from the detection results of the first sensor (stereo camera 101) mounted on the vehicle 201, calculates the change in each of the multiple feature quantities (width and height) obtained from the first sensor (stereo camera 101), determines a reference feature quantity to be used for calculating the relative distance between the vehicle 201 and the object (following vehicle) from the feature quantities of the object calculated from the detection results of the second sensor (monocular camera 102) mounted on the vehicle 201, based on the change in the multiple feature quantities, and when the object (following vehicle) moves from the recognition range 202-S of the first sensor (stereo camera 101) to the recognition range 203-M of the second sensor (monocular camera 102), the processor 105a calculates the reference feature quantity from the detection results of the second sensor (monocular camera 102), and calculates the relative distance based on the reference feature quantity and the feature quantity corresponding to the reference feature quantity from the multiple feature quantities obtained from the first sensor (stereo camera 101).

[0044] In this way, by calculating the relative distance within the recognition range 203-M of the second sensor (monocular camera 102) based on the feature quantities obtained from the second sensor (monocular camera 102) that are determined from the change in the feature quantities obtained from the first sensor (stereo camera 101) (reference feature quantities), and the feature quantities obtained from the first sensor (stereo camera 101), the relative distance can be calculated based on the feature quantities obtained from the second sensor that correspond to the change in the feature quantities obtained from the first sensor (reference feature quantities), while utilizing the accurate feature quantities of the object obtained from the first sensor (stereo camera 101). This improves the accuracy of the relative distance calculation by the second sensor. In other words, it is possible to suppress the discrepancy between the relative distance between the vehicle and the following vehicle and the actual distance when the distance calculation is handed over from the first sensor (stereo camera 101) to the second sensor (monocular camera 102). Furthermore, since the accuracy of the relative distance calculation is improved, malfunctions or failures of collision avoidance operations can be reduced.

[0045] Furthermore, it is preferable that the characteristic quantities of an object calculated and used in the calculation described in (1) above are related to the dimensions of the object.

[0046] (2) In (1) above, the first sensor is preferably a stereo camera, radar, sonar, LiDAR, and monocular camera, and the second sensor is preferably a monocular camera. Although any of the radar, sonar, LiDAR, and monocular camera sensors may be used instead of the stereo camera 101, the stereo camera 101 is the most preferable when considering the balance between monetary cost and computational accuracy (the same applies to each of the following embodiments).

[0047] (3) In (1) above, the plurality of feature quantities are the width and height of the object, and it is preferable that the processor 105a determines a reference feature quantity based on the amount of change in the width and height of the object obtained from the stereo camera 101 (first sensor).

[0048] (4) In (1) above, the plurality of feature quantities are the width and height of the object, and it is preferable that the processor 105a determines the feature quantity with the smallest change among the width W and height H of the object, based on the change in the width W and height H of the object obtained from the first sensor, as the reference feature quantity. Specifically, in this embodiment, the processor 105a calculates the width W and height H of a following vehicle (object) from the detection results of the stereo camera (first sensor) 101 mounted on the vehicle 201, calculates the change in the width W and height H of the following vehicle obtained from the stereo camera 101, and when the following vehicle moves out of the recognition range 202-S of the stereo camera 101 and into the recognition range 203-M of the monocular camera (second sensor) 102 mounted on the vehicle 201, it selects the feature quantity with the smallest change in width W and height H of the following vehicle (object) based on the change in width W and height H obtained from the stereo camera (first sensor) 101, and calculates the relative distances dw and dh based on the feature quantity (Δx or Δy) corresponding to the selected feature quantity among the width Δx and height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102.

[0049] With this configuration, when the object to be detected moves out of the recognition range 202-S of the stereo camera 101 and into the recognition range 203-M of the monocular camera 102, the relative distance is calculated based on features that change little even if the object moves due to a lane change or the like. This suppresses a decrease in the accuracy of the relative distance calculated by the monocular camera 102.

[0050] In the above description, the processor 105a "determines a feature quantity with a small change in width and height of the following vehicle (object) obtained from the stereo camera (first sensor) 101 as a reference feature quantity, and calculates the relative distances dw and dh based on the feature quantity (Δx or Δy) corresponding to the reference feature quantity from the width Δx and height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102." However, instead, the processor 105a may be configured to "determine the height Δy of the following vehicle (object) obtained from the monocular camera (second sensor) 102 as a reference feature quantity when the change in width W of the following vehicle (object) obtained from the stereo camera (first sensor) 101 exceeds a predetermined threshold, and use it to calculate the relative distance dh."

[0051] With this configuration, the relative distance dh is calculated based on "height," which is relatively less variable than "width" due to object movement. As described above, this suppresses the reduction in accuracy of the relative distance calculated by the monocular camera 102.

[0052] <Second Embodiment> In this embodiment, an example is described in which the relative distance d is calculated by weighting the relative distance (first relative distance dw) calculated from equation (1) and the relative distance (second relative distance dh) calculated from equation (2) according to the amount of change in the feature quantity of the following vehicle calculated from the camera image of the stereo camera 101 (for example, the rate of change in the width of the following vehicle).

[0053] Figure 7 shows the processing flow of back-end monitoring performed by the ECU 105 (processor 105a) according to the second embodiment.

[0054] First, steps S501 and S502 at the beginning of the flowchart are the same as those in the first embodiment shown in Figure 5.

[0055] Next, the processor 105a calculates a weight from the change amount calculated in S502 and the weighting map. FIG. 8 shows an example of the weighting map. When using this weighting map, in S502, the change rate of the width of the following vehicle by the stereo camera 101 is calculated as the "change amount". Various methods can be used to calculate the change rate. For example, there is a method of calculating the ratio of the width when the following vehicle enters the recognition area 202-S to the width immediately before the following vehicle exits the recognition area 202-S. The change rate calculated in S502 is converted into the weight wf (0 < wf ≤ 1) of the second relative distance dh by the map in FIG. 8. In FIG. 8, the greater the change rate (change amount) of the width of the following vehicle obtained from the stereo camera 101, the greater the weight wf of the second relative distance dh is weighted.

[0056] Note that the weighting map based on the change rate of this feature amount (width or height) is an example, and any map setting is possible. That is, as long as the gist of this embodiment can be reproduced, other weightings may be added. For example, in addition to or instead of the weighting in FIG. 8, the weight may be varied according to the type of the following vehicle (object), such as the weighting diagram according to the type of the following vehicle shown in FIG. 9. As shown in FIG. 9, for example, since the ratio of the length to the width of a motorcycle is larger than that of a passenger car, it is preferable to make the weight wf of the second relative distance dh larger than that of a passenger car. Further, the weight may be changed according to the relative distance between the host vehicle and the following vehicle (object), such as the weighting diagram according to the relative distance between the host vehicle and the following vehicle (object) shown in FIG. 10. For example, in the example of FIG. 10, since it is more difficult to capture the change rate of the feature amount (width and height) due to the influence of noise and the like as the relative distance to the following vehicle (object) is farther, the weighting may be set lightly.

[0057] S504 and 505 are the same as those in the first embodiment.

[0058] In S506-W, the processor 105a first calculates the first relative distance dw and the second relative distance dh from the feature quantities (width W and height H) obtained by the stereo camera 101 in S501, the feature quantities (width Δx and height Δy) obtained by the monocular camera 102 in S505, and equations (1) and (2) (in the first embodiment, only one of the two relative distances dw and dh was calculated, but in this embodiment, both relative distances dw and dh are calculated). Next, the processor calculates the relative distance d between the vehicle and the following vehicle based on the calculated first relative distance dw and second relative distance dh, the weight wf calculated in S503-W, and equation (3) below.

[0059] Equation (3) below is an equation for calculating the relative distance d from the weight wf, the first relative distance dw, and the second relative distance dh. In equation (3), wf is the weight of the second relative distance dh, dw is the first relative distance calculated from equation (1) above, and dh is the second relative distance calculated from equation (2) above. In other words, the relative distance d in this embodiment is calculated as the average (weighted average) of the first relative distance and the second relative distance, taking into account the weight wf corresponding to the change in the width of the following vehicle obtained from the stereo camera 101.

[0060]

number

[0061] The remaining sections S507, S508, and S509 are the same as in the first embodiment.

[0062] In the above explanation, weighting was performed according to the change in width (rate of change) of the following vehicle obtained from the stereo camera 101. However, weighting may also be performed according to the change in height (rate of change) of the following vehicle obtained from the stereo camera 101. In this case, it is preferable to use a weighting map that converts the change in height (rate of change) of the following vehicle into a weight hf of the first relative distance dw (the ratio of distance measurement by width hf). Furthermore, it is preferable to use the following equation (4) instead of the above equation (3).

[0063]

number

[0064] In this embodiment configured as described above, the processor 105a calculates a first relative distance dw between the self-vehicle and the following vehicle (object) based on the width of the following vehicle (object) obtained from the stereo camera 101 and the width of the following vehicle (object) obtained from the monocular camera 102 (second sensor), calculates a second relative distance dh between the self-vehicle and the following vehicle (object) based on the height of the following vehicle (object) obtained from the stereo camera 101 and the height of the following vehicle (object) obtained from the monocular camera 102 (second sensor), and calculates the relative distance d by weighting the first relative distance dw and the second relative distance dh according to the amount of change in the width (or height) of the following vehicle (object) obtained from the stereo camera 101 (first sensor) and summing the two.

[0065] By calculating the relative distance as a weighted average of the first relative distance dw and the second relative distance dh in this way, the relative distance is calculated taking into account the changes in both the width and height of the following vehicle obtained from the stereo camera 101 (first sensor), thus improving the accuracy of the relative distance.

[0066] In particular, in the above embodiment, the relative distance d is calculated such that the weight of the height relative to the width of the following vehicle increases as the rate of change of the width of the following vehicle increases. As a result, when the rate of change of the width of the following vehicle increases, the relative distance is calculated by giving more weight to the height, which has a relatively smaller rate of change, thus improving the accuracy of the relative distance calculation.

[0067] <Third Embodiment> While the above embodiments mentioned the use of a 2D bounding box for object detection, this embodiment describes the case where a 3D bounding box is used. In this embodiment, a following vehicle is detected using a 3D bounding box, and if the detection confidence falls below a predetermined value and the confidence level of object detection is low, the flow of the first and second embodiments (Figures 5 and 7) is executed. Detection confidence is an index value that indicates how accurately the features (width, height, length, and type) of the detected object are captured, and here, a larger number indicates a higher confidence level. For example, if the features (width, height, length, and type) of the object are continuously captured during object detection, the detection confidence can be said to be high.

[0068] First, let's explain 3D bounding box detection. 3D bounding box detection is an object detection method that uses a three-dimensional bounding box (3D outline) to surround an object, as shown in Figure 11. This is achieved by using AI (artificial intelligence) learning in the first processing circuit 103 and the second processing circuit 104 to extract the features of an object from the camera images of the stereo camera 101 and the monocular camera 102, thereby capturing the object in three dimensions. Through this 3D bounding box detection, the first feature calculation unit 301 and the second feature calculation unit 302 can obtain the object's feature quantities (width, height, length, type), detection confidence information, and the distance from the vehicle to the object from the detected cubic outline, and assign a recognition ID.

[0069] Figure 12 shows the processing flow for backward monitoring according to this embodiment.

[0070] First, when a following vehicle is detected by the stereo camera 101 using a 3D bounding box, the processor 105a calculates the feature quantities of the following vehicle (width, height, length, and type), the distance to the following vehicle, and the detection confidence level (S551).

[0071] Next, the processor 105a compares the detection confidence level obtained in S551 with a predetermined value (S552), and if the detection confidence level is less than or equal to the predetermined value, it executes the processing from S501 onwards in Figure 5 or Figure 7.

[0072] In other words, in this embodiment, the processor 105a assigns a 3D bounding box to the following vehicle (object) on the image obtained by the stereo camera 101, and if the detection confidence of the 3D bounding box is below a predetermined value, it performs the processing necessary for calculating the relative distance as shown in S501 or below in Figure 5 or Figure 7. In other words, according to this embodiment, the flow in Figure 5 or Figure 7 is executed only when the confidence of 3D bounding box detection is low, and the present invention can be applied to systems that use 3D bounding boxes for object detection.

[0073] <Fourth Embodiment> Next, we will describe an embodiment in which both the stereo camera 101 and the monocular camera 102 are installed in front of the vehicle to detect a vehicle in front of the vehicle and estimate its distance.

[0074] Figure 13 shows an example of the installation locations for the forward-facing stereo camera 101 and monocular camera 102. The stereo camera 101 is installed in the front of the vehicle to monitor the area in front of the vehicle 201. The monocular camera 102, on the other hand, monitors the area in front of the vehicle 201 and is installed in a position that allows it to monitor the blind spot area of ​​the stereo camera 101 located in front of the vehicle 201. The resulting recognition areas are, for example, 204-S and 205-M, respectively. Here, the recognition area may refer to the camera field of view or angle of view, or it may refer to the area within the camera field of view that can be processed by the image processing unit. In the case of the stereo camera 101, the range that can be viewed in stereo is the area 204-S where the fields of view of the left and right cameras overlap. For example, the area within a predetermined distance from the front of the vehicle 201 is the blind spot area of ​​the stereo camera 101.

[0075] The hardware configuration of the object detection device is the same as in Figure 1.

[0076] The software configuration of the ECU105 is the same as in Figure 3. However, the sensor recognition area movement determination unit 303 in this embodiment determines whether a vehicle in front of the vehicle has entered the blind spot of the recognition area 204-S of the stereo camera 101 and moved to the recognition area 205-M of only the monocular camera 102. If it is determined that the same vehicle has moved, the unit assigns the same recognition ID to the following vehicle in the recognition area 205-M as it had been in the recognition area 204-S (recognition ID is inherited).

[0077] The forward monitoring processing flow executed by ECU105 (processor 105a) is the same as in Figure 5, although it is necessary to interpret following vehicles as preceding vehicles.

[0078] Figure 14 shows an explanatory diagram of the operation of forward monitoring according to this embodiment. Reference numerals 601-604 in the figure indicate the positions of the vehicle (motorcycle) in front of the vehicle 201, and it is assumed that the vehicle in front moves from position 601 to position 604 after changing lanes.

[0079] The stereo camera 101 detects the vehicle in front of the vehicle at position 601, and the processor 105a (first feature calculation unit 301) calculates the feature quantities (width and height) and their changes from position 601 to position 603, and then to the blind spot of the recognition area 204-S (S501, 502). At this time, the processor 105a (first feature calculation unit 301) may also calculate the relative speed and relative position between the vehicle 201 and the vehicle in front. When the stereo camera 101 detects the vehicle in front, it assigns a recognition ID to that vehicle.

[0080] Next, the processor 105a (first feature calculation unit 301) determines a reference feature (width or height) for calculating the relative distance using the camera image from the monocular camera 102, in accordance with the change in the feature (width and height) of the vehicle ahead in the recognition region 204-S, which includes positions 601 to 603 (S503).

[0081] Next, suppose the vehicle in front of vehicle 201, which was traveling at position 601 in front of vehicle 201, begins to change lanes and moves to position 604 in front of vehicle 201. At this time, the processor 105a (sensor recognition area movement determination unit 303) determines that the vehicle in front has moved into the overlapping area of ​​the blind spot region of recognition region 204-S and the recognition region 205-M, as determined by the stereo camera 101 and the monocular camera 102 (S504). Whether the vehicle in front detected by the stereo camera 101 and the monocular camera 102 is the same can be determined, for example, if the relative position difference of the vehicle in front detected simultaneously by both the stereo camera 101 and the monocular camera 102 is within a predetermined value. If it is the same vehicle in front, the recognition ID is also passed.

[0082] Next, when the monocular camera 102 detects a vehicle ahead at position 604 in front of the vehicle 201, the processor 105a (second feature calculation unit 302) calculates data related to that vehicle ahead. The data calculated at this time includes feature quantities (width and height) of the vehicle ahead calculated from the camera image of the monocular camera 102 (S505).

[0083] The processor 105a (object distance estimation unit 304) calculates the relative distances dw and dh between the vehicle in front at position 604 and the vehicle itself 201 based on, for example, the feature quantities (width Δx and height Δy) of the vehicle in front acquired by the monocular camera 102 at position 604, the feature quantities (width W or height H) of the vehicle in front acquired by the stereo camera 101 selected in S503, and the above equation (1) or (2) (S506).

[0084] The system determines whether a collision has occurred based on the calculation results of the relative distances dw and dh between the vehicle 604 ahead and the vehicle 201 itself (S507). The determination result is transmitted from the CAN IF 106 to the vehicle control device 107 and the notification device 108 (S508, S509).

[0085] As described above, even when forward monitoring is performed using a stereo camera 101 and a monocular camera 102 as in this embodiment, when the object to be detected enters the blind spot of the recognition range 204-S of the stereo camera 101 and moves to the recognition range 205-M of the monocular camera 102 alone, the relative distance is calculated based on feature quantities that change little even if the object moves due to a change of course, etc., thus suppressing a decrease in the accuracy of the relative distance calculated by the monocular camera 102. In other words, when the distance calculation is handed over from the stereo camera (first sensor) 101 to the monocular camera (second sensor) 102, it is possible to suppress the deviation of the relative distances dw and dh between the vehicle and the following vehicle from the actual distance. Furthermore, since the accuracy of the relative distance calculation is improved, malfunctions or failures of collision avoidance operations can be reduced.

[0086] It goes without saying that, in forward monitoring, weighting as in the second embodiment or a 3D bounding box as in the third embodiment may also be used.

[0087] It should be noted that the present invention is not limited to the embodiments described above, and includes various modifications that do not depart from the spirit of the invention. For example, the present invention is not limited to having all the configurations described in the embodiments described above, but also includes configurations in which some of those configurations are omitted. Furthermore, it is possible to add or replace a part of the configuration of one embodiment with a configuration of another embodiment.

[0088] Furthermore, the configurations, functions, and execution processes of the ECU105 described above may be partially or entirely implemented in hardware (for example, by designing the logic for executing each function using an integrated circuit). Alternatively, the configurations of the ECU105 described above may be implemented as a program (software) that is read and executed by a processing unit (e.g., a CPU) to realize the functions of the ECU105 configuration. Information related to this program can be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (magnetic disks, optical disks, etc.).

[0089] Furthermore, in the descriptions of each embodiment above, the control lines and information lines shown are those deemed necessary for the description of that embodiment, but this does not necessarily mean that all control lines and information lines related to the product are shown. In reality, it is safe to assume that almost all components are interconnected. [Explanation of symbols]

[0090] 101...Stereo camera (first sensor), 102...Monocular camera (second sensor), 105...ECU, 105a...Processor, 201...Own vehicle, 202-S...Stereo camera recognition area (recognition range), 203-M...Monocular camera recognition area (recognition range), 204-S...Stereo camera recognition area (recognition range), 205-M...Monocular camera recognition area (recognition range), 401-404...Position of rear vehicle, 601-604...Position of front vehicle

Claims

1. An object detection device for a vehicle comprising at least one processor, The aforementioned processor, From the detection results of the first sensor mounted on the vehicle, multiple feature quantities of the object are calculated. The amount of change of each of the multiple feature quantities obtained from the first sensor is calculated, From the detection results of the second sensor mounted on the vehicle, a reference feature quantity used to calculate the relative distance between the vehicle and the object is determined based on the change in the plurality of feature quantities. When the object moves from the recognition range of the first sensor to the recognition range of the second sensor, the reference feature quantity is calculated from the detection result of the second sensor, and the relative distance is calculated based on the reference feature quantity and the feature quantity from the plurality of feature quantities obtained from the first sensor that corresponds to the reference feature quantity. An object detection device characterized by the following features.

2. In the object detection device according to claim 1, The first sensor is one of a stereo camera, radar, sonar, LiDAR, and monocular camera. The second sensor is a monocular camera. An object detection device characterized by the following features.

3. In the object detection device according to claim 1, The aforementioned multiple feature quantities are the width and height of the object, The processor determines the reference feature quantity based on the amount of change in the width and height of the object obtained from the first sensor. An object detection device characterized by the following features.

4. In the object detection device according to claim 1, The aforementioned multiple feature quantities are the width and height of the object, The processor determines the feature quantity with the smallest change in the width and height of the object obtained from the first sensor as the reference feature quantity. An object detection device characterized by the following features.

5. In the object detection device according to claim 1, The aforementioned multiple feature quantities are the width and height of the object, The processor determines the height of the object as the reference feature quantity when the amount of change in the width of the object obtained from the first sensor exceeds a predetermined threshold. An object detection device characterized by the following features.

6. In the object detection device according to claim 1, The aforementioned multiple feature quantities are the width and height of the object, The aforementioned reference features are the width and height of the object. The aforementioned processor, Based on the width of the object obtained from the first sensor and the width of the object obtained from the second sensor, a first relative distance between the vehicle and the object is calculated. Based on the height of the object obtained from the first sensor and the height of the object obtained from the second sensor, the second relative distance between the vehicle and the object is calculated. The relative distance is calculated by applying weights to the first relative distance and the second relative distance according to the amount of change in the width and height of the object obtained from the first sensor, and then summing the two. An object detection device characterized by the following features.

7. In the object detection device according to claim 6, The weight of the second relative distance is set to increase as the amount of change in the width of the object obtained from the first sensor increases. An object detection device characterized by the following features.

8. In the object detection device according to claim 6, The weighting based on the change in width and height of the object obtained from the first sensor differs depending on the type of vehicle. An object detection device characterized by the following features.

9. In the object detection device according to claim 6, The weighting based on the change in width and height of the object obtained from the first sensor differs depending on the magnitude of the first relative distance and the second relative distance. An object detection device characterized by the following features.

10. In the object detection device according to claim 1, The first sensor is a stereo camera that monitors the rear of the vehicle. The second sensor is a monocular camera that monitors the rear and side of the vehicle. An object detection device characterized by the following features.

11. In the object detection device according to claim 1, The first sensor is a stereo camera that monitors the area in front of the vehicle. The second sensor is a monocular camera installed in a position that monitors the area in front of the vehicle and can monitor the blind spot of the stereo camera located in front of the vehicle. An object detection device characterized by the following features.

12. In the object detection device according to claim 1, The first sensor is a stereo camera, The processor assigns a three-dimensional bounding box to the object on the image obtained by the stereo camera, and if the detection confidence of the three-dimensional bounding box is less than or equal to a predetermined value, it performs the calculations necessary for calculating the relative distance. An object detection device characterized by the following features.

Images