Object detection device

Abstract

This object detection device (100) comprises a radar signal processor (3), a camera image processor (4), and a fusion processor (5) that executes identity determination processing for the position of an object detected by the radar signal processor (3) and the position of an object detected by the camera image processor (4), and uses the result of the identity determination processing for vehicle control. The camera image processor (4) comprises a region classification unit (13) that performs region classification of a first structure, which is a structure on a road surface that the vehicle can pass, in the vicinity of a detected position in captured image data based on radar detection data. The fusion processor (5) comprises an obstacle determination unit (19) that collates the result of region classification and first radar detection data, which is radar detection data that has produced a failure in the identity determination processing, and if the result of region classification is not included in the position of the first radar detection data, determines that the result of region classification is an obstacle and if the result of region classification is included in the position of the first radar detection data, determines the results of region classification is a first structure.

Description

Object detection device

[0001] The present disclosure relates to an object detection device mounted on a vehicle.

[0002] Vehicle-mounted object detection devices have the function of quickly detecting people, structures, etc. in the vehicle's environment and controlling the vehicle and issuing warnings, contributing to safe vehicle driving. Vehicle-mounted object detection devices use various types of sensors, such as radar, cameras, LiDAR (Light Detection and Ranging), and ultrasonic sensors, but in recent years, fusion-type object detection devices that combine multiple sensors to improve performance have become widely used.

[0003] In such a fusion-type object detection device, when the sensor is composed of a radar and a camera, radar is excellent at detecting position and speed, and a camera is excellent at recognizing objects, so it is common to use radar detection data for position and speed, and camera recognition data for object recognition (for example, Patent Document 1).

[0004] WO 2010 / 119860

[0005] On the other hand, camera object recognition requires feature learning, such as deep learning, and camera object recognition has difficulty detecting unlearned objects. Therefore, radar detection is effective for detecting unlearned objects. However, radar can detect road surface structures that vehicles can pass through, such as cracks in the road, manholes, and gratings, as obstacles, resulting in false positives and a low accuracy rate as an object detection device.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an object detection device that can prevent structures on a road surface that a vehicle can pass through from being detected as an obstacle and reduce false detections.

[0007] In order to solve the above-mentioned problems and achieve the object, the object detection device of the present disclosure includes a radar signal processor that processes a signal received from a radar to detect the position and velocity of an object and output radar detection data including the position and velocity of the detected object, a camera image processor that detects the position and type of the object based on image data captured by a camera, and a fusion processor that performs an identity determination process between the position of the object detected by the radar signal processor and the position of the object detected by the camera image processor and uses the result of the identity determination process for vehicle control. The camera image processor includes an area classification unit that classifies a first structure, which is a structure on a road surface through which a vehicle can pass, around the detection position in the radar detection data in the captured image data. The fusion processor includes an obstacle determination unit that compares the first radar detection data, which is radar detection data that failed the identity determination process, with the result of the area classification, and determines the result of the area classification as an obstacle for vehicle passage if the position of the first radar detection data is not included in the position of the first radar detection data, and determines the result of the area classification as the first structure if the position of the first radar detection data is included in the position of the first radar detection data.

[0008] The object detection device of the present disclosure has the effect of preventing structures on a road surface that a vehicle can pass through from being detected as an obstacle, thereby reducing false detections.

[0009] FIG. 1 is a block diagram showing a configuration of an object detection device according to an embodiment; FIG. 2 is a flowchart showing an operation procedure of the object detection device according to an embodiment;

[0010] An object detection device according to an embodiment will be described in detail below with reference to the drawings.

[0011] 1 is a block diagram showing the configuration of an object detection device 100 according to an embodiment. The object detection device 100 includes a radar 1, a camera 2, a radar signal processor 3, a camera image processor 4, and a fusion processor 5. The object detection device 100 is mounted on a vehicle (not shown). The output of the object detection device 100 is input to a vehicle control unit 30 mounted on the vehicle and used to control the vehicle. In the following description, an object represents an object that is desired to be detected when controlling the vehicle, and corresponds to a person, a vehicle, an obstacle, etc. Examples of obstacles include utility poles, walls, guardrails, shelf pillars, delineators, and traffic lights. In object recognition by the camera 2, an obstacle corresponds to an object that has not yet been learned by deep learning.

[0012] The radar 1 emits electromagnetic waves toward an object and receives reflected signals from the object. The radar signal processor 3 processes the received signals from the radar 1 to detect the position and speed of the object. In vehicle-mounted applications, the radar 1 generally uses the FMCW (Frequency Modulated Continuous Wave) method or the FCM (Fast Chirp Modulation) method, and is composed of high-frequency semiconductor components, power semiconductor components, a substrate, a crystal device, chip components, an antenna, etc.

[0013] The radar signal processor 3 generally includes an MCU (Micro Control Unit), a CPU (Central Processing Unit), or the like. The radar signal processor 3 includes a distance detector 6, a speed detector 7, a horizontal angle detector 8, and a radar detection data storage unit 9. The distance detector 6, the speed detector 7, and the horizontal angle detector 8 perform Fast Fourier Transformation (FFT) in the distance direction, the speed direction, and the horizontal angle direction, respectively, to detect the distance, speed, and horizontal angle of an object. The radar detection data storage unit 9 stores the position and speed of an object detected by the radar 1. The position and speed of an object detected by the radar 1 are called radar detection data.

[0014] The camera 2 captures a forward image and acquires image data. The camera image processor 4 detects the position and type of an object based on the image data captured by the camera 2. The image data captured by the camera 2 is also referred to as captured image data. The camera image processor 4 also classifies the captured image data into multiple regions and acquires the positions, sizes, and region types of the multiple classified regions. The camera 2 is composed of components such as a lens, a holder, a CMOS (Complementary Metal Oxide Semiconductor) sensor, power semiconductor components, and a quartz device.

[0015] An MCU, a CPU, a GPU (Graphics Processing Unit), or the like is used as the camera image processor 4. The camera image processor 4 includes an object recognition unit 10, a deep learning feature database for object recognition 11, a camera-recognized object data storage unit 12, a region classification unit 13, a deep learning feature database for region classification 14, and a camera region classification data storage unit 15.

[0016] The object recognition unit 10 recognizes objects such as people and vehicles from image data captured by the camera 2 by referring to a deep learning feature database 11 for object recognition. The deep learning feature database 11 for object recognition is a database of features obtained by deep learning. The camera-recognized object data storage unit 12 stores the type and position of the recognized object obtained by the object recognition unit 10 and outputs them to the downstream fusion processor 5. The type and position of the recognized object obtained by the object recognition unit 10 are called camera-recognized object data. For object recognition by the object recognition unit 10, a network algorithm such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector) is used.

[0017] The area classification unit 13 classifies the first structure, which is a structure on a road surface through which a vehicle can pass, from the image data captured by the camera 2 based on the selection result of the recognized object data selection unit 17 of the fusion processor 5 and the deep learning feature database for area classification 14. The deep learning feature database for area classification 14 is a database of features related to structures on a road surface through which a vehicle can pass, obtained through deep learning. Examples of structures on a road surface through which a vehicle can pass include road cracks, manholes, gratings, etc. The camera area classification data storage unit 15 stores the area type, position, and size of the area for a vehicle-passable structure acquired by the area classification unit 13 and outputs the area type, position, and size to the subsequent fusion processor 5. The area type, position, and size included in the area data for a vehicle-passable structure stored in the camera area classification data storage unit 15 are referred to as camera area classification data. For example, an algorithm such as semantic segmentation is used for area classification in the area classification unit 13.

[0018] The fusion processor 5 includes a recognized object identity determining unit 16 , a recognized object data selecting unit 17 , a fusion recognized object data storage unit 18 , an obstacle determining unit 19 , and a fusion obstacle data storage unit 20 .

[0019] The recognized object identity determination unit 16 executes identity determination processing between the object position in the radar detection data and the recognized object position in the camera-recognized object data. This identity determination processing links the position and speed detected by the radar 1 with the recognized object type detected by the camera 2. If the identity determination is successful, the recognized object data selection unit 17 transfers the position and speed detected by the radar 1 and the recognized object type detected by the camera 2 to the fusion recognized object data storage unit 18. On the other hand, if the identity determination is unsuccessful, the recognized object data selection unit 17 considers the recognized object to be an obstacle other than a person or a vehicle, and transfers the radar detection data to the area classification unit 13 and the obstacle determination unit 19.

[0020] The area classification unit 13 uses the position of the radar detection data transferred from the recognized object data selection unit 17 to narrow down the area classification processing range in the image data captured by the camera 2. Generally, area classification processing imposes a high load, so this processing can speed up the area classification processing.

[0021] The obstacle determination unit 19 compares the position, size, and area type of the camera area classification data with the position of the first radar detection data, which is radar detection data that failed to be identified and is output from the recognized object data selection unit 17, and determines whether the position of the radar detection data is included in the area of ​​a structure on the road surface that the vehicle can pass through. If the position of the radar detection data is not included in the area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination unit 19 transfers and stores the position and speed of the radar detection data to the fusion obstacle data storage unit 20. If the position of the radar detection data is included in the area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination unit 19 discards the position and speed of the radar detection data.

[0022] The data stored in the fusion recognized object data storage unit 18 and the data stored in the fusion obstacle data storage unit 20 are transferred to the vehicle control unit 30 and used for vehicle control.

[0023] 2 is a flowchart showing the operation procedure of the object detection device 100 according to the embodiment. The operation of the object detection device 100 will be described with reference to FIG.

[0024] First, when object detection processing for the current frame is started (step S1), received data acquired by the radar 1 is input to the radar signal processor 3 (step S2). The distance detector 6, velocity detector 7, and horizontal angle detector 8 of the radar signal processor 3 detect the position and velocity of the object (step S3). The detected position and velocity data are stored in the radar detection data storage unit 9 (step S4).

[0025] On the other hand, when object detection processing for the current frame is started (step S1), captured image data acquired by the camera 2 is input to the camera image processor 4 (step S5). The object recognition unit 10 of the camera image processor 4 recognizes an object from the captured image data by referring to the deep learning feature database for object recognition 11 (step S6). The object recognition unit 10 stores the position and type of the recognized object in the camera recognized object data storage unit 12 (step S7).

[0026] The recognized object identity determination unit 16 of the fusion processor 5 performs identity determination processing based on the position data from the radar 1 obtained in step S4 and the recognized object position data from the camera 2 (step S8). This identity determination processing links the position and speed data detected by the radar 1 with the recognized object type data detected by the camera 2. If the identity determination processing is successful (step S9: Yes), the recognized object data selection unit 17 transfers the position and speed data detected by the radar 1 stored in the radar detection data storage unit 9 and the recognized object type data stored in the camera recognized object data storage unit 12 to the fusion recognized object data storage unit 18 (step S10). The fusion recognized object data storage unit 18 stores the position and speed data detected by the radar 1 and the recognized object type data recognized by the camera 2 (step S11). The object position, speed, and recognized object type data stored in the fusion recognized object data storage unit 18 are output to the vehicle control unit 30 (step S20).

[0027] If the identity determination fails (step S9: No), the recognized object data selection unit 17 outputs the position and speed data detected by the radar 1 stored in the radar detection data storage unit 9 to the area classification unit 13 and the obstacle determination unit 19 (step S12). The area classification unit 13 performs area classification on the image data acquired by the camera 2 around the position of the radar detection data by referring to the deep learning feature database for area classification 14, thereby classifying structures on the road surface that can be passed by a vehicle (step S13). The area classification unit 13 stores the position, size, and area type of the area-classified data in the camera area classification data storage unit 15 (step S14).

[0028] Thereafter, the obstacle determination unit 19 of the fusion processor 5 compares the position, size, and area type of the camera area classification data with the position of the radar detection data for which the identification failed, output from the recognized object data selection unit 17, and determines whether the position of the radar detection data is included in an area of ​​a structure on the road surface that the vehicle can pass through, such as a crack on the road surface, a manhole, or a grating (step S15). In this determination process, if the position of the radar detection data is not included in an area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination is determined to be successful (step S16: Yes), and the procedure proceeds to step S17. On the other hand, if the position of the radar detection data is included in an area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination is determined to be unsuccessful (step S16: No), and the procedure proceeds to step S18. In step S17, the obstacle determination unit 19 outputs the position and speed of the radar detection data to the fusion obstacle data storage unit 20. The fusion obstacle data storage unit 20 stores the position and speed of the input radar detection data (step S19). Meanwhile, in step S18, the obstacle determination unit 19 determines that the position and speed data of the radar detection data are structures on the road surface that the vehicle can pass through, and discards the data.

[0029] The object position, speed, and recognized object type data stored in the fusion recognized object data storage unit 18 and the position and speed of the radar detection data stored in the fusion obstacle data storage unit 20 are output to the vehicle control unit 30 (step S21). The vehicle control unit 30 uses the object position, speed, and recognized object type data acquired in step S21 and the position and speed of the radar detection data in the fusion obstacle data storage unit 20 for vehicle control (step S22). This completes the object detection process. Thereafter, the object detection process proceeds to the next frame (step S23), and the same process is repeated from step S1.

[0030] According to this embodiment, the area classification unit 13 classifies the first structures, which are structures on the road surface that a vehicle can pass through, into areas around the detection position in the radar detection data in the captured image data. The obstacle determination unit 19 compares the first radar detection data, which is radar detection data that failed the identity determination process, with the area classification results. If the area classification results are not included in the position of the first radar detection data, the area classification results are determined to be obstacles to vehicle passage. If the area classification results are included in the position of the first radar detection data, the area classification results are determined to be first structures. Therefore, road cracks, manholes, gratings, and other structures on the road surface that a vehicle can pass through are prevented from being detected as obstacles, thereby reducing false detections. Furthermore, by using the radar detection data to narrow the processing range of the area classification of the camera 2, the processing time of the camera image processor 4 can be increased.

[0031] The configurations shown in the above embodiments are examples of the contents of the present disclosure, and may be combined with other known technologies, and parts of the configurations may be omitted or modified within the scope of the gist of the present disclosure.

[0032] 1 Radar, 2 Camera, 3 Radar signal processor, 4 Camera image processor, 5 Fusion processor, 6 Distance detection unit, 7 Speed ​​detection unit, 8 Horizontal angle detection unit, 9 Radar detection data storage unit, 10 Object recognition unit, 11 Deep learning feature database for object recognition, 12 Camera recognized object data storage unit, 13 Area classification unit, 14 Deep learning feature database for area classification, 15 Camera area classification data storage unit, 16 Recognized object identity determination unit, 17 Recognized object data selection unit, 18 Fusion recognized object data storage unit, 19 Obstacle determination unit, 20 Fusion obstacle data storage unit, 30 Vehicle control unit, 100 Object detection device.

Claims

1. A radar signal processor that processes a signal received from a radar to detect the position and speed of an object and output radar detection data including the position and speed of the detected object; a camera image processor that detects the position and type of the object based on image data captured by a camera; and a fusion processor that performs an identity determination process between the position of the object detected by the radar signal processor and the position of the object detected by the camera image processor and uses a result of the identity determination process for vehicle control. The camera image processor comprises an area classification unit that classifies an area of ​​a first structure, which is a structure on a road surface through which a vehicle can pass, around the detection position in the radar detection data in the captured image data, the fusion processor compares the first radar detection data, which is the radar detection data that failed the identity determination process, with the result of the area classification, and if the result of the area classification is not included in the position of the first radar detection data, determines that the result of the area classification is an obstacle passing by a vehicle, and if the result of the area classification is included in the position of the first radar detection data, determines that it is the first structure.

2. The object detection device according to claim 1, characterized in that the first structure includes a road crack, a manhole, and a grating.

Images