Object detection device

Abstract

The object detection device (100) includes a radar signal processor (3), a camera image processor (4), and a fusion processor (5) that executes an identity determination process between the position of an object detected by the radar signal processor (3) and the position of an object detected by the camera image processor (4) and uses the result of the identity determination process for vehicle control. The camera image processor (4) includes an area classification unit (13) that performs area classification of a first structure, which is a structure on a road surface through which a vehicle can pass, in the vicinity of a detection position in the radar detection data in the captured image data. The fusion processor (5) includes an obstacle determination unit (19) that compares the first radar detection data, which is radar detection data that has failed the identity determination process, with the result of the area classification, and determines the result of the area classification as an obstacle for vehicle passage if the result of the area classification is not included in the position of the first radar detection data, and determines the result of the area classification as the first structure if the result of the area classification is included in the position of the first radar detection data.

Description

[Technical field]

[0001] The present disclosure relates to an object detection device mounted on a vehicle. [Background technology]

[0002] Vehicle-mounted object detection devices have the function of quickly detecting people, structures, etc. in the vehicle's environment and controlling the vehicle and issuing warning notifications, thereby contributing to safe vehicle driving. Vehicle-mounted object detection devices use various types of sensors, such as radar, cameras, LiDAR (Light Detection and Ranging), and ultrasonic sensors, but in recent years, fusion-type object detection devices that combine multiple sensors to improve performance have become widely used.

[0003] In such a fusion-type object detection device, when the sensor is composed of a radar and a camera, since the radar is excellent at detecting position and speed, and the camera is excellent at recognizing objects, it is common to use the radar detection data for position and speed, and the camera's recognized object data for object recognition (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2010 / 119860 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, camera object recognition requires learning of features such as deep learning, and it is difficult to detect unlearned objects with camera object recognition. Therefore, radar detection is effective for detecting unlearned objects. However, radar detects road cracks, manholes, gratings, and other structures on the road surface that vehicles can pass through as obstacles, which results in false detection, resulting in a problem of reduced accuracy as an object detection device.

[0006] The present disclosure has been made in consideration of the above, and aims to provide an object detection device that can prevent structures on a road surface through which a vehicle can pass from being detected as obstacles and reduce erroneous detections. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the object detection device of the present disclosure includes a radar signal processor that processes a signal received from a radar to detect the position and speed of an object and output radar detection data including the position and speed of the detected object, a camera image processor that detects the position and type of an object based on image data captured by a camera, and a fusion processor that performs an identity determination process between the position of the object detected by the radar signal processor and the position of the object detected by the camera image processor and uses the result of the identity determination process for vehicle control. The camera image processor detects an object around a detection position in the radar detection data that has failed the identity determination process in the captured image data. Narrow the scope of area classification processing and a region classification unit that classifies a first structure, which is a structure on a road surface that a vehicle can pass through, into a region. The fusion processor compares the first radar detection data, which is the radar detection data that has failed the identification process, with the result of the region classification, and To Position of first radar detection data but If it is not included, the result of the area classification is judged as an obstacle for the vehicle to pass through, and the result of the area classification is To Position of first radar detection data but The obstacle determination unit determines that the object is a first structure if the object is included. Effect of the Invention

[0008] According to the object detection device of the present disclosure, it is possible to prevent structures on a road surface through which a vehicle can pass from being detected as an obstacle, thereby reducing erroneous detections. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of an object detection device according to an embodiment; [Diagram 2] 1 is a flowchart showing an operation procedure of an object detection device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an object detection device according to an embodiment will be described in detail with reference to the drawings.

[0011] Embodiment FIG. 1 is a block diagram showing a configuration of an object detection device 100 according to an embodiment. The object detection device 100 includes a radar 1, a camera 2, a radar signal processor 3, a camera image processor 4, and a fusion processor 5. The object detection device 100 is mounted on a vehicle (not shown). The output of the object detection device 100 is input to a vehicle control unit 30 mounted on the vehicle and is used to control the vehicle. In the following description, the object represents an object to be detected when controlling the vehicle, and corresponds to a person, a vehicle, an obstacle, etc. Examples of the obstacle include a utility pole, a wall, a guard rail, a shelf pillar, a delineator, a traffic light, etc. The obstacle corresponds to an object that has not been learned by deep learning in object recognition by the camera 2.

[0012] The radar 1 emits electromagnetic waves to an object and receives a reflected signal from the object. The radar signal processor 3 processes the received signal from the radar 1 to detect the position and speed of the object. For vehicle-mounted applications, the radar 1 generally uses the FMCW (Frequency Modulated Continuous Wave) method or the FCM (Fast Chirp Modulation) method, and is composed of high-frequency semiconductor components, power semiconductor components, substrates, crystal devices, chip components, antennas, etc.

[0013] The radar signal processor 3 generally includes an MCU (Micro Control Unit), a CPU (Central Processing Unit), etc. The radar signal processor 3 includes a distance detector 6, a speed detector 7, a horizontal angle detector 8, and a radar detection data storage unit 9. The distance detector 6, the speed detector 7, and the horizontal angle detector 8 detect the distance, speed, and horizontal angle of an object by performing Fast Fourier Transformation (FFT) in the distance direction, speed direction, and horizontal angle direction, respectively. The radar detection data storage unit 9 stores the position and speed of an object detected by the radar 1. The position and speed of an object detected by the radar 1 are called radar detection data.

[0014] The camera 2 captures a forward image and acquires image data. The camera image processor 4 detects the position and type of an object based on the image data captured by the camera 2. The image data captured by the camera 2 is also called captured image data. The camera image processor 4 also classifies the captured image data into a plurality of regions, and acquires the positions, sizes, and region types of the plurality of classified regions. The camera 2 is composed of components such as a lens, a holder, a CMOS (Complementary Metal Oxide Semiconductor) sensor, power semiconductor components, and a crystal device.

[0015] An MCU, a CPU, a GPU (Graphics Processing Unit), or the like is used as the camera image processor 4. The camera image processor 4 includes an object recognition unit 10, a deep learning feature database for object recognition 11, a camera recognized object data storage unit 12, a region classification unit 13, a deep learning feature database for region classification 14, and a camera region classification data storage unit 15.

[0016] The object recognition unit 10 refers to a deep learning feature database 11 for object recognition to recognize objects such as people and vehicles from the captured image data of the camera 2. The deep learning feature database 11 for object recognition is a database of features obtained by deep learning. The camera-recognized object data storage unit 12 stores the type and position of the recognized object obtained by the object recognition unit 10, and outputs them to the subsequent fusion processor 5. The type and position of the recognized object obtained by the object recognition unit 10 are called camera-recognized object data. In object recognition by the object recognition unit 10, a network algorithm such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector) is used.

[0017] The area classification unit 13 performs area classification of the first structure, which is a structure on the road surface through which a vehicle can pass, from the image data captured by the camera 2 based on the selection result of the recognized object data selection unit 17 of the fusion processor 5 and the deep learning feature database for area classification 14. The deep learning feature database for area classification 14 is a database of features related to structures on the road surface through which a vehicle can pass, obtained by deep learning. Examples of structures on the road surface through which a vehicle can pass include road cracks, manholes, gratings, etc. The camera area classification data storage unit 15 stores the area type, position, and size of the area for the structure through which a vehicle can pass acquired by the area classification unit 13, and outputs it to the subsequent fusion processor 5. The area type, position, and size included in the area data for the structure through which a vehicle can pass stored in the camera area classification data storage unit 15 are called camera area classification data. In the area classification by the area classification unit 13, an algorithm such as semantic segmentation is used.

[0018] The fusion processor 5 includes a recognized object identity determination unit 16 , a recognized object data selection unit 17 , a fusion recognized object data storage unit 18 , an obstacle determination unit 19 , and a fusion obstacle data storage unit 20 .

[0019] The recognized object identity determination unit 16 executes identity determination processing between the object position in the radar detection data and the recognized object position in the camera recognized object data. This identity determination processing links the position and speed detected by the radar 1 with the recognized object type detected by the camera 2. If the identity determination is successful, the recognized object data selection unit 17 transfers the position and speed detected by the radar 1 and the recognized object type detected by the camera 2 to the fusion recognized object data storage unit 18. On the other hand, if the identity determination is unsuccessful, the recognized object data selection unit 17 considers the recognized object to be an obstacle other than a person or a vehicle, and transfers the radar detection data to the area classification unit 13 and the obstacle determination unit 19.

[0020] The area classification unit 13 narrows down the processing range of area classification in the image data captured by the camera 2, using the position of the radar detection data transferred from the recognized object data selection unit 17. Generally, area classification processing imposes a high load, so this processing can speed up the area classification processing.

[0021] The obstacle determination unit 19 compares the position, size, and area type of the camera area classification data with the position of the first radar detection data, which is radar detection data that has failed to be identified and is output from the recognized object data selection unit 17, and determines whether or not the position of the radar detection data is included in the area of ​​the structure on the road surface that the vehicle can pass through. If the position of the radar detection data is not included in the area of ​​the structure on the road surface that the vehicle can pass through, the obstacle determination unit 19 transfers the position and speed of the radar detection data to the fusion obstacle data storage unit 20 and stores it. If the position of the radar detection data is included in the area of ​​the structure on the road surface that the vehicle can pass through, the obstacle determination unit 19 discards the position and speed of the radar detection data.

[0022] The data stored in the fusion recognized object data storage unit 18 and the data stored in the fusion obstacle data storage unit 20 are transferred to the vehicle control unit 30 and used for vehicle control.

[0023] 2 is a flowchart showing an operation procedure of the object detection device 100 according to the embodiment. The operation of the object detection device 100 will be described with reference to FIG.

[0024] First, when object detection processing for the current frame is started (step S1), reception data acquired by the radar 1 is input to the radar signal processor 3 (step S2). The distance detector 6, speed detector 7, and horizontal angle detector 8 of the radar signal processor 3 detect the position and speed of the object (step S3). The detected position and speed data are stored in the radar detection data storage unit 9 (step S4).

[0025] On the other hand, when object detection processing for the current frame is started (step S1), captured image data acquired by the camera 2 is input to the camera image processor 4 (step S5). The object recognition unit 10 of the camera image processor 4 recognizes an object from the captured image data acquired by referring to the deep learning feature database for object recognition 11 (step S6). The object recognition unit 10 stores the position and type of the recognized object in the camera recognized object data storage unit 12 (step S7).

[0026] The recognized object identity determination unit 16 of the fusion processor 5 executes identity determination processing between the position data by the radar 1 obtained in step S4 and the position data of the recognized object by the camera 2 (step S8). This identity determination processing links the position and speed data detected by the radar 1 with the recognized object type data detected by the camera 2. If the identity determination processing is successful (step S9: Yes), the recognized object data selection unit 17 transfers the position and speed data detected by the radar 1 stored in the radar detection data storage unit 9 and the recognized object type data stored in the camera recognized object data storage unit 12 to the fusion recognized object data storage unit 18 (step S10). The fusion recognized object data storage unit 18 stores the position and speed data detected by the radar 1 and the recognized object type data recognized by the camera 2 (step S11). The object position, speed, and recognized object type data stored in the fusion recognized object data storage unit 18 are output to the vehicle control unit 30 (step S20).

[0027] If the identity determination fails (step S9: No), the recognized object data selection unit 17 outputs the position and speed data detected by the radar 1 stored in the radar detection data storage unit 9 to the area classification unit 13 and the obstacle determination unit 19 (step S12). The area classification unit 13 performs area classification on the image data acquired by the camera 2 around the position of the radar detection data by referring to the deep learning feature database for area classification 14, thereby classifying structures on the road surface that can be passed by a vehicle (step S13). The area classification unit 13 stores the position, size, and area type of the area-classified data in the camera area classification data storage unit 15 (step S14).

[0028] Thereafter, the obstacle determination unit 19 of the fusion processor 5 compares the position, size, and area type of the camera area classification data with the position of the radar detection data that has failed to be identified and is output from the recognized object data selection unit 17, and determines whether or not the position of the radar detection data is included in an area of ​​a road surface structure that the vehicle can pass through, such as a crack on the road surface, a manhole, or a grating (step S15). In this determination process, if the position of the radar detection data is not included in an area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination is determined to be successful (step S16: Yes), and the procedure moves to step S17. Also, if the position of the radar detection data is included in an area of ​​a structure on the road surface that the vehicle can pass through, the obstacle determination is determined to be unsuccessful (step S16: No), and the procedure moves to step S18. In step S17, the obstacle determination unit 19 outputs the position and speed of the radar detection data to the fusion obstacle data storage unit 20. The fusion obstacle data storage unit 20 stores the position and speed of the input radar detection data (step S19). Meanwhile, in step S18, the obstacle determination unit 19 determines that the position and speed data of the radar detection data are structures on the road surface that the vehicle can pass through, and discards the data.

[0029] The object position, speed, and recognized object type data stored in the fusion recognized object data storage unit 18, and the position and speed of the radar detection data stored in the fusion obstacle data storage unit 20 are output to the vehicle control unit 30 (step S21). The vehicle control unit 30 uses the object position, speed, and recognized object type data acquired in step S21, and the position and speed of the radar detection data in the fusion obstacle data storage unit 20 for vehicle control (step S22). This ends the object detection process. Thereafter, the object detection process proceeds to the next frame (step S23), and the same process is repeatedly executed from step S1.

[0030] Thus, according to the embodiment, the area classification unit 13 classifies the first structure, which is a structure on the road surface where a vehicle can pass, into an area around the detection position in the radar detection data in the captured image data, and the obstacle determination unit 19 compares the first radar detection data, which is the radar detection data that failed the identity determination process, with the area classification result, and if the area classification result is not included in the position of the first radar detection data, it determines that the area classification result is an obstacle to the vehicle passing, and if the area classification result is included in the position of the first radar detection data, it determines that it is the first structure. Therefore, road cracks, manholes, gratings, etc., which are structures on the road surface where a vehicle can pass, are prevented from being detected as obstacles, and false detections can be reduced. In addition, the processing range of the area classification of the camera 2 is narrowed down using the radar detection data, so that the processing time of the camera image processor 4 can be increased.

[0031] The configurations shown in the above embodiments are examples of the contents of the present disclosure, and may be combined with other known technologies, and parts of the configurations may be omitted or modified without departing from the gist of the present disclosure. [Explanation of symbols]

[0032] 1 radar, 2 camera, 3 radar signal processor, 4 camera image processor, 5 fusion processor, 6 distance detection unit, 7 speed detection unit, 8 horizontal angle detection unit, 9 radar detection data storage unit, 10 object recognition unit, 11 deep learning feature database for object recognition, 12 camera recognized object data storage unit, 13 area classification unit, 14 deep learning feature database for area classification, 15 camera area classification data storage unit, 16 recognized object identity determination unit, 17 recognized object data selection unit, 18 fusion recognized object data storage unit, 19 obstacle determination unit, 20 fusion obstacle data storage unit, 30 vehicle control unit, 100 object detection device.

Claims

1. A radar signal processor that processes a received signal from a radar to detect the position and velocity of an object and outputs radar detection data including the detected position and velocity of the object; A camera image processor that detects the position and type of the object based on the captured image data of the camera; A fusion processor that performs an identity determination process between the position of the object detected by the radar signal processor and the position of the object detected by the camera image processor, and uses the result of the identity determination process for vehicle control; Comprising: The camera image processor includes: A region classification unit that classifies a region of a first structure, which is a structure on a road surface through which the vehicle can pass, in the vicinity of the detection position in the radar detection data that has failed in the identity determination process in the captured image data; The fusion processor collates the first radar detection data, which is the radar detection data that has failed in the identity determination process, with the result of the region classification. When the result of the region classification is not included in the position of the first radar detection data, the fusion processor determines the result of the region classification as an obstacle to vehicle passage. When the result of the region classification is included in the position of the first radar detection data, the fusion processor includes an obstacle determination unit that determines that it is the first structure; An object detection device characterized by the above.

2. The first structure includes road surface cracks, manholes, and gratings The object detection device according to claim 1, characterized by the above.

Images