Obstacle detection method, system, electronic device and storage medium for matching fusion

Abstract

The application provides a matching fusion obstacle detection method and system, electronic equipment and storage medium, the method comprises the following steps: according to the joint calibration of millimeter wave radar and camera, the environmental point cloud data is projected to the plane where the camera image is located; according to the preset safe driving area, the radar target data is screened out from the point cloud data; the image data is obtained through the camera target data by detecting the target algorithm; the radar target data and the camera target data are matched with the target matching strategy, the radar target data and the camera target data belonging to the same target obstacle are data fusion; the target information corresponding to the same target obstacle detected by the millimeter wave radar and the camera is output, and the target information of the millimeter wave radar and the camera meeting the credibility condition is output. The application fully develops the matching fusion detection advantages of millimeter wave radar and camera sensors, and improves the accuracy and integrity of obstacle detection in the road in front of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent driving environment perception, and particularly relates to a matching fusion obstacle detection method and system, an electronic device and a storage medium. BACKGROUND

[0002] Environment perception is a key link in intelligent driving technology, and obtaining richer and more accurate target information is the main task of environment perception research. Since each single sensor has its own detection limitations, it is difficult to obtain comprehensive target information relying on only one sensor, and multi-sensor fusion detection technology can solve this problem. Among many vehicle-mounted sensors, millimeter wave radar has advantages in target information detection, and camera has advantages in target classification and contour detection; therefore, data fusion of millimeter wave radar and camera can complement the detection performance of the two types of sensors, so as to obtain more comprehensive and accurate information of the detected target, and also reduce the false detection and missed detection of each single sensor.

[0003] At present, the millimeter wave radar and camera fusion detection methods mainly include two types: data level fusion and decision level fusion. Data level fusion can be further divided into two types: one is to project the target points detected by the millimeter wave radar on the image plane at the same time, so as to form an image target detection region of interest, input the region to the camera target detection module, and obtain target classification and contour information; this method can reduce the time consumption of the camera detection module, but the comprehensiveness of the detection completely depends on the millimeter wave radar, and cannot avoid the missed detection of the millimeter wave radar. The other is to separately perform the millimeter wave radar detection module and the camera detection module, and then match and fuse the target data detected by the two types of sensors; this method can fully utilize the target information of the two types of sensors, and can improve the accuracy and integrity of the detection; however, the target matching and fusion performance of the two types of sensors in this method is poor. Decision level fusion refers to weighting fusion of the driving behavior decisions made by the detection modules of the two types of sensors for the detected targets, so as to improve the accuracy of the decision; however, this method is less studied in the prior art.

[0004] Therefore, how to improve the target matching and fusion performance of the millimeter wave radar and camera detection data, so as to improve the accuracy and integrity of the obstacle detection in the road in front of the vehicle, is an urgent problem to be solved. SUMMARY

[0005] In order to solve the above technical problems, the present application provides a matching fusion obstacle detection method and system, an electronic device and a storage medium, which take the advantages of the detection performance of the millimeter wave radar and camera two types of sensors, fully utilize the advantages of multi-sensor matching and fusion detection, and improve the accuracy and integrity of the obstacle detection in the road in front of the vehicle.

[0006] In a first aspect, the application provides a matching fusion obstacle detection method, comprising:

[0007] acquiring point cloud data of a road environment based on a millimeter wave radar and acquiring image data of the road environment based on a camera, wherein the image data comprises camera images when a vehicle is driving;

[0008] projecting the environment point cloud data onto a plane where the camera image is located according to joint calibration of the millimeter wave radar and the camera;

[0009] screening radar target data from the point cloud data according to a preset safe driving area;

[0010] obtaining camera target data by detecting the image data through a target detection algorithm, wherein the camera target data comprises a longitudinal distance of the vehicle relative to a front obstacle;

[0011] performing same-target matching on the radar target data and the camera target data by using a target matching strategy, and performing data fusion on the radar target data and the camera target data belonging to the same target obstacle;

[0012] outputting target information detected by the millimeter wave radar and the camera corresponding to the same target obstacle and target information of the millimeter wave radar and the camera meeting a credibility condition.

[0013] Preferably, the step of projecting the environment point cloud data onto the plane where the camera image is located according to the joint calibration of the millimeter wave radar and the camera specifically comprises:

[0014] converting three-dimensional coordinates [X r , Y r , Z r ] of the point cloud data into coordinate values in a vehicle coordinate system through a first preset matrix, wherein the first preset matrix is specifically as follows:

[0015]

[0016] In the formula, [X v , Y v , Z v ] represents coordinate values in the vehicle coordinate system, R v2r represents a rotation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system, and T v2r represents a translation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system.

[0017] The coordinate value in the vehicle coordinate system is converted into a coordinate value in the camera coordinate system through a second preset matrix, and the second preset matrix is specifically as follows:

[0018]

[0019] In the formula, [X c , Y c , Z c ] represents the coordinate value in the camera coordinate system, R v2c represents the rotation matrix between the vehicle coordinate system and the camera coordinate system, and T v2c represents the translation matrix between the vehicle coordinate system and the camera coordinate system.

[0020] The coordinate value in the camera coordinate system is converted into a coordinate value in the pixel coordinate system through a third preset matrix, and the third preset matrix is specifically as follows:

[0021]

[0022] In the formula, [u, v, w] represents the coordinate value in the pixel coordinate system in homogeneous form, R in represents the internal parameter matrix of the camera.

[0023] The coordinate value in the pixel coordinate system is normalized to obtain the pixel coordinate value of the point cloud data projected onto the plane on which the camera image is located, and the normalization process is specifically as follows:

[0024]

[0025] In the formula, u' represents the width value of the point cloud data projection, and v' represents the height value of the point cloud data projection.

[0026] Preferably, the preset safe driving area refers to a set area with a lateral distance of 10.5 m on the left and right sides and a longitudinal distance of 80 m.

[0027] Preferably, the camera target data obtained by detecting the target algorithm from the image data includes the step of calculating the longitudinal distance of the vehicle relative to the front obstacle, and the step specifically includes:

[0028] The image data is input into a trained YOLOv4 model for model inference, so as to identify the type of the obstacle on the camera image and calculate the position data of the obstacle on the camera image.

[0029] The position data is estimated through a monocular distance measurement model to obtain camera target data, and the monocular distance measurement model is specifically as follows:

[0030]

[0031] wherein Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera principal axis from the image plane y direction, p c represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the camera principal axis installation height relative to the ground, and f represents the camera focal length.

[0032] Preferably, the step of matching the radar target data and the camera target data to the same target using a target matching strategy and fusing the radar target data and the camera target data belonging to the same target obstacle includes the following steps:

[0033] Screening based on the radar target data and the camera target data respectively setting the confidence conditions, projecting the screened millimeter wave radar target points to the plane of the camera image at the same time;

[0034] Screening all millimeter wave radar target points existing in a camera target detection frame, and judging whether the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than a preset threshold value;

[0035] If yes, it is determined that the target points detected by the millimeter wave radar and the camera are the same target, and the radar target data and the camera target data are fused;

[0036] If no, the radar target data and the camera target data are not processed.

[0037] Preferably, the confidence condition of the radar target data is that the target appearance frequency is greater than 2 times, and the confidence condition of the camera target data is that the target category probability is greater than 75%.

[0038] Preferably, the target information includes one or more of the data detected by the millimeter wave radar, the data detected by the camera, and the data detected by both the millimeter wave radar and the camera.

[0039] If the target information includes the data detected by both the millimeter wave radar and the camera, the longitudinal distance completely uses the data detected by the millimeter wave radar;

[0040] If the target information includes the data detected by the millimeter wave radar, whether the radar data point continuation frequency or the longitudinal distance satisfies the set condition is judged to determine whether the millimeter wave radar target data is outputted.

[0041] If the target information includes the data detected by the camera, whether the camera target data is output is determined according to whether the target category probability is greater than a set condition.

[0042] In a second aspect, the application provides a matching and fusion obstacle detection system, comprising:

[0043] An acquisition module is configured to acquire point cloud data of a road environment based on a millimeter wave radar and acquire image data of the road environment based on a camera, wherein the image data includes camera images when a vehicle is driving.

[0044] A calibration module is configured to project the environment point cloud data onto a plane on which the camera images are located according to joint calibration of the millimeter wave radar and the camera.

[0045] A screening module is configured to screen radar target data from the point cloud data according to a preset safe driving area.

[0046] An algorithm module is configured to obtain camera target data through a detection target algorithm based on the image data, wherein the camera target data includes a longitudinal distance of the vehicle relative to a front obstacle.

[0047] A matching module is configured to perform same-target matching on the radar target data and the camera target data using a target matching strategy and perform data fusion on the radar target data and the camera target data belonging to the same target obstacle.

[0048] An output module is configured to output target information detected by the millimeter wave radar and the camera corresponding to the same target obstacle and target information of the millimeter wave radar and the camera that meets a credibility condition.

[0049] Preferably, the calibration module comprises:

[0050] A first conversion unit is configured to convert three-dimensional coordinates [X r , Y r , Z r ] of the point cloud data into coordinate values in a vehicle coordinate system through a first preset matrix, wherein the first preset matrix is specifically as follows:

[0051]

[0052] In the formula, [X v , Y v , Z v ] represents the coordinate values in the vehicle coordinate system, R v2r represents a rotation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system, and T v2ra translation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system;

[0053] a second conversion unit configured to convert the coordinate value in the vehicle coordinate system into a coordinate value in a camera coordinate system by using a second preset matrix, wherein the second preset matrix is specifically as follows:

[0054]

[0055] wherein [X c , Y c , Z c ] represents the coordinate value in the camera coordinate system, R v2c represents a rotation matrix between the vehicle coordinate system and the camera coordinate system, and T v2c represents a translation matrix between the vehicle coordinate system and the camera coordinate system;

[0056] a third conversion unit configured to convert the coordinate value in the camera coordinate system into a coordinate value in a pixel coordinate system by using a third preset matrix, wherein the third preset matrix is specifically as follows:

[0057]

[0058] wherein [u, v, w] represents the coordinate value in the pixel coordinate system in a homogeneous form, and v' represents an intrinsic parameter matrix of the camera;

[0059] a normalization unit configured to perform normalization processing on the coordinate value in the pixel coordinate system to obtain a pixel coordinate value of the point cloud data projected onto a plane on which the camera image is located, wherein the normalization processing is specifically as follows:

[0060]

[0061] wherein u' represents a width value of the point cloud data projection, and v' represents a height value of the point cloud data projection.

[0062] Preferably, the algorithm module comprises:

[0063] a recognition unit configured to input the image data into a trained YOLOv4 model for model inference, so as to recognize a type of the obstacle on the camera image and calculate position data of the obstacle on the camera image;

[0064] an estimation unit configured to estimate camera target data from the position data by using a monocular distance measurement model, wherein the monocular distance measurement model is specifically as follows:

[0065]

[0066] In the formula, Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera main axis relative to the image plane y direction, p c represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the camera main axis installation height relative to the ground, and f represents the camera focal length.

[0067] Preferably, the matching module comprises:

[0068] The screening unit is configured to screen based on the respective confidence conditions of the radar target data and the camera target data, and project the screened millimeter wave radar target point to the plane of the camera image at the same time.

[0069] The judgment unit is configured to screen all millimeter wave radar target points existing in a camera target detection frame, and judge whether the longitudinal distance error of the millimeter wave radar target point and the camera target detection frame is less than a preset threshold.

[0070] The fusion unit is configured to determine that the target points detected by the millimeter wave radar and the camera are the same target if the longitudinal distance error of the millimeter wave radar target point and the camera target detection frame is less than the preset threshold, and perform data fusion on the radar target data and the camera target data.

[0071] The non-processing unit is configured to not process the radar target data and the camera target data if the longitudinal distance error of the millimeter wave radar target point and the camera target detection frame is not less than the preset threshold.

[0072] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the matching and fusion obstacle detection method according to the first aspect when executing the computer program.

[0073] In a fourth aspect, a storage medium is provided, which stores a computer program executable by a processor to implement the matching and fusion obstacle detection method according to the first aspect.

[0074] Compared with the prior art, the matching and fusion obstacle detection method, system, electronic device, and storage medium provided by the present application have the following advantages:

[0075] 1. The camera target detection categories used in the present application can identify more abundant obstacle categories, and all of them are common obstacles when the vehicle is driving on a normal road. Identifying more obstacle categories can avoid the situation that the front obstacle is missed when the vehicle is driving.

[0076] 2、The target matching strategy provided by the application, under the condition that the millimeter wave radar projection point is in the camera target detection frame and the longitudinal distance error measured by the two sensors is calculated, and the corresponding processing is carried out for complex situations, can better guarantee the accuracy of the target matching of the two sensors.

[0077] 3、The application can fully utilize the detection performance of the two sensors by comprehensively outputting the target data detected by the two sensors, including the target data detected by only a single sensor and the target data detected by both sensors, and improve the accuracy and integrity of the obstacle detection in the road in front of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0079] Figure 1 The flow chart of the matching and fusion obstacle detection method provided for embodiment 1 of the application;

[0080] Figure 2 The schematic diagram of the millimeter wave radar projection to the image plane provided for embodiment 1 of the application;

[0081] Figure 3 The flow chart of the matching strategy provided for embodiment 1 of the application;

[0082] Figure 4 The specific flow chart of step S150 of the matching and fusion obstacle detection method provided for embodiment 1 of the application;

[0083] Figure 5a 、 Figure 5b The result comparison diagram of the fusion detection effect provided for embodiment 1 of the application;

[0084] Figure 6 The structural block diagram of the matching and fusion obstacle detection system corresponding to the method of embodiment 1 provided by embodiment 2 of the application;

[0085] Figure 7 The specific flow chart of step S250 of the matching and fusion obstacle detection method provided for embodiment 3 of the application;

[0086] Figure 8 The structural block diagram of the matching module in the matching and fusion obstacle detection system corresponding to the method of embodiment 3 provided by embodiment 4 of the application;

[0087] Figure 9is a schematic diagram of the hardware structure of the device provided in Embodiment 5 of the present application.

[0088] The drawing identification is explained as follows:

[0089] 10 - acquisition module;

[0090] 20 - calibration module, 21 - first conversion unit, 22 - second conversion unit, 23 - third conversion unit, 24 - normalization unit;

[0091] 30 - screening module;

[0092] 40 - algorithm module, 41 - identification unit, 42 - estimation unit;

[0093] 50 - matching module, 51 - screening unit, 52 - judgment unit, 53 - fusion unit, 54 - non-processing unit;

[0094] 60 - output module;

[0095] 70 - bus, 71 - processor, 72 - memory, 73 - communication interface. DETAILED DESCRIPTION

[0096] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art.

[0097] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the implementations can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, and operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

[0098] The block diagrams in the drawings show only the functionality of the example implementations and do not imply any particular physical or architectural arrangement of the example implementations. No inference should be made regarding the architecture or configuration of a hardware implementation from the description of the example implementations provided herein. Also, the functionality provided by the example implementations can be split into additional components not explicitly described, for example, physical components. Additionally, the examples can be performed on electrical components, software routines, and / or any combination thereof. It is further noted that the example implementations can be carried out on physical computers, hardware chips, and / or any combination thereof.

[0099] The flowchart shown in the drawing is only an exemplary illustration, and is not necessarily required to include all contents and operations / steps, nor is it necessarily required to be executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0100] Embodiment 1

[0101] Specifically, Figure 1 A flowchart of a matching fusion obstacle detection method provided by the embodiment is shown.

[0102] As Figure 1 The matching fusion obstacle detection method of the embodiment includes the following steps:

[0103] S110, obtaining point cloud data of a road environment based on a millimeter wave radar and obtaining image data of the road environment based on a camera.

[0104] The millimeter wave radar point cloud data includes a plurality of scanning points containing three-dimensional coordinate information and motion information, and the image data includes a camera image when the vehicle is driving.

[0105] Specifically, the millimeter wave radar point cloud data is an obstacle point cloud data set obtained by a millimeter wave radar device emitting millimeter waves for spatial detection, and each point contains information such as the distance, speed, angle, and radar cross section (RCS) of the target. In the specific implementation process, the millimeter wave radar point cloud data is obtained by the millimeter wave radar device emitting electromagnetic wave signals with a wavelength of 1-10 mm outward, then receiving the electromagnetic wave signals reflected by the obstacles, and then obtaining the spatial motion information of the point cloud through data processing and operation. In addition, the camera can be used to realize a variety of functions, and gradually evolves according to the development law of autonomous driving. The vehicle-mounted camera has some advantages over other perception sensors, because the camera has a higher resolution than other sensors, can obtain sufficient environmental details, and help the vehicle to recognize the environment; the vehicle-mounted camera can depict the appearance and shape of objects, read signs, and other functions that other sensors cannot achieve; based on the camera, the road state and characteristics can be sensed in advance to provide effective information for the vehicle planning control system, thereby improving the safety and driving comfort of the vehicle; using visual information for road preview has been proven to be an effective solution.

[0106] S120, projecting the environment point cloud data onto the plane where the camera image is located according to the joint calibration of the millimeter wave radar and the camera.

[0107] Specifically, the purpose of the spatial fusion of the millimeter wave radar and the camera is to correspond the objects in the three-dimensional world detected by the millimeter wave radar with the objects in the image detected by the camera. The millimeter wave radar and the camera are sensors in different coordinate systems, so to realize the spatial fusion of the millimeter wave radar and the camera, a conversion model of the coordinate systems of the two sensors must be established. The embodiment specifically relates to four coordinate systems: a millimeter wave radar coordinate system, a vehicle coordinate system, a camera coordinate system, and a coordinate value-pixel coordinate system. The specific effect is shown in Figure 2 .

[0108] Further, the specific steps of step S120 include:

[0109] S121, converting the three-dimensional coordinates [X r , Y r , Z r ] of the point cloud data into coordinate values in the vehicle coordinate system through a first preset matrix, wherein the first preset matrix is specifically as follows:

[0110]

[0111] In the formula, [X v , Y v , Z v ] represents the coordinate values in the vehicle coordinate system, R v2r represents a rotation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system, and T v2r represents a translation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system.

[0112] Specifically, the three-dimensional coordinates detected by the millimeter wave radar are first converted to the vehicle coordinate system. Since the rotation matrix R v2r is used for rotating the vehicle coordinate system to the millimeter wave radar coordinate system, when the rotation matrix is used to convert from the millimeter wave radar coordinate system to the vehicle coordinate system, the inverse matrix of the rotation matrix is used to realize the conversion.

[0113] S122, converting the coordinate values in the vehicle coordinate system into coordinate values in the camera coordinate system through a second preset matrix, wherein the second preset matrix is specifically as follows:

[0114]

[0115] In the formula, [X c , Y c , Z c ] represents the coordinate values in the camera coordinate system, R v2c represents a rotation matrix between the vehicle coordinate system and the camera coordinate system, and T v2c represents a translation matrix between the vehicle coordinate system and the camera coordinate system.

[0116] Specifically, when the vehicle coordinate system is converted into the camera coordinate system, the translation matrix needs to be calculated first, otherwise the values in the translation matrix will also be affected by the rotation matrix, thereby affecting the projection result.

[0117] S123, convert the coordinate value in the camera coordinate system into a coordinate value in the pixel coordinate system through a third preset matrix, wherein the third preset matrix is specifically as follows:

[0118]

[0119] In the formula, [u, v, w] represents the coordinate value in the pixel coordinate system in homogeneous form, R in represents the intrinsic matrix of the camera.

[0120] Specifically, the millimeter wave radar data is converted into the camera coordinate, and then converted from the camera coordinate system to the pixel coordinate, which requires the intrinsic matrix R in of the camera. The most commonly used method for calculating the intrinsic matrix of the camera is the Zhang Zhengyou calibration method, which is a linear calibration method for a nonlinear model camera. A two-dimensional plane target is used to collect multiple different viewpoints of images to realize the calibration of the camera. The camera is calibrated by selecting the Zhang Zhengyou calibration method to obtain the relevant parameters. The Zhang Zhengyou calibration method has the characteristics of simple operation, strong adaptability and high calibration precision.

[0121] S124, performing normalization processing on the coordinate value in the pixel coordinate system to obtain the pixel coordinate value of the point cloud data projected onto the plane where the camera image is located, wherein the normalization processing is specifically as follows:

[0122]

[0123] In the formula, u' represents the width value of the point cloud data projection, and v' represents the height value of the point cloud data projection.

[0124] Specifically, [u, v, w] in the normalization processing is the homogeneous representation of the pixel coordinate, and the pixel coordinate also needs to be normalized to obtain a two-dimensional plane, which is the pixel coordinate value after the millimeter wave radar projection. The normalization processing can further improve the detection accuracy of the obstacle.

[0125] S130, according to the preset safe driving area, screen out the radar target data from the point cloud data.

[0126] The preset safe driving area refers to a set area with a lateral distance of 10.5 m on the left and right sides and a longitudinal distance of 80 m. The preset safe driving area of the embodiment is set according to big data of road safety, and of course the range set according to specific road conditions is different. Through the setting of the preset safe driving area, the purpose is to further screen the point cloud data of the millimeter wave radar and reduce the data operation amount.

[0127] S140, obtaining camera target data of the image data through a detection target algorithm, wherein the camera target data includes a longitudinal distance of the vehicle relative to an obstacle in front of the vehicle.

[0128] Specifically, the embodiment sets the condition that the millimeter wave radar projection point is in the camera target detection frame and calculates the longitudinal distance error measured by the two sensors, and correspondingly processes the complex situation, which can better guarantee the accuracy of the target matching of the two sensors.

[0129] Further, the specific steps of step S140 include:

[0130] S141, inputting the image data into a trained YOLOv4 model for model inference, so as to identify the type of the obstacle on the camera image and calculate the position data of the obstacle on the camera image.

[0131] In the structure of YOLOv4, the backbone network is selected as CSPDarkent53, and the spatial pyramid pooling module (SPPNet) is added to the backbone network. The function of SPPNet is to change the input of CNN from a fixed size to an arbitrary size. In order to better utilize the features extracted by the backbone network, PANet is added in the middle of the network. Through the enhancement of the path from the bottom to the top, the accuracy of the low-level positioning signal is utilized to enhance the hierarchy of the entire feature, so as to shorten the information path between the low-level and top-level features. Finally, the head model of the network is consistent with YOLOv3.

[0132] Specifically, the camera used in the embodiment can detect up to 7 types of targets (sedan, truck, bus, bicycle, electric vehicle, pedestrian, and triangular cone barrel), can identify more types of obstacles, and all are common obstacles when the vehicle is driving on a normal road. Identifying more types of obstacles can avoid the situation that the front obstacle is missed when the vehicle is driving.

[0133] S142, estimating the camera target data through a monocular distance measurement model, wherein the monocular distance measurement model is as follows:

[0134]

[0135] In the formula, Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera principal axis from the image plane y direction, p c represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the camera principal axis installation height relative to the ground, and f represents the camera focal length.

[0136] Specifically, the monocular distance measurement model utilizes camera imaging principles for modeling and combines deep learning model inference results for calculation, and has good real-time performance. The distance measurement error of this method is maintained within 2 m at a distance of 70 m.

[0137] In S150, the radar target data and the camera target data are matched with the same target using a target matching strategy, and the radar target data and the camera target data belonging to the same target obstacle are fused.

[0138] The target matching strategy of the embodiment has a specific process as shown in Figure 3 In addition, the data fusion of the millimeter wave radar and the camera mainly refers to the obstacle information processed by both at the same time, i.e., the synchronization of the sensors in time. Since the sampling frequencies of different sensors are not the same, the sensors do not collect information at the same time, and therefore, the data collected by both are not at the same time. In this embodiment, the camera sensor with a lower working frequency is taken as the reference, and the information of the two sensors is synchronously sampled in a multi-threaded manner to realize the fusion of the data of both in time. Specifically, the camera collects the information obtained at the current time after being triggered by the camera time, and then triggers the collection thread of the millimeter wave radar. Similarly, after being triggered, the millimeter wave radar collects the information detected at the current time, and then combines the image data of the camera and the millimeter wave radar at the current time and adds them to the tail of the buffer queue for processing in the main thread of the data processing.

[0139] Further, as shown in Figure 4 the specific steps of S150 include:

[0140] In S151, the radar target data and the camera target data are filtered based on the respective confidence conditions, and the filtered millimeter wave radar target points are projected onto the plane of the image at the same time.

[0141] Specifically, in order to ensure the success rate of the target matching strategy, the millimeter wave radar target data and the camera target data are set with respective confidence conditions. Specifically, the number of occurrences of the millimeter wave radar target is greater than 2, and the probability of the camera target category is greater than 75%, and the target points detected by the filtered millimeter wave radar are projected onto the image plane at the same time.

[0142] S152, screen all millimeter wave radar target points existing in a camera target detection frame of the millimeter wave radar, and determine whether a longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than a preset threshold.

[0143] Specifically, it is determined whether there are multiple millimeter wave radar target points in a camera target detection frame and a single millimeter wave radar target point in multiple camera target detection frames. When the first case exists, the longitudinal distance error of the two sensors is used to screen the wrong millimeter wave radar target point. According to the accuracy of the monocular ranging model, when in the same lane, the error is less than 20% of the camera target distance, and when in different lanes, the error is less than 30% of the camera target distance. If there are still multiple millimeter wave radar target points, the point with the smallest longitudinal distance is removed to ensure the safe driving distance. When the second case exists, the maximum pixel value in the longitudinal direction of the two camera target detection frames is compared to determine the distance between the two targets. According to the principle that the millimeter wave radar cannot detect the occluded object, the millimeter wave radar data is associated with the target detection frame that is closer.

[0144] S153, if yes, it is determined that the target points detected by the millimeter wave radar and the camera are the same target, and the radar target data and the camera target data are fused.

[0145] Specifically, the longitudinal distance error between the millimeter wave radar target in a certain camera target detection frame and the target frame is determined. When in the same lane, the error is less than 20% of the camera target distance, and when in different lanes, the error is less than 30% of the camera target distance. When the condition is met, it is considered that the two are the same target.

[0146] S160, output the target information of the target corresponding to the millimeter wave radar and the camera detected by the millimeter wave radar and the camera, and the target information of the millimeter wave radar and the camera that meets the credibility condition.

[0147] In the embodiment, the target data detected by the two sensors is comprehensively output, including the target data detected by only a single sensor and the target data detected by both sensors, so that the detection performance of the two sensors can be fully utilized, and the accuracy and integrity of the obstacle detection in the front road of the vehicle can be improved. Specifically, the target information includes one or more of the data detected by the millimeter wave radar, the data detected by the camera, and the data detected by both the millimeter wave radar and the camera. The specific conditions of the embodiment are as follows:

[0148] If the target information includes data detected by both the millimeter-wave radar and the camera, then its longitudinal distance is entirely based on the data detected by the millimeter-wave radar. In practice, the successfully fused data from the millimeter-wave radar and camera should be used as the system output. The fused target information is more complete and reliable; it includes information such as radar distance and velocity as well as camera target category and image location, specifically as follows: Figure 5b RC4:Car:1.00 in the example.

[0149] If the target information includes data detected by the millimeter-wave radar, then the output of the millimeter-wave radar target data is determined based on whether the number of times the radar data point continues or the longitudinal distance meets a set condition. Specifically, when only millimeter-wave radar data is available, the output of the millimeter-wave radar target data is determined based on whether the number of times the radar data point continues or the longitudinal distance meets a set threshold condition. In practice, factors such as lighting and training data can affect the camera, potentially causing some targets to go undetected. However, if the millimeter-wave radar detects this target data and the data meets the confidence criteria, it should still be output by the system. Specifically, for example... Figure 5b The fourth car from the left in the image is shown.

[0150] If the target information includes data detected by the camera, then the output of the camera target data is determined based on whether the probability of the target category is greater than a set condition. Specifically, when only camera data is available, the output of the camera target information is determined based on whether the probability of the target category is greater than 75%. In practice, millimeter-wave radar has poor pedestrian detection performance, which can easily lead to missed detections, but cameras can clearly detect pedestrian targets. In this case, the pedestrian target information cannot be deleted. Specifically, as follows... Figure 5b The C1 target in the image should not be considered as system output. Furthermore, due to the millimeter-wave radar's installation height and target obstruction, the radar may be unable to detect the vehicle, but the camera can. This too should be considered as system output. Specifically, as shown below... Figure 5b As shown in C6:car:0.72.

[0151] Example 2

[0152] This embodiment provides a structural block diagram of a system corresponding to the method described in Embodiment 1. Figure 6 This is a structural block diagram of the obstacle detection system based on the matching and fusion method of this embodiment, as shown below. Figure 6 As shown, the system includes:

[0153] The acquisition module 10 is used to acquire point cloud data of the road environment based on millimeter-wave radar and image data of the road environment based on a camera, wherein the image data includes camera images of the vehicle in motion.

[0154] A calibration module 20 is configured to project the point cloud data onto a plane on which the camera image is located according to joint calibration of the millimeter wave radar and the camera.

[0155] A screening module 30 is configured to screen radar target data from the point cloud data according to a preset safe driving area, wherein the preset safe driving area refers to a set area with a lateral distance of 10.5 m on the left and right sides and a longitudinal distance of 80 m.

[0156] An algorithm module 40 is configured to obtain camera target data through a detection target algorithm, wherein the camera target data includes a longitudinal distance of the vehicle relative to a front obstacle.

[0157] A matching module 50 is configured to perform target matching on the radar target data and the camera target data using a target matching strategy, and perform data fusion on the radar target data and the camera target data belonging to the same target obstacle.

[0158] An output module 60 is configured to output target information of the same target obstacle detected by the millimeter wave radar and the camera and target information of the millimeter wave radar and the camera meeting a credibility condition, wherein the target information includes one or more of data detected by the millimeter wave radar, data detected by the camera, and data detected by both the millimeter wave radar and the camera.

[0159] If the target information includes data detected by both the millimeter wave radar and the camera, a longitudinal distance thereof is completely determined by data detected by the millimeter wave radar.

[0160] If the target information includes data detected by the millimeter wave radar, whether the millimeter wave radar target data is output is determined according to whether a radar data point continuation number or a longitudinal distance meets a set condition.

[0161] If the target information includes data detected by the camera, whether the camera target data is output is determined according to whether a target category probability is greater than a set condition.

[0162] Further, the calibration module 20 includes:

[0163] A first conversion unit 21 is configured to convert three-dimensional coordinates [X r , Y r , Z r ] of the point cloud data into coordinate values in a vehicle coordinate system through a first preset matrix, wherein the first preset matrix is specifically as follows:

[0164]

[0165] wherein, [X v , Y v , Z v ] represents the coordinate value in the vehicle coordinate system, R v2r represents the rotation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system, T v2r represents the translation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system;

[0166] The second conversion unit 22 is configured to convert the coordinate value in the vehicle coordinate system into a coordinate value in a camera coordinate system by using a second preset matrix, wherein the second preset matrix is specifically as follows:

[0167]

[0168] wherein, [X c , Y c , Z c ] represents the coordinate value in the camera coordinate system, R v2c represents the rotation matrix between the vehicle coordinate system and the camera coordinate system, T v2c represents the translation matrix between the vehicle coordinate system and the camera coordinate system;

[0169] The third conversion unit 23 is configured to convert the coordinate value in the camera coordinate system into a coordinate value in a pixel coordinate system by using a third preset matrix, wherein the third preset matrix is specifically as follows:

[0170]

[0171] wherein, [u, v, w] represents the coordinate value in the pixel coordinate system in the homogeneous form, R in represents the intrinsic parameter matrix of the camera;

[0172] The normalization unit 24 is configured to perform normalization processing on the coordinate value in the pixel coordinate system to obtain a pixel coordinate value of the point cloud data projected onto the plane on which the camera image is located, wherein the normalization processing is specifically as follows:

[0173]

[0174] wherein, u' represents the width value of the point cloud data projection, and v' represents the height value of the point cloud data projection.

[0175] Further, the algorithm module 40 comprises:

[0176] The identification unit 41 is configured to input the image data into a trained YOLOv4 model for model inference, so as to identify the type of the obstacle in the camera image and calculate the position data of the obstacle in the camera image.

[0177] The estimation unit 42 is configured to estimate camera target data from the position data by using a monocular distance measurement model, wherein the monocular distance measurement model is specifically as follows:

[0178]

[0179] In the formula, Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera main shaft relative to the y direction of the image plane, p represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the installation height of the camera main shaft relative to the ground, and f represents the camera focal length. c In the formula, Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera main shaft relative to the y direction of the image plane, p represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the installation height of the camera main shaft relative to the ground, and f represents the camera focal length.

[0180] Further, the matching module 50 comprises:

[0181] The screening unit 51 is configured to screen the radar target data and the camera target data based on the respective confidence conditions, and project the screened millimeter wave radar target point to the plane of the camera image at the same time; wherein the confidence condition of the radar target data is that the target appearance frequency is greater than 2 times, and the confidence condition of the camera target data is that the target category probability is greater than 75%.

[0182] The judgment unit 52 is configured to screen all millimeter wave radar target points existing in a camera target detection frame, and judge whether the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than a preset threshold value.

[0183] The fusion unit 53 is configured to determine that the target points detected by the millimeter wave radar and the camera are the same target if the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than the preset threshold value, and fuse the radar target data and the camera target data.

[0184] The non-processing unit 54 is configured to not process the radar target data and the camera target data if the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is not less than the preset threshold value.

[0185] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can be located in different processors in any combination.

[0186] Embodiment 3

[0187] The difference between this embodiment and Embodiment 1 is that the specific implementation steps of S250 of this embodiment are different from those of S150 of Embodiment 1, as shown in the following table: Figure 7 The specific steps of S250 of this embodiment include:

[0188] S251, filtering based on the confidence conditions set for the radar target data and the camera target data respectively, and projecting the filtered millimeter-wave radar target points to the plane of the camera image at the same time;

[0189] S252, filtering out all millimeter-wave radar target points existing in a camera target detection frame, and determining whether the longitudinal distance error between the millimeter-wave radar target point and the camera target detection frame is less than a preset threshold;

[0190] S253, if not, the radar target data and the camera target data are not processed.

[0191] Embodiment 4

[0192] The difference between this embodiment and Embodiment 2 is that the specific function flow of the matching module of this embodiment is different from that of Embodiment 1, as shown in the following table: Figure 8 The specific function flow of the matching module of this embodiment includes:

[0193] The filtering unit 51 is configured to filter based on the confidence conditions set for the radar target data and the camera target data respectively, and project the filtered millimeter-wave radar target points to the plane of the camera image at the same time;

[0194] The judging unit 52 is configured to filter out all millimeter-wave radar target points existing in a camera target detection frame, and determine whether the longitudinal distance error between the millimeter-wave radar target point and the camera target detection frame is less than a preset threshold;

[0195] The non-processing unit 54 is configured to, if the longitudinal distance error between the millimeter-wave radar target point and the camera target detection frame is not less than the preset threshold, not process the radar target data and the camera target data.

[0196] Embodiment 5

[0197] The matching and fusion obstacle detection method described in combination Figure 1 with the drawings can be implemented by an electronic device. Figure 9 The hardware structure of the device according to this embodiment is shown in the following figure.

[0198] The electronic device can include a processor 71 and a memory 72 having stored computer program instructions.

[0199] In particular, the processor 71 described above can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the application.

[0200] The memory 72 can include a mass storage for data or instructions. By way of example, and without limitation, the memory 72 can include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash drive, a compact disc (CD) or DVD, a tape, a magnetic or optical or other recording device, a universal serial bus (USB) drive, or two or more of these or other devices in combination. The memory 72 can be removable or non-removable (or fixed) as appropriate. The memory 72 can be internal or external as appropriate. In particular embodiments, the memory 72 is a non-volatile memory. In particular embodiments, the memory 72 includes a read-only memory (ROM) and a random access memory (RAM). The ROM can be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a FLASH memory, or two or more of these or other devices in combination, as appropriate. The RAM can be a static RAM (SRAM) or a dynamic RAM (DRAM), which can be a Fast Page Mode DRAM (FPM DRAM), an Extended Data Output DRAM (EDO DRAM), a synchronous DRAM (SDRAM), or the like, as appropriate.

[0201] The memory 72 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 71.

[0202] The processor 71 realizes the matching fusion obstacle detection method of the above-mentioned embodiment 1 by reading and executing the computer program instructions stored in the memory 72.

[0203] In some embodiments, the electronic device can further include a communication interface 73 and a bus 70. As shown, the processor 71, the memory 72, and the communication interface 73 are connected through the bus 70 and complete communication with each other. Figure 9

[0204] The communication interface 73 is used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application. The communication interface 73 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, image / data processing workstations, etc.

[0205] ​Bus 70 includes hardware, software, or both, to couple components of the device to each other and to couple components of the device to other devices. What is considered a component of the device can vary depending on the particular view of the device that is considered. By way of example, a bus 70 can be a Data Bus, an Address Bus, a Control Bus, an Expansion Bus, a Local Bus, or a combination of one or more of these buses. By way of example and not limitation, bus 70 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or combination of two or more of these buses. Where appropriate, bus 70 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0206] The device can acquire the matched fusion obstacle detection system, and perform the matched fusion obstacle detection method of the embodiment 1.

[0207] In addition, in combination with the matched fusion obstacle detection method in the above-mentioned embodiment 1, the application can provide a storage medium for implementation. The storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement the matched fusion obstacle detection method of the above-mentioned embodiment 1.

[0208] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of detecting obstacles that match a fusion, characterized by, The application relates to a method for fusing millimeter wave radar and camera target data. The application comprises the following steps: Obtaining point cloud data of a road environment based on millimeter wave radar and image data of the road environment based on a camera, wherein the image data comprises camera images when a vehicle is running; Projecting the environment point cloud data onto a plane where the camera image is located according to joint calibration of the millimeter wave radar and the camera; Screening radar target data from the point cloud data according to a preset safe driving area; Obtaining camera target data from the image data through a target detection algorithm, wherein the camera target data comprises a longitudinal distance of the vehicle relative to a front obstacle; Matching the radar target data and the camera target data through a target matching strategy, and fusing the radar target data and the camera target data belonging to the same target obstacle; Outputting target information detected by the millimeter wave radar and the camera corresponding to the same target obstacle and target information of the millimeter wave radar and the camera meeting a credibility condition; The step of obtaining camera target data from the image data through a target detection algorithm, wherein the camera target data comprises a longitudinal distance of the vehicle relative to a front obstacle, specifically comprises the following steps: Inputting the image data into a trained YOLOv4 model for model inference, so as to identify the type of an obstacle in the camera image and calculate position data of the obstacle in the camera image; In the formula, Z represents the longitudinal distance of the vehicle relative to the obstacle in front of it, Estimating camera target data from the position data through a monocular distance measurement model, wherein the monocular distance measurement model is specifically as follows: represents the offset of the camera main axis from the image plane y direction, represents the camera pixel density, Y represents the maximum longitudinal pixel value of the target frame, H represents the camera main axis relative to the ground installation height, f represents the camera focal length; dy The step of matching the radar target data and the camera target data through a target matching strategy, and fusing the radar target data and the camera target data belonging to the same target obstacle, specifically comprises the following steps: Screening the radar target data and the camera target data according to their respective credibility conditions, and projecting the screened millimeter wave radar target points onto a plane where a camera image at the same time is located; Screening all millimeter wave radar target points existing in a camera target detection frame, and judging whether a longitudinal distance error between the millimeter wave radar target points and the camera target detection frame is less than a preset threshold value; If yes, the target points detected by the millimeter wave radar and the camera are determined as the same target, and the radar target data and the camera target data are fused; 2. The matched fusion obstacle detection method of claim 1, wherein, If no, the radar target data and the camera target data are not processed. The three-dimensional coordinates of the point cloud data The coordinates are converted into vehicle coordinates using a first preset matrix, which is as follows: In the formula, denotes a coordinate value in the vehicle coordinate system, denotes a rotation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system, denotes a translation matrix between the vehicle coordinate system and the millimeter wave radar coordinate system; The step of projecting the environment point cloud data onto a plane where the camera image is located according to joint calibration of the millimeter wave radar and the camera, specifically comprises the following steps: In the formula, represents a coordinate value in the camera coordinate system, represents a rotation matrix between the vehicle coordinate system and the camera coordinate system, represents a translation matrix between the vehicle coordinate system and the camera coordinate system; Converting coordinate values in a vehicle coordinate system into coordinate values in a camera coordinate system through a second preset matrix, wherein the second preset matrix is specifically as follows: Converting coordinate values in the camera coordinate system into coordinate values in a pixel coordinate system through a third preset matrix, wherein the third preset matrix is specifically as follows: In the formula, denotes a coordinate value in a homogeneous form under a pixel coordinate system, denotes an intrinsic parameter matrix of the camera; The coordinate value in the pixel coordinate system is normalized to obtain a pixel coordinate value of the point cloud data projected onto a plane where the camera image is located. In the formula, represents a width value of the point cloud data projection, represents a height value of the point cloud data projection.

3. The matched fusion obstacle detection method of claim 1, wherein, The preset safe driving area refers to a set area with a lateral distance of 10.5 m on the left and right sides and a longitudinal distance of 80 m.

4. The matched fusion obstacle detection method of claim 1, wherein, The credibility condition of the radar target data is that the target appearance times are greater than 2, and the credibility condition of the camera target data is that the target category probability is greater than 75%.

5. The matched fusion obstacle detection method of claim 1, wherein, The target information includes one or more of the data detected by the millimeter wave radar, the data detected by the camera, and the data detected by both the millimeter wave radar and the camera. If the target information includes the data detected by both the millimeter wave radar and the camera, the longitudinal distance is completely based on the data detected by the millimeter wave radar. If the target information includes the data detected by the millimeter wave radar, whether the radar data point extension times or the longitudinal distance meets the set condition is determined to determine whether the millimeter wave radar target data is output. If the target information includes the data detected by the camera, whether the target category probability is greater than the set condition is determined to determine whether the camera target data is output.

6. A matched fusion obstacle detection system characterized by, It includes: An acquisition module is configured to acquire point cloud data of a road environment based on a millimeter wave radar and image data of the road environment based on a camera, wherein the image data includes a camera image when a vehicle is driving; A calibration module is configured to project the environment point cloud data onto a plane where the camera image is located based on joint calibration of the millimeter wave radar and the camera; A screening module is configured to screen radar target data from the point cloud data based on a preset safe driving area; An algorithm module is configured to obtain camera target data by detecting target algorithms based on the image data, wherein the camera target data includes a longitudinal distance of the vehicle relative to a front obstacle; A matching module is configured to match the radar target data and the camera target data to the same target matching strategy, and to fuse the radar target data and the camera target data belonging to the same target obstacle; An output module is configured to output target information detected by the millimeter wave radar and the camera corresponding to the same target obstacle and target information of the millimeter wave radar and the camera that meets the credibility condition. The algorithm module includes: An identification unit is configured to input the image data into a trained YOLOv4 model for model inference, so as to identify the type of the obstacle in the camera image and calculate the position data of the obstacle in the camera image; An estimation unit is configured to estimate camera target data based on monocular distance measurement model, wherein the monocular distance measurement model is as follows: In the formula, Z represents the longitudinal distance of the vehicle relative to the front obstacle, dy represents the offset of the camera principal axis from the image plane y direction, represents the camera pixel density, Y represents the target frame longitudinal pixel maximum value, H represents the camera principal axis relative to the ground installation height, f represents the camera focal length; The matching module includes: A screening unit is configured to screen based on the set credibility conditions of the radar target data and the camera target data, and project the screened millimeter wave radar target points onto the plane where the camera image at the same time is located. A judging unit is configured to screen all millimeter wave radar target points existing in a camera target detection frame of a camera, and determine whether a longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than a preset threshold value; A fusion unit is configured to determine that the target points detected by the millimeter wave radar and the camera are the same target if the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is less than the preset threshold value, and perform data fusion on the radar target data and the camera target data. A non-processing unit is configured to not process the radar target data and the camera target data if the longitudinal distance error between the millimeter wave radar target point and the camera target detection frame is not less than the preset threshold value.

7. An apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor executes the computer program to implement the matching and fusion obstacle detection method in any one of claims 1 to 5.

8. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the matching and fusion obstacle detection method in any one of claims 1 to 5.

Images