Fusion method and device of radar sensors, computer equipment and storage medium

Abstract

The application provides a fusion method and device of a radar sensor, a computer device and a storage medium. On the one hand, two radar sensors are used for detection, compared with a single radar sensor, the false detection can be reduced, and the possibility of obstacle detection can be improved. On the other hand, when the detection results of the two radar sensors are fused, each obstacle detected by the two radar sensors is matched by a representative point to determine a target fusion obstacle, that is, a plurality of point clouds of an obstacle are represented by a representative point, the number of point clouds to be matched is reduced, and the number of calculations required for matching is reduced. According to the fusion of the detection results of the two sensors corresponding to the target fusion obstacle, the fusion efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, computer device, and storage medium for fusing radar sensors. Background Technology

[0002] For autonomous driving technology, accurately perceiving obstacles around driverless vehicles provides a foundation for making correct decisions for autonomous driving control, ensuring driving safety.

[0003] Currently, most autonomous driving technologies rely on a single sensor for obstacle detection, such as a LiDAR sensor or simply images. While some solutions use two or more sensors, these require fusion of the results. Sensor fusion often involves combining visual sensors with millimeter-wave radar or visual sensors with LiDAR. However, these solutions require extensive processing of image data, resulting in high time costs and low efficiency. Summary of the Invention

[0004] According to the content of this application, a method, apparatus, computer device, and storage medium for fusion of radar sensors are provided.

[0005] A radar sensor fusion method, comprising:

[0006] Obtain the first obstacle detection result from the first radar sensor, and determine the coordinate information of the representative points of each first obstacle in the first obstacle detection result;

[0007] The second obstacle detection result of the second radar sensor is obtained, and the coordinate information of the representative point of each second obstacle in the second obstacle detection result is determined; wherein, the first sensor and the second sensor are different types of radar sensors;

[0008] Based on the coordinate information, the representative points of the first obstacle and / or the representative points of the second obstacle are projected, and the projected representative points of the first obstacle and the second obstacle are in the same coordinate system.

[0009] The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched to determine the target fusion obstacle.

[0010] The first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle are fused together to output the fusion result of the target fusion obstacle.

[0011] In one embodiment, the first radar sensor is a lidar sensor, and the step of acquiring the first obstacle detection result of the first radar sensor and determining the coordinate information of representative points of each first obstacle in the first obstacle detection result includes:

[0012] Acquire the first point cloud data collected by the first radar sensor;

[0013] The first point cloud data is identified to determine each first obstacle in the first point cloud data, and the first obstacle detection result is obtained;

[0014] The first obstacle is marked with a preset shape in the first radar coordinate system;

[0015] Based on the center point of the preset shape, the coordinate information of the representative points of each first obstacle in the first obstacle detection result is obtained.

[0016] In one embodiment, acquiring the second obstacle detection result from the second radar sensor and determining the coordinate information of representative points of each second obstacle in the second obstacle detection result includes:

[0017] Acquire the second point cloud data collected by the second radar sensor;

[0018] Cluster the second point cloud data to identify each second obstacle in the second point cloud data and obtain the second obstacle detection result;

[0019] Based on the center point of each of the second obstacles, the coordinate information of the representative point of each of the second obstacles in the second obstacle detection result is obtained.

[0020] In one embodiment, the method of projecting representative points of the first obstacle and / or the second obstacle based on the coordinate information includes:

[0021] Based on the coordinate information, the representative points of each of the first obstacles are projected onto the coordinate system of the second radar sensor; or

[0022] Based on the coordinate information, the representative points of each of the second obstacles are projected onto the coordinate system of the first radar sensor; or

[0023] Based on the coordinate information, the representative points of each of the first obstacles and the representative points of the second obstacles are projected onto the third coordinate system.

[0024] In one embodiment, representative points of the first obstacle and representative points of the second obstacle in the same coordinate system are matched to determine the target fusion obstacle, including:

[0025] The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance. If the distance is less than a preset distance, the target fused obstacle is obtained.

[0026] In one embodiment, the first radar sensor is a lidar sensor, and the second radar sensor is a millimeter-wave radar sensor. The step of fusing the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle to output the fused result of the target fusion obstacle includes:

[0027] Based on the coordinate information of the target fused obstacle in the first obstacle detection result and the coordinate information of the target fused obstacle in the second obstacle detection result, the fused coordinate information of the target fused obstacle is determined;

[0028] The target fusion obstacle is fused based on the detection results of the target fusion obstacle in the first obstacle detection result, the fusion coordinate information, and the velocity information of the target fusion obstacle in the second obstacle detection result to obtain the fusion result of the target fusion obstacle.

[0029] In one embodiment, the representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance. If the distance is less than a preset distance, a target fused obstacle is obtained, including:

[0030] The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance to obtain candidate fusion obstacles whose distance is less than a preset distance;

[0031] Obtain the first confidence level of the candidate fused obstacle in the first obstacle detection result and the second confidence level of the candidate fused obstacle in the second obstacle detection result, and determine the final confidence level of the candidate fused obstacle;

[0032] The target fusion obstacle is obtained from the candidate fusion obstacles whose final confidence level is greater than the threshold.

[0033] A radar sensor fusion device, comprising:

[0034] The first detection module is used to acquire the first obstacle detection result of the first radar sensor and determine the coordinate information of the representative points of each first obstacle in the first obstacle detection result;

[0035] The second detection module is used to acquire the second obstacle detection results of the second radar sensor and determine the coordinate information of the representative points of each second obstacle in the second obstacle detection results; wherein, the first sensor and the second sensor are different types of radar sensors;

[0036] The projection module is used to project the representative points of the first obstacle and / or the representative points of the second obstacle according to the coordinate information, wherein the projected representative points of the first obstacle and the representative points of the second obstacle are in the same coordinate system.

[0037] The matching module is used to match the representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle to be matched.

[0038] The fusion module is used to fuse the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle, and output the fusion result of the target fusion obstacle.

[0039] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the methods described in the above embodiments.

[0040] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the methods described in the above embodiments.

[0041] The aforementioned radar sensor fusion method, apparatus, computer equipment, and storage medium, on the one hand, utilize two radar sensors for detection, which reduces false detections and increases the likelihood of obstacle detection compared to using a single radar sensor. On the other hand, when fusing the detection results from two radar sensors, each obstacle detected by the two radar sensors is matched with representative points to determine the target fusion obstacle. That is, a representative point represents multiple point clouds of the obstacle, reducing the number of point clouds to be matched, and thus reducing the amount of computation required for matching. By fusing the detection results from the two sensors corresponding to the target fusion obstacle, fusion efficiency can be improved. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the application environment of a radar sensor fusion method in one embodiment;

[0043] Figure 2 This is a flowchart illustrating a radar sensor fusion method in one embodiment;

[0044] Figure 3 This is a flowchart illustrating the radar sensor fusion method in another embodiment;

[0045] Figure 4 This is a schematic diagram of the structure of a radar sensor fusion device in one embodiment;

[0046] Figure 5 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] The radar sensor fusion method provided in this application can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the system includes: an intelligent driving device 101, and a first radar sensor 102, a second radar sensor 103, and a vehicle controller 104 mounted on the intelligent driving device 101. During operation, the first radar sensor 102 and the second radar sensor 103 collect point cloud data respectively, and the vehicle controller 104 fuses the point cloud data from the two radar sensors. The intelligent driving device includes, but is not limited to, autonomous vehicles, walking robots, and other driving devices.

[0049] like Figure 2 As shown, a radar sensor fusion method is provided, which is applied to... Figure 1 The vehicle controller shown includes:

[0050] S202, acquire the first obstacle detection result of the first radar sensor, and determine the coordinate information of the representative points of each first obstacle in the first obstacle detection result.

[0051] Specifically, the first radar sensor collects point cloud data of surrounding objects, identifies obstacles based on the point cloud data, and obtains the first obstacle detection result. It can be understood that the first obstacle detection result includes multiple obstacles within the range of the first radar sensor, such as vehicles, traffic lights, and pedestrians surrounding the intelligent driving device during its operation.

[0052] Specifically, the first radar sensor can be a lidar sensor, a millimeter-wave radar sensor, an ultrasonic radar, etc.

[0053] In this embodiment, a representative point is determined for each detected obstacle. It is understood that the principle for determining each representative point is the same, and it can be the center point of the obstacle. For example, if the first obstacle detected in the first obstacle detection result is marked with a two-dimensional rectangular box in the bird's-eye view, that is, the first obstacle in the bird's-eye view is represented by a rectangle, then the representative point of the obstacle is the center point of the rectangle. Specifically, the bird's-eye view is based on the first radar sensor coordinate system established by the first radar sensor, and the position information of the representative point is represented by a Cartesian coordinate in the first radar sensor coordinate system.

[0054] S204, acquire the second obstacle detection result of the second radar sensor, and determine the coordinate information of the representative point of each second obstacle in the second obstacle detection result; wherein, the first sensor and the second sensor are different types of radar sensors.

[0055] Specifically, the second radar sensor collects point cloud data of surrounding objects, identifies obstacles based on the point cloud data, and obtains a second obstacle detection result. It can be understood that the second obstacle detection result includes multiple obstacles within the range of the second radar sensor, such as vehicles, traffic lights, and pedestrians surrounding the intelligent driving device during its operation.

[0056] Specifically, the first radar sensor and the second radar sensor are different types of radar sensors. For example, if the first radar sensor is a lidar sensor, then the second radar sensor can be a millimeter-wave radar sensor; or if the first radar sensor is a millimeter-wave radar sensor, then the second radar sensor can be a lidar sensor. Different types of radar sensors have different advantages and disadvantages. Using two different types of radar sensors for data fusion can improve the detection accuracy.

[0057] In this embodiment, a representative point is determined for each detected obstacle. It is understood that the principle for determining each representative point is the same, and it can be the center point of the obstacle. The position information of the representative point is represented by a Cartesian coordinate in the second radar sensor coordinate system.

[0058] S206, Project the representative points of the first obstacle and / or the representative points of the second obstacle according to the coordinate information, and the representative points of the first obstacle and the second obstacle are in the same coordinate system after the projection processing.

[0059] Specifically, the representative points of each first obstacle detected by the first radar sensor are represented in the first radar sensor coordinate system, and the representative points of each second obstacle detected by the second radar sensor are represented in the second radar sensor coordinate system. They are in different coordinate systems. For easy matching, they need to be in the same coordinate system to accurately match the obstacle detection results from the two radar sensors.

[0060] Specifically, the representative point of the first obstacle can be projected onto the second radar coordinate system, or the representative point of the second obstacle can be projected onto the first radar coordinate system, or the representative points of both obstacles can be projected onto a third coordinate system. The third coordinate system can be any coordinate system other than the first and second radar coordinate systems, such as the world coordinate system or the GPS coordinate system. The projection of the representative point is performed using calibration parameters between the two coordinate systems; these calibration parameters are pre-calibrated.

[0061] All three projection methods described above can ensure that the representative points of the first obstacle and the second obstacle are in the same coordinate system after projection processing.

[0062] S208, Match the representative points of the first obstacle with the representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle.

[0063] Specifically, for the representative point of the first obstacle that is projected into the same coordinate system, it represents the position information of the representative points of each obstacle detected by the first radar sensor in that coordinate system. For the representative point of the second obstacle that is projected into the same coordinate system, it represents the position information of the representative points of each obstacle detected by the second radar sensor in that coordinate system. If the representative points of the first obstacle and the second obstacle match, it means that the representative points of the first obstacle and the second obstacle correspond to the same obstacle, that is, both the first radar sensor and the second radar sensor have detected the obstacle, and the obstacle is identified as the target fusion obstacle. In other words, if the representative points of the first obstacle and the second obstacle do not match, it means that either the first radar sensor or the second radar sensor has made a false detection.

[0064] S210, fuse the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle, and output the fusion result of the target fusion obstacle.

[0065] As mentioned earlier, the first and second obstacle detection results were obtained from different radar sensors. Different radar sensors have different advantages and disadvantages. For example, lidar sensors have advantages such as high detection accuracy, wide detection range, and strong stability. However, lidar sensors detect by emitting a beam of light, making them highly susceptible to environmental influences. If the beam is blocked, they cannot function properly, thus they cannot be used in severe weather conditions such as rain, snow, fog, haze, or sandstorms. Millimeter-wave radar sensors, on the other hand, have a strong ability to penetrate fog, smoke, and dust through their millimeter-wave seekers, allowing them to detect obstacles in bad weather. However, their detection range is directly limited by frequency band loss (to detect further distances, high-frequency radar must be used), they cannot detect pedestrians, and they cannot accurately model all surrounding obstacles.

[0066] In this embodiment, by equipping the intelligent driving device with two different types of radar sensors, the results of the target and obstacle detection are fused from the two different types of radar sensors. Compared with a single radar sensor, this can reduce false detections and improve the reliability of obstacle detection.

[0067] Specifically, fusion can be achieved by averaging the coordinate information of the obstacle identified in the first obstacle identification result and the coordinate information of the obstacle identified in the second obstacle identification result, or by taking a value with a high confidence level. Alternatively, it can be achieved by taking a portion of the results from one of the obstacle identification results and combining the two as the final identification result. For example, taking the coordinate detection result from the lidar sensor and the velocity detection result from the millimeter-wave radar sensor and fusing them to obtain the final obstacle detection result.

[0068] The aforementioned radar sensor fusion method, on the one hand, utilizes two radar sensors for detection, which reduces false detections and increases the likelihood of obstacle detection compared to a single radar sensor. On the other hand, when fusing the detection results from two radar sensors, each obstacle detected by the two sensors is matched with representative points to determine the target fusion obstacle. That is, a representative point represents multiple point clouds of the obstacle, reducing the number of point clouds to be matched and thus reducing the amount of computation required for matching. Furthermore, fusing the detection results from the two sensors corresponding to the target fusion obstacle improves fusion efficiency.

[0069] In another embodiment, the first radar sensor is a lidar sensor. Acquiring the first obstacle detection result from the first radar sensor and determining the coordinate information of representative points of each first obstacle in the first obstacle detection result includes: acquiring first point cloud data collected by the first radar sensor; identifying the first point cloud data to determine each first obstacle in the first point cloud data, thus obtaining the first obstacle detection result; marking the first obstacle with a preset shape in the first radar coordinate system; and obtaining the coordinate information of representative points of each first obstacle in the first obstacle detection result based on the center point of the preset shape. In this embodiment, if the first radar sensor is a lidar sensor, then the second radar sensor is a non-lidar sensor, such as a millimeter-wave radar sensor.

[0070] A lidar sensor is a sensor that uses laser technology for measurement. Lidar sensors are characterized by being unaffected by lighting conditions and directly obtaining accurate three-dimensional information, thus exhibiting high detection accuracy, a wide detection range, and strong stability.

[0071] The lidar sensor collects data at a set frame rate, acquiring the first point cloud data within its range. For each frame of the first point cloud data, a deep learning model is used to detect and identify the first obstacles within the data. It's understood that the deep learning model is pre-trained on a neural network model using a large amount of labeled point cloud data; this neural network model can be a convolutional neural network, but this is not a specific limitation. The output of the deep learning model is the coordinate information of all obstacles in each frame of the first point cloud data.

[0072] For each first obstacle, based on its size information (which can be determined by the coordinate range), a preset shape adapted to that size is created, and the position of each first obstacle is marked by the preset shape. For example, if the preset shape is a rectangle, then in the bird's-eye view corresponding to the first point cloud data, i.e., in the first radar coordinate system, the position of the first obstacle is marked by a rectangle. It can be understood that the size of each preset shape is related to the size of the obstacle; the larger the identified obstacle, the larger the size of the marked preset shape. The number of rectangles is related to the number of obstacles.

[0073] By utilizing the geometric relationships of a preset shape, the center point of the preset shape is determined, thus obtaining the representative point of each first obstacle in the first obstacle detection result. For example, the center point of a rectangle is the intersection of its two diagonals, and the center point of a circle is its center. In other words, in this embodiment, the center point of the preset shape that identifies each first obstacle is used as the representative point of the first obstacle, and the coordinate information of the representative point is the coordinate information of the center point.

[0074] By using representative points to represent obstacles, the amount of computation during matching can be reduced, and the matching speed can be improved.

[0075] In another embodiment, obtaining the second obstacle detection result of the second radar sensor and determining the coordinate information of the representative point of each second obstacle in the second obstacle detection result includes: obtaining the second point cloud data collected by the second radar sensor; clustering the second point cloud data to identify each second obstacle in the second point cloud data and obtain the second obstacle detection result; and obtaining the coordinate information of the representative point of each second obstacle in the second obstacle detection result based on the center point of each second obstacle.

[0076] In this embodiment, the first radar sensor can be a lidar sensor, and the second radar sensor can be a millimeter-wave radar sensor or other types of sensors, as long as they are different types of sensors from the first radar sensor.

[0077] Regarding the method for identifying representative points of obstacles by the second radar sensor, there are two scenarios. In one scenario, the second radar sensor outputs point cloud data, in which case the obstacle detection result can be obtained by clustering the point cloud data collected by the second radar sensor. In the other scenario, the second radar sensor outputs object-level point cloud results after clustering the point cloud data, and the representative points of the obstacles are determined based on the identification results.

[0078] Specifically, if the second radar sensor does not perform clustering processing on the clustered point cloud, then the data needs to be clustered. Taking a millimeter-wave radar sensor as an example, the detection result of the millimeter-wave radar sensor is a point cloud, and points easily cluster on the surface of materials that are difficult to penetrate, such as metal. In this embodiment, all points in the current frame are clustered according to the degree of clustering of the point cloud to identify each second obstacle in the second point cloud. The clustering method can use the DBSCAN algorithm. For each identified second obstacle, its center point is determined. The center point can be determined by marking the identified obstacle with a preset shape, the size of the preset shape matching the size of the obstacle, and determining the representative point of the second obstacle based on the center point of the preset shape. One point will represent one obstacle. It should be noted that the marking shape of the first obstacle is the same as the marking shape of the second obstacle. This point is represented by a Cartesian coordinate in the bird's-eye view.

[0079] In another embodiment, matching representative points of the first obstacle and representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle includes: matching the distance between representative points of the first obstacle and representative points of the second obstacle in the same coordinate system; if the distance is less than a preset distance, the target fusion obstacle is obtained.

[0080] Specifically, to match the distance between the representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system, the Hungarian algorithm can be used. The Hungarian algorithm is a combinatorial optimization algorithm that solves the task assignment problem in polynomial time and promoted the later primal-dual method. Concept: Let G=[V, E] be an undirected graph. If a set of vertices V can be partitioned into two disjoint subsets V1 and V2, the subset with the largest number of edges is called the maximum matching problem of the graph. If in a matching, ||V1|| <= ||V2|| and the number of matchings |M| = |V1|, then this matching is called a complete matching.

[0081] Specifically, representative points of the first obstacle and the second obstacle in the same coordinate system are matched by distance using Euclidean distance. If the distance between the representative points of the two obstacles is less than a threshold, it indicates that the detection result points to the same obstacle, i.e., the target is fused to the obstacle. In other embodiments, the matching can also set different thresholds for the reflection intensity of the millimeter-wave radar sensor, the confidence level returned by the sensor, etc.

[0082] For representative points of the first and second obstacles that do not match, corresponding to falsely detected obstacles, the corresponding detection results are removed as a screening for false detections, thereby improving the false detection rate.

[0083] In another embodiment, the first radar sensor is a lidar sensor, and the second radar sensor is a millimeter-wave radar sensor.

[0084] The target fusion obstacle is fused from the first obstacle detection result and the second obstacle detection result, and the fusion result of the target fusion obstacle is output. This includes: determining the fusion coordinate information of the target fusion obstacle based on the coordinate information of the target fusion obstacle in the first obstacle detection result and the coordinate information of the target fusion obstacle in the second obstacle detection result; and fusing the target fusion obstacle based on the detection result of the target fusion obstacle in the first obstacle detection result, the fusion coordinate information, and the velocity information of the target fusion obstacle in the second obstacle detection result to obtain the fusion result of the target fusion obstacle.

[0085] In this embodiment, a lidar sensor is used as the first radar sensor, and a millimeter-wave radar sensor is used as the second radar sensor. Lidar sensors have advantages such as high detection accuracy, wide detection range, and strong stability. However, lidar sensors detect objects by emitting a beam, making them highly susceptible to environmental influences. If the beam is blocked, they cannot function properly, thus they cannot be activated in adverse weather conditions such as rain, snow, fog, haze, or sandstorms. Millimeter-wave radar sensors, on the other hand, have strong penetrating capabilities through their millimeter-wave seekers, allowing them to detect objects in poor weather conditions. Furthermore, millimeter-wave radar sensors rely on Doppler frequency shift to reliably detect the speed of obstacles. However, their detection range is directly limited by frequency band loss (to detect further distances, higher-frequency radar must be used), they cannot detect pedestrians, and they cannot accurately model all surrounding obstacles.

[0086] In this embodiment, the advantages of both lidar and millimeter-wave radar are combined during fusion. Specifically, during fusion, the coordinate information of the target and obstacle in the coverage detection results of the two radars is smoothed, and the average value is taken to obtain the fused coordinate information. Then, the velocity detected by the millimeter-wave radar sensor is assigned to the corresponding lidar sensor detection result, so that the obstacle has reliable velocity information. That is, the coordinate fusion information, the high-precision detection result of lidar, and the velocity of millimeter-wave radar are used as the final result of target and obstacle fusion.

[0087] On the one hand, the average value of the coordinate information detected by the two radars is taken as the fused coordinate information of the obstacle, which takes into account the two coordinate detection results and improves the detection accuracy. On the other hand, the velocity information of the millimeter-wave radar sensor is taken and given to the lidar sensor. The high accuracy of velocity detection of the millimeter-wave radar sensor is utilized. Thus, the more accurate result of each radar sensor is taken during the fusion process, which improves the accuracy of the final obstacle detection result.

[0088] In another embodiment, distance matching is performed between representative points of a first obstacle and representative points of a second obstacle in the same coordinate system. If the distance is less than a preset distance, a target fused obstacle is obtained. This includes: performing distance matching between representative points of a first obstacle and representative points of a second obstacle in the same coordinate system to obtain candidate fused obstacles whose distance is less than a preset distance; obtaining the first confidence level of the candidate fused obstacles in the first obstacle detection result and the second confidence level of the candidate fused obstacles in the second obstacle detection result, and determining the final confidence level of the candidate fused obstacles; obtaining the target fused obstacle based on the candidate fused obstacles whose final confidence level is greater than a threshold.

[0089] Specifically, to further improve the accuracy of obstacle detection, when outputting the fusion results, the confidence levels of the outputs from the two sensors can be weighted. Only matching results and results with a confidence level greater than a set threshold are retained. Here, confidence level represents the reliability of the result; it can be the confidence level of the recognition output or it can be calculated. For example, a simple method is to count the number of points in all existing obstacle clusters and take the maximum value (or even group them), then divide the number of points in each cluster by this value to obtain the confidence level.

[0090] In this embodiment, the detection results of two obstacles are matched by combining confidence level and distance threshold, which can effectively eliminate obstacles that do not meet the distance or confidence level requirements, serving as a screening for false detections and improving detection accuracy.

[0091] Taking the first radar sensor as a lidar sensor and the second radar sensor as a millimeter-wave radar sensor as an example, such as... Figure 3 As shown:

[0092] Both lidar and millimeter-wave radar sensors collect point cloud data, yielding lidar and millimeter-wave radar detection results, respectively. The lidar detection results can be obtained by identifying the lidar point cloud data using a trained model. Both lidar and millimeter-wave radar detection results include the coordinate information of representative points of the detected obstacles.

[0093] Specifically, the representative point of an obstacle for a lidar sensor is the center point determined based on the geometric relationship of the rectangular frame after the obstacle is marked with a rectangle. The size of the rectangle is related to the size of the obstacle.

[0094] The representative point of an obstacle in a millimeter-wave radar sensor is the center point of the obstacle identified by clustering the point cloud data of the millimeter-wave radar.

[0095] Based on the coordinate information, the representative points of the first obstacle and / or the representative points of the second obstacle are projected, and the projected representative points of the first obstacle and the second obstacle are in the same coordinate system.

[0096] Specifically, the representative point of the first obstacle can be projected onto the second radar coordinate system, or the representative point of the second obstacle can be projected onto the first radar coordinate system, or the representative points of both obstacles can be projected onto a third coordinate system. The third coordinate system can be any coordinate system other than the first and second radar coordinate systems, such as the world coordinate system or the GPS coordinate system. The projection of the representative point is performed using calibration parameters between the two coordinate systems; these calibration parameters are pre-calibrated.

[0097] All three projection methods described above can ensure that the representative points of the first obstacle and the second obstacle are in the same coordinate system after projection processing.

[0098] The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched to determine the target fusion obstacle.

[0099] The first obstacle detection result and the second obstacle detection result of the target fusion obstacle are merged to output the fusion result of the target fusion obstacle.

[0100] Specifically, mismatched obstacle representative points are deleted, and matched obstacle representative points are merged. Specifically, based on the coordinate information of the target fused obstacle in the first obstacle detection result and the coordinate information of the target fused obstacle in the second obstacle detection result, the fusion coordinate information of the target fused obstacle is determined; fusion is performed based on the detection result of the target fused obstacle in the first obstacle detection result, the fusion coordinate information, and the velocity information of the target fused obstacle in the second obstacle detection result to obtain the fusion result of the target fused obstacle.

[0101] In this embodiment, a fusion of LiDAR and millimeter-wave radar sensors is used. On one hand, compared to a single radar sensor, this reduces false detections and increases the likelihood of obstacle detection. On the other hand, since each obstacle detected by the two radar sensors is matched with representative points to determine the target fusion obstacle—that is, one representative point represents multiple point clouds of an obstacle—the number of point clouds to be matched is reduced, thus reducing the amount of computation required for matching. The fusion is performed based on the detection results of the two sensors corresponding to the target fusion obstacle, improving fusion efficiency. Furthermore, the fusion result uses the higher accuracy of both the LiDAR and millimeter-wave radar sensors, improving the accuracy of obstacle recognition.

[0102] Figure 2-3 This is a flowchart illustrating a radar sensor fusion method in one embodiment. It should be understood that, although... Figure 2-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2-3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0103] A radar sensor fusion device, such as Figure 4 As shown, it includes:

[0104] The first detection module 402 is used to acquire the first obstacle detection result of the first radar sensor and determine the coordinate information of the representative points of each first obstacle in the first obstacle detection result;

[0105] The second detection module 404 is used to acquire the second obstacle detection results of the second radar sensor and determine the coordinate information of the representative points of each second obstacle in the second obstacle detection results; wherein, the first sensor and the second sensor are different types of radar sensors.

[0106] The projection module 406 is used to project the representative points of the first obstacle and / or the representative points of the second obstacle according to the coordinate information. The representative points of the first obstacle and the representative points of the second obstacle are in the same coordinate system after the projection processing.

[0107] The matching module 408 is used to match the representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle to be matched.

[0108] The fusion module 410 is used to fuse the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle, and output the fusion result of the target fusion obstacle.

[0109] The aforementioned radar sensor fusion device, on the one hand, utilizes two radar sensors for detection, which reduces false detections and increases the likelihood of obstacle detection compared to a single radar sensor. On the other hand, when fusing the detection results from the two radar sensors, each obstacle detected by the two sensors is matched with representative points to determine the target fusion obstacle. That is, a representative point represents multiple point clouds of the obstacle, reducing the number of point clouds to be matched, and thus reducing the amount of computation required for matching. By fusing the detection results from the two sensors corresponding to the target fusion obstacle, the fusion efficiency can be improved.

[0110] In another embodiment, the first radar sensor is a lidar sensor, and the first detection module is used to acquire first point cloud data collected by the first radar sensor; identify the first point cloud data, determine each first obstacle in the first point cloud data, and obtain the first obstacle detection result; mark the first obstacle with a preset shape in the first radar coordinate system; and obtain the coordinate information of the representative point of each first obstacle in the first obstacle detection result according to the center point of the preset shape.

[0111] In another embodiment, the second detection module is used to acquire second point cloud data collected by the second radar sensor; cluster the second point cloud data to identify each second obstacle in the second point cloud data and obtain the second obstacle detection result; and obtain the coordinate information of the representative point of each second obstacle in the second obstacle detection result based on the center point of each second obstacle.

[0112] In another embodiment, the projection module is used to project representative points of each first obstacle onto the coordinate system of the second radar sensor according to coordinate information; or to project representative points of each second obstacle onto the coordinate system of the first radar sensor according to coordinate information; or to project representative points of each first obstacle and representative points of the second obstacle onto a third coordinate system according to coordinate information.

[0113] In another embodiment, the matching module is used to perform distance matching between the representative point of the first obstacle and the representative point of the second obstacle in the same coordinate system. If the distance is less than a preset distance, the target fused obstacle is obtained.

[0114] In another embodiment, the first radar sensor is a lidar sensor, the second radar sensor is a millimeter-wave radar sensor, and the fusion module is used to determine the fused coordinate information of the target fused obstacle based on the coordinate information of the target fused obstacle in the first obstacle detection result and the coordinate information of the target fused obstacle in the second obstacle detection result; and to fuse the target fused obstacle based on the detection result of the target fused obstacle in the first obstacle detection result, the fused coordinate information, and the velocity information of the target fused obstacle in the second obstacle detection result to obtain the fused result of the target fused obstacle.

[0115] In another embodiment, the matching module is used to perform distance matching between representative points of the first obstacle and representative points of the second obstacle in the same coordinate system to obtain candidate fusion obstacles whose distance is less than a preset distance; to obtain the first confidence level of the candidate fusion obstacles in the first obstacle detection result and the second confidence level of the candidate fusion obstacles in the second obstacle detection result, and to determine the final confidence level of the candidate fusion obstacles; and to obtain the target fusion obstacle based on the candidate fusion obstacles whose final confidence level is greater than a threshold.

[0116] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. Specifically, this computer device may be... Figure 1 The vehicle controller in the system. For example... Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program that, when executed by the processor, enables the processor to implement a radar sensor fusion method.

[0117] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one embodiment, the radar sensor fusion method provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 4 The computer device shown operates on this device. The computer device's memory can store various program modules that constitute the fusion apparatus of the radar sensor. The computer program, composed of these program modules, causes the processor to execute the steps in the radar sensor fusion methods of the various embodiments of this application described in this specification.

[0119] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the aforementioned radar sensor fusion method. The steps of the radar sensor fusion method here can be the steps in the radar sensor fusion methods of the various embodiments described above.

[0120] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps of the radar sensor fusion method described above. The steps of the radar sensor fusion method here can be the steps from the radar sensor fusion methods of the various embodiments described above.

[0121] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for fusing radar sensors, characterized in that, include: Acquire first point cloud data collected by the first radar sensor; identify the first point cloud data, determine each first obstacle in the first point cloud data, and obtain the first obstacle detection result; The first obstacle is marked with a preset shape in the first radar coordinate system; Based on the center point of the preset shape, obtain the coordinate information of the representative points of each first obstacle in the first obstacle detection result; Acquire the second point cloud data collected by the second radar sensor; Cluster the second point cloud data to identify each second obstacle in the second point cloud data and obtain the second obstacle detection result; based on the center point of each second obstacle, obtain the coordinate information of the representative point of each second obstacle in the second obstacle detection result; Based on the coordinate information, the representative points of the first obstacle and / or the representative points of the second obstacle are projected, and the projected representative points of the first obstacle and the second obstacle are in the same coordinate system. The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched to determine the target fusion obstacle. The first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle are fused together to output the fusion result of the target fusion obstacle; Where the first radar sensor is a lidar sensor and the second radar sensor is a millimeter-wave radar sensor, the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle are fused together to output the fusion result of the target fusion obstacle, including: Based on the coordinate information of the target fused obstacle in the first obstacle detection result and the coordinate information of the target fused obstacle in the second obstacle detection result, the fused coordinate information of the target fused obstacle is determined; The target fusion obstacle is fused based on the detection results of the target fusion obstacle in the first obstacle detection result, the fusion coordinate information, and the velocity information of the target fusion obstacle in the second obstacle detection result to obtain the fusion result of the target fusion obstacle.

2. The method according to claim 1, characterized in that, The method of projecting representative points of the first obstacle and / or the second obstacle based on the coordinate information includes: Based on the coordinate information, representative points of each of the first obstacles are projected onto the coordinate system of the second radar sensor.

3. The method according to claim 1, characterized in that, The method of projecting representative points of the first obstacle and / or the second obstacle based on the coordinate information includes: Based on the coordinate information, representative points of each of the second obstacles are projected onto the coordinate system of the first radar sensor.

4. The radar sensor fusion method according to claim 1, characterized in that, The method of projecting representative points of the first obstacle and / or the second obstacle based on the coordinate information includes: Based on the coordinate information, the representative points of each of the first obstacles and the representative points of the second obstacles are projected onto the third coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that, Matching representative points of the first obstacle with representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle, including: The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance. If the distance is less than a preset distance, the target fused obstacle is obtained.

6. The method according to any one of claims 1 to 4, characterized in that, The preset shape is either rectangular or circular.

7. The method according to any one of claims 1 to 4, characterized in that, The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance. If the distance is less than a preset distance, the target fused obstacle is obtained, including: The representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system are matched by distance to obtain candidate fusion obstacles whose distance is less than a preset distance; Obtain the first confidence level of the candidate fused obstacle in the first obstacle detection result and the second confidence level of the candidate fused obstacle in the second obstacle detection result, and determine the final confidence level of the candidate fused obstacle; The target fusion obstacle is obtained from the candidate fusion obstacles whose final confidence level is greater than the threshold.

8. A radar sensor fusion device, characterized in that, include: The first detection module is used to acquire first point cloud data collected by the first radar sensor; identify the first point cloud data, determine each first obstacle in the first point cloud data, and obtain the first obstacle detection result; The first obstacle is marked with a preset shape in the first radar coordinate system; Based on the center point of the preset shape, obtain the coordinate information of the representative points of each first obstacle in the first obstacle detection result; The second detection module is used to acquire the second point cloud data collected by the second radar sensor; Cluster the second point cloud data to identify each second obstacle in the second point cloud data and obtain the second obstacle detection result; based on the center point of each second obstacle, obtain the coordinate information of the representative point of each second obstacle in the second obstacle detection result; The projection module is used to project the representative points of the first obstacle and / or the representative points of the second obstacle according to the coordinate information, wherein the projected representative points of the first obstacle and the representative points of the second obstacle are in the same coordinate system. The matching module is used to match the representative points of the first obstacle and the representative points of the second obstacle in the same coordinate system to determine the target fusion obstacle to be matched. The fusion module is used to fuse the first obstacle detection result and the second obstacle detection result corresponding to the target fusion obstacle, and output the fusion result of the target fusion obstacle; The fusion module is further configured to, when the first radar sensor is a lidar sensor and the second radar sensor is a millimeter-wave radar sensor, determine the fusion coordinate information of the target fusion obstacle based on the coordinate information of the target fusion obstacle in the first obstacle detection result and the coordinate information of the target fusion obstacle in the second obstacle detection result; and fuse the target fusion obstacle based on the detection result of the target fusion obstacle in the first obstacle detection result, the fusion coordinate information, and the velocity information of the target fusion obstacle in the second obstacle detection result to obtain the fusion result of the target fusion obstacle.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

10. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as claimed in any one of claims 1 to 7.

Images