Obstacle fusion and model training method and electronic equipment

Abstract

The invention discloses an obstacle fusion method, a model training method and electronic equipment. The obstacle fusion method comprises the following steps: respectively pairing a plurality of target-level visual obstacles with a plurality of target-level millimeter wave radar obstacles to obtain a plurality of corresponding target obstacle matching pairs, inputting the target obstacle matching pairs into a target matching model to obtain a corresponding reasoning result, and according to the reasoning result, carrying out target-level millimeter wave radar obstacle fusion on the target-level visual obstacles. And under the condition of determining that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet a fusion condition, performing fusion processing on the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair to obtain obstacle fusion information. Therefore, the matching degree of the target-level visual obstacle and the target-level millimeter-wave radar obstacle for fusion can be improved, so that the precision of obstacle fusion information is improved, and the precision of obstacle detection is further improved.

Description

Technical Field

[0001] The present application relates to the field of assisted driving technology, and in particular to an obstacle fusion, model training method and electronic equipment. Background Art

[0002] Assisted driving relies on obstacle detection around the vehicle. Obstacle detection typically relies on multiple sensors to acquire information about obstacles around the vehicle, such as pedestrians and other vehicles. Sensors include visual cameras (also called cameras), lidar (LiDAR), and millimeter-wave radar. Different sensors have different advantages and disadvantages. For example, visual cameras can provide rich color and texture information about objects, but lack depth perception, and object recognition accuracy is easily affected by lighting. LiDAR provides precise three-dimensional (3D) information about objects, but is costly and sensitive to rain and dust. Millimeter-wave radar can operate in all weather conditions, but has low resolution. Therefore, obstacle detection using multiple sensors is often combined to compensate for the shortcomings of a single sensor, providing a more comprehensive environmental perception and enabling more accurate obstacle detection.

[0003] Currently, some vehicles often use a low-cost solution, abandoning LiDAR and adopting a solution based on the fusion of obstacle images obtained by visual cameras and millimeter-wave radar to achieve obstacle detection. Therefore, how to fuse obstacle images obtained by visual cameras and millimeter-wave radar to achieve more accurate obstacle detection is particularly important for better assisted driving in these vehicles. Summary of the Invention

[0004] The purpose of this application is to solve the problem of how to fuse obstacle images obtained based on visual cameras and millimeter-wave radar to achieve more accurate obstacle detection.

[0005] To address the above technical problems, embodiments of the present application disclose an obstacle fusion method, comprising: determining a plurality of target-level visual obstacles, the target-level visual obstacles being obtained based on visual information acquired by a visual camera; and determining a plurality of target-level millimeter-wave radar obstacles, the target-level millimeter-wave radar obstacles being obtained based on millimeter-wave radar point cloud information acquired by a millimeter-wave radar; pairing each target-level visual obstacle with each target-level millimeter-wave radar obstacle to obtain a corresponding plurality of target obstacle matching pairs; inputting each target obstacle matching pair into a target matching model so that the target matching model obtains a corresponding inference result based on the target obstacle matching pair, the inference result being used to indicate whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet a fusion condition; and if it is determined, based on the inference result, that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion condition, fusing the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair to obtain obstacle fusion information.

[0006] Using the above-mentioned obstacle fusion method, the target obstacle matching pairs formed by pairing the target-level visual obstacles with the target-level millimeter-wave radar obstacles are input into the target matching model to determine whether the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions. This can improve the matching degree of the fused target-level visual obstacles and the target-level millimeter-wave radar obstacles, thereby improving the accuracy of the obstacle fusion information and further improving the accuracy of obstacle detection, that is, achieving more accurate obstacle detection.

[0007] In addition, pairing each target-level visual obstacle with each target-level millimeter-wave radar obstacle can ensure that each target-level visual obstacle and each target-level millimeter-wave radar obstacle can be matched once.

[0008] In one embodiment of the obstacle fusion method provided herein, whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion condition can, for example, refer to whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are the same obstacle. Furthermore, if the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are the same obstacle, the fusion condition is met; if the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are different obstacles, the fusion condition is not met.

[0009] In this way, the matching degree between the target-level visual obstacles and the target-level millimeter-wave radar obstacles for fusion can be improved, thereby improving the accuracy of obstacle fusion information and further improving the accuracy of obstacle detection, that is, achieving more accurate obstacle detection.

[0010] Of course, whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion condition may refer to whether other conditions are met, which can be set as needed.

[0011] In one embodiment of the obstacle fusion method provided in the present application, the method further includes obtaining a target-level visual obstacle based on the visual information obtained by the visual camera in the following manner: obtaining a corresponding visual obstacle based on the visual information obtained by the visual camera, removing visual obstacles that are not within the target detection range from the visual obstacles, and obtaining a target-level visual obstacle.

[0012] By adopting the above obstacle fusion method, visual obstacles that are not within the target detection range are removed, and the presence of visual obstacles outside the target detection range in the target-level visual obstacles is prevented. This can ensure that the generated target-level visual obstacles are more accurate, and further ensure that the target-level visual obstacles are more closely matched with the target-level millimeter-wave radar obstacles.

[0013] In one embodiment of the obstacle fusion method provided in the present application, the method further includes obtaining a target-level millimeter-wave radar obstacle based on the millimeter-wave radar point cloud information obtained by the millimeter-wave radar in the following manner: obtaining a point cloud-level millimeter-wave radar obstacle based on the millimeter-wave radar point cloud information obtained by the millimeter-wave radar; performing point cloud clustering processing on the point cloud-level millimeter-wave radar obstacle to obtain multiple point cloud clustering results, and representing each point cloud clustering result with an anchor point; and performing tracking and matching processing on the anchor points to obtain multiple tracked target points, where the target points are target-level millimeter-wave radar obstacles.

[0014] The obstacle fusion method is adopted to obtain point cloud-level millimeter wave radar obstacles based on the millimeter wave radar point cloud information obtained by the millimeter wave radar. Point cloud-level millimeter wave radar obstacles are subjected to point cloud clustering processing to accurately obtain point cloud clustering results. Each point cloud clustering result is represented by an anchor point, and the anchor point is tracked and matched to accurately identify multiple target points. The target points are target-level millimeter wave radar obstacles, thereby ensuring that the obtained target-level millimeter wave radar obstacles are more accurate.

[0015] In one embodiment of the obstacle fusion method provided in the present application, each target-level visual obstacle itself has corresponding target-level visual obstacle information, and the target-level visual obstacle information includes target-level visual obstacle motion information; each target-level millimeter-wave radar obstacle itself has corresponding target-level millimeter-wave radar obstacle information, and the target-level millimeter-wave radar obstacle information includes target-level millimeter-wave radar obstacle motion information.

[0016] Furthermore, the target-level visual obstacle motion information may include multiple visual obstacle motion information, that is, multiple frames of historical motion information; the target-level millimeter-wave radar obstacle motion information may also include multiple millimeter-wave radar obstacle motion information, that is, multiple frames of historical motion information.

[0017] Therefore, the obstacle fusion method described above can be used to match and fuse multiple frames of historical motion information. This effectively avoids the problem of pairing and fusing single-frame motion information, which can lead to pairing and fusion errors and inaccurate fused information. This effectively ensures the accuracy of the final fused obstacle information.

[0018] Of course, the target-level visual obstacle information may also include obstacle identification information and speed and distance measurement information corresponding to the target-level visual obstacle, and the target-level millimeter-wave radar obstacle information may also include speed and distance measurement information corresponding to the target-level millimeter-wave radar obstacle.

[0019] Furthermore, the above-mentioned multi-frame historical motion information may be, for example, N frames of historical motion information corresponding to N frames of images, where N is a value such as 20, which may be set as needed.

[0020] In one embodiment of the obstacle fusion method provided in the present application, the target obstacle matching pair includes corresponding motion trajectory information of a target-level visual obstacle and motion trajectory information of a target-level millimeter-wave radar obstacle.

[0021] Using this obstacle fusion method, target obstacle matching pairs have corresponding obstacle motion trajectory information. Based on this obstacle motion trajectory information, it is possible to further determine whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle match, thereby ensuring more accurate obstacle fusion information. Furthermore, when fusing the target-level visual obstacle and the target-level millimeter-wave radar obstacle, the corresponding obstacle motion trajectory information is also integrated to further ensure the accuracy of the resulting obstacle fusion information.

[0022] In one embodiment of the obstacle fusion method provided in the present application, point cloud clustering processing is performed based on point cloud-level millimeter wave radar obstacles, including: performing point cloud clustering processing based on point cloud-level millimeter wave radar obstacles through a density-based clustering algorithm.

[0023] In one embodiment of the obstacle fusion method provided in the present application, tracking and matching the anchor points includes: matching the anchor points using a Hungarian matching algorithm and an identification information matching algorithm, wherein the identification information may be, for example, an ID.

[0024] The obstacle fusion method described above, combined with a density-based clustering algorithm for point cloud clustering, effectively improves the efficiency of millimeter-wave radar obstacle processing at the point cloud level. Furthermore, the resulting point cloud clustering results are more accurate. Anchor point matching using the Hungarian matching algorithm and the identification information matching algorithm effectively improves anchor point matching efficiency and reduces the number of unmatched anchor points.

[0025] In one embodiment of the obstacle fusion method provided in the present application, the target matching model is a fully connected matching model.

[0026] By adopting the above-mentioned obstacle fusion method and using the fully connected matching model as the target matching model, global feature extraction can be performed, and the fully connected matching model has a high nonlinear expression ability, thereby effectively improving the accuracy of the inference results (also called matching results) obtained based on the fully connected matching model reasoning.

[0027] In one embodiment of the obstacle fusion method provided in the present application, the visual camera is a vehicle-mounted camera, and the millimeter-wave radar is a vehicle-mounted millimeter-wave radar.

[0028] An embodiment of the present application also discloses a model training method, which includes: determining a training data set, the training data set including multiple positive training samples and multiple negative training samples, the positive training samples being obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that meet fusion conditions, and the negative training samples being obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that do not meet fusion conditions; performing model training based on the training data set to obtain a target matching model, the target matching model being used to detect whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions.

[0029] The above-mentioned model training method is used to train a target matching model based on a training data set including positive training samples and negative training samples, so as to effectively improve the accuracy of the trained target matching model in detecting whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions.

[0030] An embodiment of the present application also discloses an electronic device, which includes: a processor, a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the obstacle fusion method provided in any one of the above embodiments, or implements the model training method provided in any one of the above embodiments.

[0031] By using the above-mentioned electronic equipment, it is possible to determine whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle meet the fusion conditions through the target matching model, so as to improve the matching degree of the target-level visual obstacle and the target-level millimeter-wave radar obstacle to be fused, thereby improving the accuracy of the obstacle fusion information and further improving the accuracy of obstacle detection.

[0032] Furthermore, the above-mentioned electronic device can train a target matching model based on a training data set including positive training samples and negative training samples, so as to effectively improve the accuracy of detecting whether the target obstacle matching pair including the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions according to the trained target matching model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a flow chart of the obstacle fusion method provided in an embodiment of the present application;

[0034] Figure 2A A schematic diagram showing that target-level visual obstacles and target-level millimeter-wave radar obstacles have their own information in the obstacle fusion method provided in an embodiment of the present application;

[0035] Figure 2B A schematic diagram of target-level visual obstacle motion information in the target-level visual obstacle information and target-level millimeter-wave radar obstacle motion information in the target-level millimeter-wave radar obstacle information in the obstacle fusion method provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the obstacle trajectories corresponding to target-level visual obstacles and target-level millimeter-wave radar obstacles in the obstacle fusion method provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a process for obtaining target-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by the millimeter-wave radar in the obstacle fusion method provided in an embodiment of the present application;

[0038] Figure 5A schematic diagram of a flow chart for determining whether a target-level visual obstacle and a target-level millimeter-wave radar obstacle included in a target obstacle matching pair can be fused in the obstacle fusion method provided in an embodiment of the present application;

[0039] Figure 6 Another flowchart for determining whether a target-level visual obstacle and a target-level millimeter-wave radar obstacle included in a target obstacle matching pair can be fused in the obstacle fusion method provided in an embodiment of the present application;

[0040] Figure 7 A flow chart of a model training method provided in an embodiment of the present application;

[0041] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] As mentioned above, some vehicles currently use a solution that fuses obstacle images acquired by visual cameras and millimeter-wave radar to achieve obstacle detection. However, when using existing technologies to fuse obstacle images acquired by visual cameras and millimeter-wave radar to achieve obstacle detection and enable assisted driving, unresolvable edge cases and other issues can easily arise, affecting obstacle detection accuracy.

[0043] For example, existing technologies often use post-fusion methods to fuse obstacle detection data from visual cameras and millimeter-wave radars to achieve more accurate obstacle detection. Post-fusion methods involve fusing obstacle detection results from multiple sensors when an obstacle is detected. However, using existing post-fusion methods to fuse obstacle detection data from multiple sensors can easily lead to unresolvable edge cases. For example, at the edges of images captured by a visual camera and those captured by a millimeter-wave radar, object detection discrepancies can occur between the visual information obtained by the visual camera and the millimeter-wave radar point cloud information obtained by the millimeter-wave radar. For example, monocular ranging from visual cameras can lead to depth estimation errors, while millimeter-wave radars have lower resolution. In the Bird's-Eyes-View (BEV) coordinate system, if the error covariance characteristics of the visual information obtained by the visual camera and the millimeter-wave radar point cloud information are not considered, edge objects with inconsistent depths may appear. For example, when the millimeter-wave radar detects a distant object, the visual camera may mistake it for a nearby obstacle due to depth errors.

[0044] Therefore, how to fuse obstacles obtained based on visual cameras and millimeter-wave radar to achieve more accurate obstacle detection is a problem that is currently being explored in the field.

[0045] Based on the above problems, the embodiment of the present application discloses an obstacle fusion method, such as Figure 1 As shown, the obstacle fusion method includes the following steps.

[0046] Step S11 : determining a plurality of target-level visual obstacles, where the target-level visual obstacles are obtained based on visual information acquired by a visual camera.

[0047] Step S12: determining a plurality of target-level millimeter-wave radar obstacles, where the target-level millimeter-wave radar obstacles are obtained based on millimeter-wave radar point cloud information acquired by the millimeter-wave radar.

[0048] In step S13, each target-level visual obstacle is paired with each target-level millimeter-wave radar obstacle to obtain a corresponding plurality of target obstacle matching pairs.

[0049] In step S14, each target obstacle matching pair is input into the target matching model so that the target matching model obtains a corresponding inference result based on the target obstacle matching pair. The inference result is used to indicate whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions.

[0050] In step S15, when it is determined based on the inference result that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion condition, the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are fused to obtain obstacle fusion information.

[0051] Using the above-mentioned obstacle fusion method, the target obstacle matching pairs formed by pairing the target-level visual obstacles with the target-level millimeter-wave radar obstacles are input into the target matching model to determine whether the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions. This can improve the matching degree of the fused target-level visual obstacles and the target-level millimeter-wave radar obstacles, thereby improving the accuracy of the obstacle fusion information and further improving the accuracy of obstacle detection, that is, achieving more accurate obstacle detection.

[0052] In one embodiment of the obstacle fusion method provided in the present application, the specific steps of step S11 may include: obtaining image information such as a video recorded by a visual camera or multiple pictures taken, determining visual information based on the image information, and determining multiple target-level visual obstacles based on the visual information.

[0053] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, the visual information may include: the lateral and longitudinal distance measurements of the visual camera relative to the obstacle, the lateral and longitudinal speed measurements of the visual camera relative to the obstacle, the obstacle identification (ID) and obstacle category, and other information.

[0054] In one embodiment of the obstacle fusion method provided in the present application, the specific steps of step S12 may include: obtaining radar data obtained by a millimeter-wave radar, determining millimeter-wave radar point cloud information based on the radar data, and determining multiple target-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information.

[0055] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, the millimeter wave radar point cloud information may include: the lateral and longitudinal ranging of the millimeter wave radar relative to the obstacle, the lateral and longitudinal speed measurement of the millimeter wave radar relative to the obstacle, and other information.

[0056] Furthermore, in one embodiment of the obstacle fusion method provided in this application, each target-level visual obstacle corresponds to one obstacle, and each target-level millimeter-wave radar obstacle corresponds to one obstacle. Each frame of visual information can contain visual information of multiple obstacles, and each frame of millimeter-wave radar point cloud information can contain millimeter-wave radar point cloud information of multiple obstacles. The multiple obstacles involved in the visual information and the multiple obstacles involved in the millimeter-wave radar point cloud information can be the same obstacle.

[0057] In one embodiment of the obstacle fusion method provided herein, step S13 may include pairing the multiple target-level visual obstacles with the multiple target-level millimeter-wave radar obstacles, with each target-level visual obstacle being paired with each target-level millimeter-wave radar obstacle, to obtain corresponding target obstacle matching pairs. Based on the above steps, the multiple target-level visual obstacles can be paired with the multiple target-level millimeter-wave radar obstacles to obtain multiple target obstacle matching pairs.

[0058] In one embodiment of the obstacle fusion method provided in the present application, each target-level visual obstacle itself has corresponding target-level visual obstacle information, and the target-level visual obstacle information includes target-level visual obstacle motion information; each target-level millimeter-wave radar obstacle itself has corresponding target-level millimeter-wave radar obstacle information, and the target-level millimeter-wave radar obstacle information includes target-level millimeter-wave radar obstacle motion information.

[0059] For example, Figure 2AAs shown, multiple target-level visual obstacles include, for example, target-level visual obstacle 1 (referred to as visual 1), target-level visual obstacle 2 (referred to as visual 2), target-level visual obstacle 3 (referred to as visual 3), etc. Each target-level visual obstacle has corresponding target-level visual obstacle information. The target-level visual obstacle information includes target-level visual obstacle motion information (referred to as historical information, or historical motion information). The target-level visual obstacle information also includes obstacle identification information such as the aforementioned ID and speed and distance measurement information. In addition, multiple target-level millimeter-wave radar obstacles include, for example, target-level millimeter-wave radar obstacle 1 (referred to as Radar 1), target-level millimeter-wave radar obstacle 2 (referred to as Radar 2), target-level millimeter-wave radar obstacle 3 (referred to as Radar 3), target-level millimeter-wave radar obstacle 4 (referred to as Radar 4), etc. Each target-level millimeter-wave radar obstacle has corresponding target-level millimeter-wave radar obstacle information. The target-level millimeter-wave radar obstacle information includes target-level millimeter-wave radar obstacle motion information (referred to as historical information, or historical motion information). The target-level millimeter-wave radar obstacle information also includes, for example, the aforementioned speed and ranging information.

[0060] Furthermore, the target-level visual obstacle motion information may include multiple visual obstacle motion information, which is also multiple historical motion information. For example, the target-level visual obstacle motion information includes multiple historical motion information, such as N historical motion information whose generation time is closest to the target fusion time. In the case where N is 20, Figure 2B As shown in part (a), the target-level visual obstacle motion information includes the visual obstacle motion information obtained at time k (also known as time k information) and the historical visual obstacle motion information obtained at time k-1…k-18, and k-19 (also known as historical time data: k-1…k-18, k-19). Time k can be understood as the current time.

[0061] Furthermore, the target-level millimeter-wave radar obstacle motion information may include multiple millimeter-wave radar obstacle motion information, which is also multiple historical motion information. For example, the target-level millimeter-wave radar obstacle motion information includes multiple historical motion information, such as N historical motion information whose generation time is closest to the target fusion time. In the case where N is 20, Figure 2B As shown in part (b), the target-level millimeter-wave radar obstacle motion information includes the millimeter-wave radar obstacle motion information obtained at time k (i.e., the k-time information) and the historical millimeter-wave radar obstacle motion information obtained at times k-1…k-18, and k-19 (i.e., the historical data at time k-1…k-18, k-19). Time k can be understood as the current time.

[0062] Based on this, the target obstacle matching pair formed is a matching pair including the corresponding target-level visual obstacle and the target-level millimeter-wave radar obstacle itself with information.

[0063] Furthermore, the motion information may include information such as the position, speed, and corresponding timestamp of the obstacle when it is collected.

[0064] In one embodiment of the obstacle fusion method provided herein, step S14 may include inputting the obtained multiple target obstacle matching pairs into a target matching model, performing inference analysis on the target-level visual obstacles and target-level millimeter-wave radar obstacles, and obtaining corresponding inference results. Furthermore, the inference results indicate whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pairs meet fusion conditions. For example, the inference results may indicate that the target obstacle matching pairs are fusible or fusible. Of course, the inference results may also include other information used to implement the aforementioned indications.

[0065] Whether the target-level visual obstacle and target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions. For example, this can refer to whether the historical motion information included in the target obstacle matching pair corresponds to the motion information of the same obstacle. If so, the fusion conditions are met; otherwise, the fusion conditions are not met.

[0066] That is, if the obstacle corresponding to the target-level visual obstacle and the obstacle corresponding to the target-level millimeter-wave radar are the same obstacle, the inference result is that the target obstacle matching pair can be fused. Correspondingly, if the obstacle corresponding to the target-level visual obstacle and the obstacle corresponding to the target-level millimeter-wave radar are different obstacles, the inference result is that the target obstacle matching pair cannot be fused.

[0067] In this way, the matching degree between the target-level visual obstacles and the target-level millimeter-wave radar obstacles for fusion can be improved, thereby improving the accuracy of obstacle fusion information and further improving the accuracy of obstacle detection.

[0068] Of course, whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions may refer to whether other conditions are met, such as whether the timestamp information of the target-level visual obstacles included in the target obstacle matching pair is consistent with the timestamp information of the target-level millimeter-wave radar obstacle, etc., which can be set as needed.

[0069] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, the specific steps of step S14 may also include: the target matching model performs inference analysis on the aforementioned target-level visual obstacle information corresponding to the target-level visual obstacle in the target obstacle matching pair and the aforementioned target-level millimeter-wave radar obstacle information corresponding to the target-level millimeter-wave radar obstacle to obtain a corresponding inference result.

[0070] In one embodiment of the obstacle fusion method provided in the present application, the specific steps of step S15 may include: when the inference result is that the target obstacle matching pair can be fused, that is, when it can be determined that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions, the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are fused to obtain obstacle fusion information.

[0071] In addition, one embodiment of the obstacle fusion method provided herein further includes: if the inference result indicates that the target obstacle matching pair cannot be fused, that is, if, for example, the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are not the same obstacle, further pairing the unpaired target-level visual obstacle and the target-level millimeter-wave radar obstacle, or pairing the newly obtained target-level visual obstacle and the target-level millimeter-wave radar obstacle to obtain a new target obstacle matching pair, inputting the new target obstacle matching pair into the target matching model to obtain the corresponding inference result. In other words, if the inference result corresponding to the current frame indicates that the target obstacle matching pair cannot be fused, inference processing related to the next frame is performed.

[0072] Furthermore, the obstacle fusion method provided in the embodiments of the present application can be used to detect obstacles near a vehicle. Furthermore, the visual camera can be a vehicle-mounted camera, and the millimeter-wave radar can be a vehicle-mounted millimeter-wave radar.

[0073] In one embodiment of the obstacle fusion method provided in the present application, the method further includes obtaining a target-level visual obstacle based on the visual information obtained by the visual camera in the following manner: obtaining a corresponding visual obstacle based on the visual information obtained by the visual camera, removing visual obstacles that are not within the target detection range from the visual obstacles, and obtaining a target-level visual obstacle.

[0074] By adopting the above obstacle fusion method, visual obstacles that are not within the target detection range are removed, and the presence of visual obstacles that are not within the target detection range in the target-level visual obstacles is prevented. This can ensure that the generated target-level visual obstacles are more accurate, and further ensure that the target-level visual obstacles are more closely matched with the target-level millimeter-wave radar obstacles.

[0075] In one embodiment of the obstacle fusion method provided in the present application, a target detection range is preset in the vehicle. In the process of determining target-level visual obstacles based on visual information, visual obstacles in the visual information that are not within the target detection range are determined based on the preset target detection range, and visual obstacles in the visual information that are not within the target detection range are filtered out to obtain target-level visual obstacles.

[0076] Furthermore, the target detection range can be limited based on information such as the vehicle situation and user needs, and is not limited here.

[0077] Furthermore, in one embodiment of the obstacle fusion method provided in this application, step S11 and step S12 can be performed simultaneously, or step S11 can be performed first and then step S12, or step S12 can be performed first and then step S11. Specifically, the order of performing steps S11 and S12 can be determined based on the specific application scenario and is not limited here.

[0078] In one embodiment of the obstacle fusion method provided in the present application, the method further includes obtaining a target-level millimeter-wave radar obstacle based on the millimeter-wave radar point cloud information obtained by the millimeter-wave radar in the following manner: obtaining a point cloud-level millimeter-wave radar obstacle based on the millimeter-wave radar point cloud information obtained by the millimeter-wave radar; performing point cloud clustering processing on the point cloud-level millimeter-wave radar obstacle to obtain multiple point cloud clustering results, and representing each point cloud clustering result with an anchor point; and performing tracking and matching processing on the anchor points to obtain multiple tracked target points, where the target points are target-level millimeter-wave radar obstacles.

[0079] The obstacle fusion method is adopted to obtain point cloud-level millimeter wave radar obstacles based on the millimeter wave radar point cloud information obtained by the millimeter wave radar. Point cloud-level millimeter wave radar obstacles are subjected to point cloud clustering processing to accurately obtain point cloud clustering results. Each point cloud clustering result is represented by an anchor point, and the anchor point is tracked and matched to accurately identify multiple target points. The target points are target-level millimeter wave radar obstacles, thereby ensuring that the obtained target-level millimeter wave radar obstacles are more accurate.

[0080] In one embodiment of the obstacle fusion method provided herein, when a point cloud-level millimeter-wave radar obstacle is obtained, each cluster of point clouds in the point cloud-level millimeter-wave radar obstacle can be clustered using a clustering algorithm to obtain a clustering result corresponding to each cluster of point clouds. An anchor point is then selected from the clustering results to represent the clustering result of each cluster of point clouds. Tracking and matching processing is then performed on the anchor points using a tracking and matching algorithm to obtain multiple tracked target points, which are the target-level millimeter-wave radar obstacles.

[0081] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, when multiple target points are obtained after tracking, target points that are not in the preset target detection range can be removed to ensure the accuracy of obtaining target-level millimeter-wave radar obstacles.

[0082] In one embodiment of the obstacle fusion method provided herein, the target-level visual obstacle motion information corresponding to each target-level visual obstacle may include multiple visual obstacle motion information pieces. These multiple visual obstacle motion information pieces may be stored in a target-level visual obstacle historical information array corresponding to each target-level visual obstacle. The multiple visual obstacle motion information pieces are referred to as multiple historical motion information pieces. The target-level millimeter-wave radar obstacle motion information corresponding to each target-level millimeter-wave radar obstacle may include multiple millimeter-wave radar obstacle motion information pieces. These multiple millimeter-wave radar obstacle motion information pieces are stored in a target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle. The multiple millimeter-wave radar obstacle motion information pieces are referred to as multiple historical motion information pieces. The number of multiple historical motion information pieces included in the target-level visual obstacle motion information corresponding to each target-level visual obstacle may be the same as or different from the number of multiple historical motion information pieces included in the target-level millimeter-wave radar obstacle operation information corresponding to each target-level millimeter-wave radar obstacle.

[0083] The obstacle fusion method described above matches and fuses multiple frames of obstacle motion information. This effectively avoids the matching and fusion errors and inaccurate fused information that can occur when matching and fusing a single frame of obstacle motion information. Therefore, the accuracy of the resulting fused obstacle information is effectively guaranteed.

[0084] Furthermore, the time information of the visual obstacle motion information and the millimeter-wave radar obstacle motion information stored at the same time is consistent, thereby ensuring that the timestamps of the target-level visual obstacles and the target-level millimeter-wave radar obstacles are aligned, ensuring that the target-level visual obstacles and the target-level millimeter-wave radar obstacles can match, thereby further ensuring the accuracy of the obstacle fusion information obtained by fusion.

[0085] In one embodiment of the obstacle fusion method provided in the present application, a visual camera acquires visual obstacle motion information of a visual obstacle in real time, and stores the acquired visual obstacle motion information in a target-level visual obstacle history information array corresponding to the target-level visual obstacle for easy retrieval.

[0086] Furthermore, in one embodiment of the obstacle fusion method provided herein, the visual obstacle motion information stored in the target-level visual obstacle historical information array corresponding to each target-level visual obstacle can be obtained in the following manner: when the first visual obstacle motion information for a target-level visual obstacle is initially acquired, the target-level visual obstacle historical information array corresponding to the target-level visual obstacle is empty. In this case, the real-time acquired visual obstacle motion information corresponding to the target-level visual obstacle is continuously stored in the corresponding target-level visual obstacle historical information array until the amount of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array reaches N. Furthermore, during the storage process, the corresponding visual obstacle motion information is obtained based on the visual information acquired in real time by the visual camera, and it is determined whether the amount of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is greater than or equal to N. If the amount of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is less than N, the acquired visual obstacle motion information is stored in the corresponding target-level visual obstacle historical information array. When the amount of visual obstacle motion information stored in the corresponding target-level visual obstacle history information array is equal to or greater than N, the earliest visual obstacle motion information stored in the corresponding target-level visual obstacle history information array is deleted until the amount of visual obstacle motion information stored in the corresponding target-level visual obstacle history information array is N-1, so that the newly obtained visual obstacle motion information can be stored in the corresponding target-level visual obstacle history information array.

[0087] That is, at the initial moment, the target-level visual obstacle history information array corresponding to each target-level visual obstacle (i.e., the target-level visual obstacle history information array) is empty. When the first frame of visual obstacle motion information corresponding to a target-level visual obstacle is obtained, the first frame of visual obstacle motion information (i.e., the first frame of data) is stored at the beginning of the target-level visual obstacle history information array corresponding to the target-level visual obstacle, and the position of the first frame of visual obstacle motion information in the target-level visual obstacle history information array is recorded. When the number of visual obstacle motion information stored in the target-level visual obstacle history information array reaches N, when the next frame of visual obstacle motion information is stored, if the array is full, the old data (i.e., the visual obstacle motion information with the earliest storage time) is discarded to ensure that the multiple visual obstacle motion information stored in the target-level visual obstacle history information array are the N visual obstacle motion information whose generation time is closest to the target fusion time.

[0088] Furthermore, in one embodiment of the obstacle fusion method provided herein, the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array corresponding to each target-level millimeter-wave radar obstacle can be obtained in the following manner: when the first millimeter-wave radar obstacle motion information for a target-level millimeter-wave radar obstacle is initially acquired, the target-level millimeter-wave radar obstacle history information array corresponding to the target-level millimeter-wave radar obstacle is empty. In this case, the real-time acquired millimeter-wave radar obstacle motion information corresponding to the target-level millimeter-wave radar obstacle is continuously stored in the corresponding target-level millimeter-wave radar obstacle history information array until the amount of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array equals N. Furthermore, during the storage process, the corresponding millimeter-wave radar obstacle motion information is obtained based on the millimeter-wave radar point cloud information acquired in real time by the millimeter-wave radar, and it is determined whether the amount of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array is greater than or equal to N. If the amount of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array is less than N, the acquired millimeter-wave radar obstacle motion information is stored in the corresponding target-level millimeter-wave radar obstacle history information array. If the amount of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array is equal to or greater than N, the earliest millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array is deleted until the amount of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array is N-1, so that the newly acquired millimeter-wave radar obstacle motion information can be stored in the corresponding target-level millimeter-wave radar obstacle history information array.

[0089] That is, at the initial moment, the target-level millimeter-wave radar obstacle history information array corresponding to each target-level millimeter-wave radar obstacle (i.e., the target-level millimeter-wave radar obstacle history information array) is empty. When the first frame of millimeter-wave radar obstacle motion information for a target-level millimeter-wave radar obstacle is obtained, the first frame of millimeter-wave radar obstacle motion information (i.e., the first frame of data) is stored at the beginning of the corresponding target-level millimeter-wave radar obstacle history information array, and the position of the first frame of millimeter-wave radar obstacle motion information in the corresponding target-level millimeter-wave radar obstacle history information array is recorded. When the number of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array reaches N, when the next frame of millimeter-wave radar obstacle motion information is stored, if the array is full, the old data (i.e., the millimeter-wave radar obstacle motion information with the earliest storage time) is discarded to ensure that the multiple millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle history information array are the N millimeter-wave radar obstacle motion information whose generation time is closest to the target fusion time.

[0090] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, when N is 20, as described above, Figure 2B As shown in part (a), the target-level visual obstacle history information array stores visual obstacle motion information, including, for example, motion information obtained at time k, as well as historical motion information obtained at time k-1, k-18, and k-19. Time k can be understood as the current time.

[0091] Furthermore, in one embodiment of the obstacle fusion method provided in the present application, when N is 20, as described above, Figure 2B As shown in part (b), the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array includes, for example, motion information obtained at time k and historical motion information obtained at times k-1, k-18, and k-19. Time k can be understood as the current time.

[0092] In one embodiment of the obstacle fusion method provided in the present application, the target obstacle matching pair includes corresponding motion trajectory information of a target-level visual obstacle and motion trajectory information of a target-level millimeter-wave radar obstacle.

[0093] Using the aforementioned obstacle fusion method, a target obstacle matching pair includes the corresponding motion trajectory information of the target-level visual obstacle and the motion trajectory information of the target-level millimeter-wave radar obstacle. This allows for further determination of whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle match based on the corresponding motion trajectory information, thereby ensuring more accurate fused obstacle information. Furthermore, when fusing the target-level visual obstacle and the target-level millimeter-wave radar obstacle, the corresponding motion trajectory information is also integrated to further ensure the accuracy of the resulting fused obstacle information.

[0094] In one embodiment of the obstacle fusion method provided herein, when a target-level visual obstacle and a target-level millimeter-wave radar obstacle are matched to obtain a target obstacle matching pair, the target obstacle matching includes: motion trajectory information corresponding to the target-level visual obstacle and motion trajectory information corresponding to the target-level millimeter-wave radar obstacle. When the target obstacle matching pair is input into a target matching model, the target matching model infers the motion trajectory information corresponding to the target-level visual obstacle and the motion trajectory information corresponding to the target-level millimeter-wave radar obstacle, and outputs a corresponding inference result.

[0095] In one embodiment of the obstacle fusion method provided in this application, a real vehicle can record data for example for one hour, during which the motion information of the target-level visual obstacle and the motion information of the target-level millimeter-wave radar obstacle can be recorded simultaneously, and the corresponding information can be visualized using a visualization tool. Figure 3 The figure shows the information displayed at a specific moment, where Track 1 is the motion track information (i.e., visual track points) corresponding to the target-level visual obstacle, and Track 2 is the motion track information (i.e., radar track points) corresponding to the target-level millimeter-wave radar obstacle. Furthermore, once Track 1 and Track 2 are determined, Track 1 and Track 2 can be fused to obtain the motion track information corresponding to the target obstacle matching pair as the target obstacle matching pair.

[0096] In one embodiment of the obstacle fusion method provided in this application, point cloud clustering is performed based on point cloud-level millimeter wave radar obstacles, including: performing point cloud clustering based on point cloud-level millimeter wave radar obstacles using a density-based clustering algorithm; and tracking and matching anchor points is performed, including: matching anchor points using a Hungarian matching algorithm and an identification information matching algorithm. The identification information is, for example, an ID.

[0097] The obstacle fusion method described above, combined with a density-based clustering algorithm for point cloud clustering, effectively improves the efficiency of millimeter-wave radar obstacle processing at the point cloud level. Furthermore, the resulting point cloud clustering results are more accurate. Anchor point matching using the Hungarian matching algorithm and the identification information matching algorithm effectively improves anchor point matching efficiency and reduces the number of unmatched anchor points.

[0098] like Figure 4 As shown, in one embodiment of the obstacle fusion method provided in the present application, obtaining a target-level millimeter-wave radar obstacle according to millimeter-wave radar point cloud information obtained by the millimeter-wave radar includes the following steps.

[0099] Step S121: Obtain point cloud-level millimeter wave radar obstacles based on millimeter wave radar point cloud information acquired by the millimeter wave radar.

[0100] Furthermore, step S121 may be to determine a point cloud-level millimeter wave radar obstacle based on the vehicle-mounted radar to obtain a point cloud-level millimeter wave radar obstacle.

[0101] In step S122 , a density-based clustering algorithm (DBSCAN) is used to perform point cloud clustering processing on the point cloud-level millimeter wave radar obstacles to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point.

[0102] Furthermore, step S122 is to perform clustering using a density-based clustering algorithm (DBSCAN).

[0103] In step S123 , the anchor point is tracked and matched by using the Hungarian matching algorithm and the ID matching algorithm to obtain a target-level millimeter-wave radar obstacle.

[0104] Furthermore, step S123 is to track according to the Hungarian matching algorithm and the ID matching algorithm to determine the target-level millimeter-wave radar obstacle.

[0105] In one embodiment of the obstacle fusion method provided in the present application, the target matching model is a fully connected matching model.

[0106] By adopting the above obstacle fusion method, the fully connected matching model is used as the target matching model to perform global feature extraction, and the fully connected matching model has a high nonlinear expression ability, which can effectively improve the accuracy of the matching results obtained based on the fully connected matching model.

[0107] Furthermore, if Figure 5 and Figure 6As shown, in one embodiment of the obstacle fusion method provided in this application, taking the case where the number of target-level visual obstacles and target-level millimeter-wave radar obstacles is N (i.e., the number of visual obstacles and millimeter-wave radar obstacles is N, although the number of these obstacles can also be different), and True indicating that the target obstacle matching pair can be fused, and False indicating that the target obstacle matching pair cannot be fused, as an example, the method will be used to determine whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair can be fused. Specifically, the method includes the following process and steps.

[0108] The visual camera acquires multiple visual information in real time or periodically, and obtains N target-level visual obstacles based on the visual information acquired by the visual camera. The target-level visual obstacle history information array corresponding to each target-level visual obstacle stores, for example, 20 frames of visual obstacle motion information.

[0109] Furthermore, obtaining N target-level visual obstacles and obtaining 20 frames of visual obstacle motion information stored in the target-level visual obstacle history information array corresponding to each target-level visual obstacle specifically includes the following steps.

[0110] Step S100: A visual camera acquires visual information.

[0111] In step S101, corresponding visual obstacles are obtained according to the visual information acquired by the visual camera, and visual obstacles that are not within the target detection range are removed from the visual obstacles to obtain N target-level visual obstacles.

[0112] Furthermore, the visual camera may be a vehicle-mounted camera. The target detection range may be the vehicle's arrival area, or an area within a preset distance around the vehicle, determined based on the vehicle's driving information, to facilitate detection of obstacles within the vehicle's arrival area or within a preset distance around the vehicle. Furthermore, obstacles may include traffic signs, pedestrians, surrounding vehicles, and other obstacles that impede the vehicle's movement or parking.

[0113] Step S102: storing or saving the visual obstacle motion information of the target-level visual obstacle obtained at the current moment into a historical information storage array corresponding to the target-level visual obstacle, that is, into the corresponding target-level visual obstacle historical information array.

[0114] Step S103: Determine whether the number of visual obstacle motion information stored in the target-level visual obstacle history information array is greater than or equal to 20. If yes, execute step S104; if not, execute step S105.

[0115] That is, step S103 may be to determine whether the historical information array is greater than 20.

[0116] Step S104: If yes, delete the earliest visual obstacle motion information stored in the target-level visual obstacle history information array, and store the visual obstacle motion information obtained at the current moment into the target-level visual obstacle history information array.

[0117] Furthermore, in step S104, if the number of visual obstacle motion information stored in the target-level visual obstacle historical information array is determined to be greater than or equal to 20, the visual obstacle motion information stored in the target-level visual obstacle historical information array needs to be updated. In this case, one frame of historical data in the target-level visual obstacle historical information array is discarded to allow for the storage of new data.

[0118] Step S105: If not, the visual obstacle motion information obtained at the current moment is stored in the target-level visual obstacle history information array in order.

[0119] Furthermore, the millimeter-wave radar may acquire multiple millimeter-wave radar point cloud information in real time or periodically, obtain N target-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information, and obtain, for example, 20 frames of millimeter-wave radar obstacle motion information stored in a target-level millimeter-wave radar obstacle history information array corresponding to each target-level millimeter-wave radar obstacle.

[0120] Furthermore, obtaining N target-level millimeter-wave radar obstacles and 20 frames of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array corresponding to each target-level millimeter-wave radar obstacle specifically includes the following steps.

[0121] Step S106: Obtaining point cloud-level millimeter wave radar obstacles based on the millimeter wave radar point cloud information acquired by the millimeter wave radar.

[0122] Furthermore, step S106 may be to determine the point cloud level millimeter wave radar obstacle based on the vehicle-mounted millimeter wave radar to obtain the point cloud level millimeter wave radar obstacle.

[0123] In step S107 , point cloud clustering processing is performed on the point cloud-level millimeter wave radar obstacles by using a density-based clustering algorithm (DBSCAN) to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point.

[0124] Furthermore, step S107 is to perform clustering using a density-based clustering algorithm (DBSCAN).

[0125] In step S108 , the anchor point is tracked and matched by using the Hungarian matching algorithm and the ID matching algorithm to obtain a target-level millimeter-wave radar obstacle.

[0126] Furthermore, step S108 is to perform tracking according to the Hungarian matching algorithm and the ID matching algorithm to determine the target-level millimeter-wave radar obstacle.

[0127] Step S109: storing the obtained millimeter-wave radar obstacle motion information corresponding to the target-level millimeter-wave radar obstacle into the historical information of the target-level millimeter-wave radar obstacle.

[0128] Step S110: Determine whether the number of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array corresponding to the target-level millimeter-wave radar obstacle is greater than or equal to 20. If so, execute step S111; if not, execute step S112.

[0129] Step S111: If yes, delete the earliest millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array, and store the millimeter-wave radar obstacle motion information obtained at the current moment into the target-level millimeter-wave radar obstacle history information array.

[0130] Furthermore, in step S111, if the number of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array is determined to be greater than or equal to 20, the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array needs to be updated. In this case, one frame of historical data in the target-level millimeter-wave radar obstacle history information array is removed to store the corresponding data.

[0131] Step S112: If not, the millimeter-wave radar obstacle motion information obtained at the current moment is stored in the target-level millimeter-wave radar obstacle history information array in sequence.

[0132] Furthermore, the target-level millimeter-wave radar obstacle history information array for each target-level millimeter-wave radar obstacle may be a history information array of motion information of the target-level millimeter-wave radar obstacle. Furthermore, in step S112, the currently acquired millimeter-wave radar obstacle motion information may be stored in the corresponding target-level millimeter-wave radar obstacle history information array in the order in which the millimeter-wave radar obstacle motion information was acquired.

[0133] After determining multiple target-level visual obstacles and 20 frames of visual obstacle motion information stored in the target-level visual obstacle historical information array corresponding to each target-level visual obstacle according to the above steps, and determining multiple target-level millimeter-wave radar obstacles and 20 frames of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle, calling the above information to determine whether the corresponding target obstacle matching pairs can be fused specifically includes the following steps.

[0134] In step S113 , the N target-level visual obstacles and the N target-level millimeter-wave radar obstacles are paired in pairs to obtain a plurality of corresponding target obstacle matching pairs.

[0135] Since the target-level visual obstacle itself has the corresponding aforementioned target-level visual obstacle information, and the target-level millimeter-wave radar obstacle itself has the corresponding aforementioned target-level millimeter-wave radar obstacle information, pairing the target-level visual obstacle and the target-level millimeter-wave radar obstacle can be understood as pairing the target-level visual obstacle information and the target-level millimeter-wave radar obstacle information.

[0136] In step S114 , each target obstacle matching pair is input into the target matching model, so that the target matching model outputs an inference result indicating that the target obstacle matching pair can be fused (ie, True) or that the target obstacle matching pair cannot be fused (ie, False) based on the target obstacle matching pair.

[0137] Furthermore, in step S114, the target matching model may be a fully connected matching model.

[0138] In step S115 , when the target matching model outputs that the target obstacle matching pair can be fused (ie, True), it can be determined that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair can be fused.

[0139] Furthermore, after executing step S115, the target-level visual obstacles and the target-level millimeter-wave radar obstacles included in the target obstacle matching pair are fused to obtain obstacle fusion information.

[0140] In step S116, when the target matching model outputs that the target obstacle matching pair cannot be fused (ie, False), it can be determined that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair cannot be fused.

[0141] Furthermore, when paired target-level visual obstacles and target-level millimeter-wave radar obstacles, for example, target-level visual obstacles and target-level millimeter-wave radar obstacles corresponding to the same obstacle at the same time, it can be considered that the target-level visual obstacles and target-level millimeter-wave radar obstacles can be fused; otherwise, it can be considered that the corresponding target-level visual obstacles and target-level radar point cloud obstacles cannot be fused.

[0142] By using the above-mentioned obstacle fusion method, a target obstacle matching pair including a target-level visual obstacle and a target-level millimeter-wave radar obstacle is input into the target matching model to determine whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle meet the fusion conditions. This can improve the matching degree of the target-level visual obstacle and the target-level millimeter-wave radar obstacle to be fused, thereby improving the accuracy of the obstacle fusion information and further improving the accuracy of obstacle detection.

[0143] Furthermore, while ensuring that the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions, the corresponding target-level visual obstacles and target-level millimeter-wave radar obstacles are fused to generate corresponding obstacle fusion information, which can ensure that the generated obstacle fusion information is more accurate.

[0144] like Figure 7 As shown, the embodiment of the present application also discloses a model training method, which includes the following steps.

[0145] Step S31: Determine a training data set, where the training data set includes multiple positive training samples and multiple negative training samples. The positive training samples are obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that meet fusion conditions, and the negative training samples are obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that do not meet fusion conditions.

[0146] Step S32: Perform model training based on the training data set to obtain a target matching model. The target matching model is used to detect whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions.

[0147] Specifically, the motion information of target-level visual obstacles with trajectories and target-level millimeter-wave radar obstacles with trajectories can be recorded in stages or simultaneously to generate a training dataset. This training dataset is then fed into, for example, a fully connected matching model for training, generating a target matching model as the training result for use in real-vehicle reasoning.

[0148] In real-vehicle reasoning, as described in the obstacle fusion method section above, at the target fusion time, there will be N target-level visual obstacles and N target-level millimeter-wave radar obstacles, which are paired into matching pairs and input into the target matching model to obtain the corresponding reasoning results.

[0149] By adopting the above-mentioned model training method, a target matching model is obtained by training a training data set including positive training samples and negative training samples, which can effectively improve the accuracy of the trained target matching model in detecting whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions.

[0150] Furthermore, in one embodiment of the model training method provided herein, positive training samples in the training dataset can be manually selected obstacle matching pairs, such as a target-level visual obstacle and a target-level millimeter-wave radar obstacle corresponding to the same obstacle, that meet the fusion criteria. Negative samples can be randomly generated obstacle matching pairs, such as a target-level visual obstacle and a target-level millimeter-wave radar obstacle corresponding to different obstacles, that do not meet the fusion criteria.

[0151] Furthermore, in one embodiment of the model training method provided in the present application, the ratio of the number of positive training samples to the number of negative training samples in the training data set can be, for example, 1:3.

[0152] Furthermore, the ratio of the number of positive training samples to the number of negative training samples in the training data set is merely an example and is not limited here, and the user can determine it according to actual needs.

[0153] Furthermore, the camera-related data and millimeter-wave radar-related data obtained above can be saved in a tool for recording and replaying data, such as rosbag, for corresponding presentation and use. And the corresponding camera-related data and millimeter-wave radar-related data in rosbag can be converted into a file format that is easy to read and write, such as a json file. The information in the json file is input into the fully connected matching model for training. The weights of the trained model can be saved as a txt file and deployed to the domain control for inference. Multi-frame millimeter-wave point cloud information and multi-frame camera obstacle information are input into the model. The model gives true or false to determine whether the input point cloud information and camera information are an obstacle to determine whether the fusion conditions are met.

[0154] like Figure 8 As shown, an embodiment of the present application also discloses an electronic device, which includes: a transceiver 121, a processor 122, and a memory 123 communicatively connected to the processor 122; the memory 123 stores computer-executable instructions; the processor 122 executes the computer-executable instructions stored in the memory 123, so that the electronic device implements the obstacle fusion method provided in any one of the above embodiments, or implements the model training method provided in any one of the above embodiments.

[0155] The electronic device described above can use a target matching model to determine whether target-level visual obstacles and target-level millimeter-wave radar obstacles meet fusion conditions. This ensures that the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair are more closely matched, thereby ensuring that the obstacle fusion information obtained based on the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair is more accurate. Based on the target-level visual obstacles and target-level millimeter-wave radar obstacles that meet the fusion conditions, corresponding obstacle fusion information is generated to ensure that the generated obstacle fusion information is more accurate.

[0156] Furthermore, the above-mentioned electronic device can train a target matching model based on a training data set including positive training samples and negative training samples, so as to effectively improve the accuracy of detecting whether the target obstacle matching pair including the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions according to the trained target matching model.

[0157] Furthermore, processor 122 executes computer-executable instructions stored in memory 123, causing processor 122 to implement the obstacle fusion method or model training method described in the above-mentioned embodiments. Processor 122 can be a general-purpose processor, including a central processing unit (CPU) or a network processor (NP). It can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component.

[0158] The memory 123 is connected to the processor 122 via a system bus and communicates with the processor 122. The memory 123 is used to store computer program instructions.

[0159] Furthermore, in one embodiment of the electronic device of the present application, the electronic device may be a vehicle, a computer, a mobile phone or other electronic device.

[0160] In one embodiment of the present application, the electronic device may further include a transceiver 121. The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0161] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. System buses can be divided into address buses, data buses, and control buses. For ease of illustration, the diagram uses only a single thick line, but this does not imply a single bus or type of bus. Transceivers enable communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and non-volatile memory.

[0162] Another embodiment of the present application further discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the obstacle fusion method provided in any one of the above embodiments, or implements the model training method provided in any one of the above embodiments.

[0163] Another embodiment of the present application further discloses a computer program product, including a computer program, which, when executed by a processor, implements the obstacle fusion method provided in any of the above embodiments, or implements the model training method provided in any of the above embodiments.

[0164] It should be noted that, in addition to the implementation methods of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this application are limited to this implementation method. On the contrary, the purpose of introducing the application in conjunction with the implementation method is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other in the absence of conflict.

[0165] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0166] In the description of this embodiment, it should be noted that the terms "upper", "lower", "inner", "bottom", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the product of the application is usually placed when in use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0167] The terms “first”, “second”, etc. are only used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0168] In the description of this embodiment, it should be noted that, unless otherwise specified or limited, the terms "disposed," "connected," and "connected" should be understood broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this embodiment based on specific circumstances.

[0169] Although the present application has been illustrated and described with reference to certain preferred embodiments of the present application, those skilled in the art will appreciate that the above descriptions are provided as further details of the present application in conjunction with specific embodiments, and that the specific implementation of the present application should not be limited to these descriptions. Those skilled in the art may make various changes in form and detail, including simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. An obstacle fusion method, characterized in that: The method comprises: determining a plurality of target-level visual obstacles, wherein the target-level visual obstacles are obtained based on visual information acquired by a visual camera; and determining a plurality of target-level millimeter-wave radar obstacles, where the target-level millimeter-wave radar obstacles are obtained based on millimeter-wave radar point cloud information acquired by the millimeter-wave radar; Pairing each of the target-level visual obstacles with each of the target-level millimeter-wave radar obstacles to obtain a corresponding plurality of target obstacle matching pairs; Inputting each target obstacle matching pair into a target matching model, so that the target matching model obtains a corresponding inference result based on the target obstacle matching pair, wherein the inference result is used to indicate whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet a fusion condition; When it is determined, based on the inference result, that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet a fusion condition, a fusion process is performed on the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair to obtain obstacle fusion information.

2. The method according to claim 1, characterized in that The method further includes obtaining the target-level visual obstacle based on the visual information acquired by the visual camera in the following manner: According to the visual information acquired by the visual camera, corresponding visual obstacles are obtained, and the visual obstacles that are not within the target detection range are removed from the visual obstacles to obtain the target-level visual obstacles.

3. The method according to claim 2, characterized in that The method further includes obtaining the target-level millimeter-wave radar obstacle based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar in the following manner: Obtaining a point cloud-level millimeter-wave radar obstacle according to the millimeter-wave radar point cloud information acquired by the millimeter-wave radar; Performing point cloud clustering processing according to the point cloud-level millimeter wave radar obstacle to obtain multiple point cloud clustering results, and representing each point cloud clustering result with an anchor point; Tracking and matching processing is performed on the anchor point to obtain multiple tracked target points, where the target points are the target-level millimeter-wave radar obstacles.

4. The method according to claim 3, characterized in that Each of the target-level visual obstacles has corresponding target-level visual obstacle information, and the target-level visual obstacle information includes target-level visual obstacle motion information; Each of the target-level millimeter-wave radar obstacles has corresponding target-level millimeter-wave radar obstacle information, and the target-level millimeter-wave radar obstacle information includes target-level millimeter-wave radar obstacle motion information.

5. The method according to claim 4, characterized in that The target obstacle matching pair includes corresponding motion trajectory information of the target-level visual obstacle and motion trajectory information of the target-level millimeter-wave radar obstacle.

6. The method according to claim 5, characterized in that Point cloud clustering processing is performed based on the point cloud-level millimeter wave radar obstacle, including: Performing point cloud clustering processing according to the point cloud-level millimeter wave radar obstacles by using a density-based clustering algorithm; Tracking and matching the anchor point includes: The anchor points are matched using the Hungarian matching algorithm and the identification information matching algorithm.

7. The method according to claim 6, characterized in that The target matching model is a fully connected matching model.

8. The method according to any one of claims 1 to 7, characterized in that The visual camera is a vehicle-mounted camera, and the millimeter-wave radar is a vehicle-mounted millimeter-wave radar.

9. A model training method, characterized in that: The method comprises: Determine a training data set, where the training data set includes a plurality of positive training samples and a plurality of negative training samples, where the positive training samples are obstacle matching pairs formed by a target-level visual obstacle and a target-level millimeter-wave radar obstacle that meet a fusion condition, and the negative training samples are obstacle matching pairs formed by a target-level visual obstacle and a target-level millimeter-wave radar obstacle that do not meet the fusion condition; Model training is performed according to the training data set to obtain a target matching model, where the target matching model is used to detect whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in a target obstacle matching pair meet a fusion condition.

10. An electronic device, characterized in that: The electronic device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory so that the electronic device implements the obstacle fusion method according to any one of claims 1 to 8, or implements the model training method according to claim 9.

Images