Obstacle fusion, model training method and electronic device

Abstract

The application discloses an obstacle fusion, a model training method and an electronic device. The obstacle fusion method comprises the following steps: pairing a plurality of target-level visual obstacles with a plurality of target-level millimeter wave radar obstacles respectively to obtain a plurality of corresponding target obstacle matching pairs; inputting the target obstacle matching pairs into a target matching model to obtain corresponding inference results; and in the case that the target-level visual obstacles and the target-level millimeter wave radar obstacles included in the target obstacle matching pairs meet the fusion condition according to the inference results, performing fusion processing on the target-level visual obstacles and the target-level millimeter wave radar obstacles included in the target obstacle matching pairs to obtain obstacle fusion information. In this way, the matching degree of the target-level visual obstacles and the target-level millimeter wave radar obstacles subjected to fusion can be improved, the accuracy of the obstacle fusion information is improved, and the accuracy of obstacle detection is further improved.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to an obstacle fusion, model training method and electronic device. Background Technology

[0002] The realization of assisted driving in vehicles relies heavily on obstacle detection around the vehicle. Obstacle detection typically depends on multiple sensors to acquire information about obstacles such as pedestrians and other vehicles. Sensors include visual cameras (also known as cameras), LiDAR, and millimeter-wave radar. Different sensors have different advantages and disadvantages. For example, visual cameras can provide rich color and texture information about objects, but lack depth perception capabilities, and their object recognition accuracy is easily affected by lighting conditions. LiDAR can provide precise three-dimensional (3D) information about objects, but it is costly and sensitive to rain and dust. Millimeter-wave radar can operate in all weather conditions, but its object resolution is relatively low. Therefore, current methods typically compensate for the shortcomings of a single sensor by fusing obstacle information acquired from multiple sensors, providing more comprehensive environmental perception and achieving more accurate obstacle detection.

[0003] Currently, some vehicles typically employ a low-cost solution, which forgoes LiDAR and instead uses a fusion approach based on visual cameras and millimeter-wave radar to achieve obstacle detection. Therefore, how to fuse obstacle data acquired by visual cameras and millimeter-wave radar to achieve more accurate obstacle detection is particularly important for these vehicles to better enable assisted driving. Summary of the Invention

[0004] The purpose of this application is to solve the problem of how to fuse obstacle data acquired by visual cameras and millimeter-wave radar to achieve more accurate obstacle detection.

[0005] To address the aforementioned technical problems, this application discloses an obstacle fusion method. The method includes: determining multiple target-level visual obstacles, obtained from visual information acquired by a visual camera; determining multiple target-level millimeter-wave radar obstacles, obtained from millimeter-wave radar point cloud information acquired by the millimeter-wave radar; pairing each target-level visual obstacle with each target-level millimeter-wave radar obstacle to obtain multiple corresponding target-obstacle matching pairs; inputting each target-obstacle matching pair into a target matching model, so that the target matching model obtains a corresponding inference result based on the target-obstacle matching pair, the inference result indicating whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target-obstacle matching pair meet the fusion conditions; and, if, based on the inference result, it is determined that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target-obstacle matching pair meet the fusion conditions, performing fusion processing on the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target-obstacle matching pair to obtain obstacle fusion information.

[0006] By employing the aforementioned obstacle fusion method, target obstacle matching pairs, formed by pairing target-level visual obstacles with target-level millimeter-wave radar obstacles, are input into the target matching model to determine whether the target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions. This improves the matching degree between the fused target-level visual obstacles and target-level millimeter-wave radar obstacles, thereby enhancing the accuracy of obstacle fusion information and ultimately improving the accuracy of obstacle detection, thus achieving more accurate obstacle detection.

[0007] In addition, pairing each target-level visual obstacle with each target-level millimeter-wave radar obstacle ensures that each target-level visual obstacle and each target-level millimeter-wave radar obstacle can be matched once.

[0008] In one embodiment of the obstacle fusion method provided in this application, whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions can, for example, refer to whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are the same obstacle. Furthermore, if the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are the same obstacle, then the fusion conditions are met; if the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are not the same obstacle, then the fusion conditions are not met.

[0009] This improves the matching degree between target-level visual obstacles and target-level millimeter-wave radar obstacles, thereby improving the accuracy of obstacle fusion information and thus improving the accuracy of obstacle detection, which means achieving obstacle detection more accurately.

[0010] Of course, whether the target obstacle matching pair, including target-level visual obstacles and target-level millimeter-wave radar obstacles, meets the fusion conditions can refer to whether other conditions are met, which can be set as needed.

[0011] In one embodiment of the obstacle fusion method provided in this application, the method further includes obtaining target-level visual obstacles based on visual information acquired by a visual camera in the following manner: obtaining corresponding visual obstacles based on visual information acquired by a visual camera, removing visual obstacles that are not within the target detection range from the visual obstacles, and obtaining target-level visual obstacles.

[0012] By employing the aforementioned obstacle fusion method, visual obstacles outside the target detection range are removed, preventing the presence of visual obstacles outside the target detection range in the target-level visual obstacles. This ensures that the generated target-level visual obstacles are more accurate, thereby guaranteeing a better match between the target-level visual obstacles and the target-level millimeter-wave radar obstacles.

[0013] In one embodiment of the obstacle fusion method provided in this application, the method further includes obtaining target-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by millimeter-wave radar in the following manner: obtaining point cloud-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by millimeter-wave radar; performing point cloud clustering processing on point cloud-level millimeter-wave radar obstacles to obtain multiple point cloud clustering results, and representing each point cloud clustering result with an anchor point; performing tracking and matching processing on the anchor points to obtain multiple tracked target points, where the target points are target-level millimeter-wave radar obstacles.

[0014] Using the obstacle fusion method described above, point cloud-level millimeter-wave radar obstacles are obtained based on the millimeter-wave radar point cloud information. Point cloud-level millimeter-wave radar obstacles are then subjected to point cloud clustering processing to accurately obtain the point cloud clustering results. Each point cloud clustering result is represented by an anchor point, and the anchor point is tracked and matched to accurately identify multiple target points. The target points are the target-level millimeter-wave radar obstacles, thus ensuring that the obtained target-level millimeter-wave radar obstacles are more accurate.

[0015] In one embodiment of the obstacle fusion method provided in this application, each target-level visual obstacle has its own corresponding target-level visual obstacle information, which includes target-level visual obstacle motion information; each target-level millimeter-wave radar obstacle has its own corresponding target-level millimeter-wave radar obstacle information, which includes target-level millimeter-wave radar obstacle motion information.

[0016] Furthermore, target-level visual obstacle motion information can include multiple visual obstacle motion information, that is, multiple frames of historical motion information; target-level millimeter-wave radar obstacle motion information can also include multiple millimeter-wave radar obstacle motion information, that is, multiple frames of historical motion information.

[0017] Therefore, by employing the aforementioned obstacle fusion method, pairing and fusing multi-frame historical motion information can be achieved. This effectively avoids pairing and fusion errors caused by pairing and fusion of single-frame motion information, resulting in inaccurate fused information. Thus, the accuracy of the final fused obstacle fusion information can be effectively guaranteed.

[0018] Of course, target-level visual obstacle information can also include obstacle identification information and speed and distance measurement information for the corresponding target-level visual obstacle, and target-level millimeter-wave radar obstacle information can also include speed and distance measurement information for the corresponding target-level millimeter-wave radar obstacle.

[0019] Furthermore, the aforementioned multi-frame historical motion information can be, for example, N frames of historical motion information corresponding to N frames of images, where N is a value such as 20, which can be set as needed.

[0020] In one embodiment of the obstacle fusion method provided in this application, the target obstacle matching pair includes the motion trajectory information of the corresponding target-level visual obstacle and the motion trajectory information of the target-level millimeter-wave radar obstacle.

[0021] Using the aforementioned obstacle fusion method, the target obstacle matching pair possesses corresponding obstacle motion trajectory information. Based on this information, it can further determine whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle match, ensuring more accurate obstacle fusion information. Furthermore, when fusing the target-level visual obstacle and the target-level millimeter-wave radar obstacle, the corresponding obstacle motion trajectory information is also fused to further guarantee the accuracy of the obtained obstacle fusion information.

[0022] In one embodiment of the obstacle fusion method provided in this application, point cloud clustering processing is performed based on point cloud-level millimeter-wave radar obstacles, including: performing point cloud clustering processing based on point cloud-level millimeter-wave radar obstacles using a density-based clustering algorithm.

[0023] In one embodiment of the obstacle fusion method provided in this application, the tracking and matching process for anchor points includes: matching the anchor points using a Hungarian matching algorithm and an identifier information matching algorithm. The identifier information may be, for example, an ID.

[0024] By employing the aforementioned obstacle fusion method and using a density-based clustering algorithm for point cloud clustering, the processing efficiency of point cloud-level millimeter-wave radar obstacles can be effectively improved. Furthermore, the point cloud clustering results obtained through this process are more accurate. Using the Hungarian matching algorithm and an identifier matching algorithm to match anchor points effectively improves the matching efficiency and reduces the number of mismatched anchor points.

[0025] In one embodiment of the obstacle fusion method provided in this application, the target matching model is a fully connected matching model.

[0026] By employing the obstacle fusion method described above, the fully connected matching model can be used as the target matching model for global feature extraction. Furthermore, the fully connected matching model has high nonlinear expression capabilities, which can effectively improve the accuracy of the reasoning results (also known as matching results) obtained based on the fully connected matching model.

[0027] In one embodiment of the obstacle fusion method provided in this application, the visual camera is an in-vehicle camera and the millimeter-wave radar is an in-vehicle millimeter-wave radar.

[0028] The embodiments of this application also disclose a model training method, which includes: determining a training dataset, the training dataset including multiple positive training samples and multiple negative training samples, the positive training samples being obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that satisfy the fusion conditions, and the negative training samples being obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that do not satisfy the fusion conditions; training a model based on the training dataset to obtain a target matching model, the target matching model being used to detect whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair satisfy the fusion conditions.

[0029] Using the above model training method, a target matching model is trained based on a training dataset including positive and negative training samples. This effectively improves the accuracy of detecting whether target-obstacle matching pairs, including target-level visual obstacles and target-level millimeter-wave radar obstacles, meet the fusion conditions based on the trained target matching model.

[0030] The embodiments of this application also disclose an electronic device, which includes: a processor, a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to enable the electronic device to implement the obstacle fusion method provided in any of the above embodiments, or to implement the model training method provided in any of the above embodiments.

[0031] By using the aforementioned electronic equipment, it is possible to determine whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle meet the fusion conditions through the target matching model. This can improve the matching degree between the target-level visual obstacle and the target-level millimeter-wave radar obstacle being fused, thereby improving the accuracy of obstacle fusion information and thus improving the accuracy of obstacle detection.

[0032] Furthermore, the aforementioned electronic device can train a target matching model based on a training dataset including positive and negative training samples, thereby effectively improving the accuracy of detecting whether target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions based on the trained target matching model. Attached Figure Description

[0033] Figure 1 A flowchart illustrating an obstacle fusion method provided in an embodiment of this application;

[0034] Figure 2A A schematic diagram illustrating that the target-level visual obstacle and the target-level millimeter-wave radar obstacle themselves possess information in the obstacle fusion method provided in the embodiments of this application;

[0035] Figure 2B A schematic diagram illustrating the motion information of the target-level visual obstacle itself and the motion information of the target-level millimeter-wave radar obstacle itself in the obstacle fusion method provided in the embodiments of this application;

[0036] Figure 3 A schematic diagram of the obstacle trajectory corresponding to a target-level visual obstacle and the obstacle trajectory corresponding to a target-level millimeter-wave radar obstacle in the obstacle fusion method provided in the embodiments of this application;

[0037] Figure 4 A schematic diagram of a process for obtaining target-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information obtained by millimeter-wave radar in the obstacle fusion method provided in the embodiments of this application;

[0038] Figure 5A schematic diagram illustrating a process for determining whether a target-level visual obstacle and a target-level millimeter-wave radar obstacle, included in a target obstacle fusion method provided in this application, can be fused.

[0039] Figure 6 This is another flowchart illustrating the process of determining whether a target-level visual obstacle and a target-level millimeter-wave radar obstacle, which are included in the obstacle fusion method provided in the embodiments of this application, can be fused.

[0040] Figure 7 A schematic flowchart of a model training method provided for an embodiment of this application;

[0041] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] As mentioned earlier, some vehicles currently employ a solution that fuses obstacle data acquired by visual cameras and millimeter-wave radar to achieve obstacle detection. However, when using existing technologies to fuse obstacle data acquired by visual cameras and millimeter-wave radar to achieve obstacle detection and enable vehicle driver assistance functions, problems such as unresolved edge cases can easily arise, affecting the accuracy of obstacle detection.

[0043] For example, in existing technologies, post-fusion methods are typically used to fuse obstacle detection data acquired by visual cameras and millimeter-wave radar to achieve more accurate obstacle detection. Post-fusion methods involve fusing obstacle detection results from multiple sensors. However, when using existing post-fusion methods to fuse obstacle data from multiple sensors, unresolved edge cases can easily arise. For instance, in the edge regions of images acquired by both visual cameras and millimeter-wave radar, target detection discrepancies can occur between the visual information obtained by the visual camera and the millimeter-wave radar's point cloud information. For example, visual cameras may have depth estimation errors due to monocular ranging, while millimeter-wave radar has lower resolution. In a bird's-eye view (BEV) coordinate system, if the error covariance characteristics of the visual information from the visual camera and the millimeter-wave radar's point cloud information are not considered, edge targets with inconsistent depths may appear. For example, when millimeter-wave radar detects a distant target, the visual camera may misidentify it as a nearby obstacle due to depth errors.

[0044] Therefore, how to fuse obstacle data acquired by visual cameras and millimeter-wave radar to achieve more accurate obstacle detection is a problem that is currently being explored in the field.

[0045] To address the aforementioned problems, this application discloses an obstacle fusion method, such as... Figure 1 As shown, the obstacle fusion method includes the following steps.

[0046] Step S11: Identify multiple target-level visual obstacles, which are obtained based on the visual information acquired by the visual camera.

[0047] Step S12: Identify multiple target-level millimeter-wave radar obstacles. The target-level millimeter-wave radar obstacles are obtained based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar.

[0048] Step S13: Pair each target-level visual obstacle with each target-level millimeter-wave radar obstacle to obtain multiple corresponding target-obstacle matching pairs.

[0049] Step S14: Input each target obstacle matching pair into the target matching model so that the target matching model can obtain the corresponding inference result based on the target obstacle matching pair. The inference result is used to indicate whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions.

[0050] Step S15: Based on the reasoning results, if the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions, perform fusion processing on the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair to obtain obstacle fusion information.

[0051] By employing the aforementioned obstacle fusion method, target obstacle matching pairs, formed by pairing target-level visual obstacles with target-level millimeter-wave radar obstacles, are input into the target matching model to determine whether the target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions. This improves the matching degree between the fused target-level visual obstacles and target-level millimeter-wave radar obstacles, thereby enhancing the accuracy of obstacle fusion information and ultimately improving the accuracy of obstacle detection, thus achieving more accurate obstacle detection.

[0052] In one embodiment of the obstacle fusion method provided in this application, step S11 may include: acquiring image information such as video recorded by a visual camera or multiple pictures captured by a camera; determining visual information based on the image information; and determining multiple target-level visual obstacles based on the visual information.

[0053] Furthermore, in one embodiment of the obstacle fusion method provided in this application, the visual information may include: lateral and longitudinal distance measurement of the visual camera relative to the obstacle, lateral and longitudinal velocity measurement of the visual camera relative to the obstacle, obstacle identification document (ID), and obstacle category, etc.

[0054] In one embodiment of the obstacle fusion method provided in this application, step S12 may include: acquiring radar data obtained from millimeter-wave radar, determining millimeter-wave radar point cloud information based on the radar data, and determining multiple target-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information.

[0055] Furthermore, in one embodiment of the obstacle fusion method provided in this application, the millimeter-wave radar point cloud information may include: millimeter-wave radar lateral and longitudinal ranging relative to the obstacle, millimeter-wave radar lateral and longitudinal velocity relative to the obstacle, and other information.

[0056] Furthermore, in one embodiment of the obstacle fusion method provided in this application, each target-level visual obstacle corresponds to one obstacle, each target-level millimeter-wave radar obstacle corresponds to one obstacle, each frame of visual information may contain visual information of multiple obstacles, and each frame of millimeter-wave radar point cloud information may contain millimeter-wave radar point cloud information of multiple obstacles. The multiple obstacles involved in the visual information and the multiple obstacles involved in the millimeter-wave radar point cloud information can be the same obstacles.

[0057] In one embodiment of the obstacle fusion method provided in this application, step S13 may include: pairing each of the multiple target-level visual obstacles with each of the multiple target-level millimeter-wave radar obstacles to obtain corresponding target-obstacle matching pairs. Specifically, based on the above steps, multiple target-level visual obstacles can be paired with multiple target-level millimeter-wave radar obstacles to obtain multiple target-obstacle matching pairs.

[0058] In one embodiment of the obstacle fusion method provided in this application, each target-level visual obstacle has its own corresponding target-level visual obstacle information, which includes target-level visual obstacle motion information; each target-level millimeter-wave radar obstacle has its own corresponding target-level millimeter-wave radar obstacle information, which includes target-level millimeter-wave radar obstacle motion information.

[0059] For example, such as Figure 2AAs shown, multiple target-level visual obstacles include, for example, target-level visual obstacle 1 (referred to as Vision 1), target-level visual obstacle 2 (referred to as Vision 2), target-level visual obstacle 3 (referred to as Vision 3), etc. Each target-level visual obstacle has its own corresponding target-level visual obstacle information, which includes target-level visual obstacle motion information (referred to as historical information or historical motion information), and also includes obstacle identification information such as the aforementioned ID, as well as speed and distance measurement information. In addition, multiple target-level millimeter-wave radar obstacles, such as target-level millimeter-wave radar obstacle 1 (referred to as radar 1), target-level millimeter-wave radar obstacle 2 (referred to as radar 2), target-level millimeter-wave radar obstacle 3 (referred to as radar 3), target-level millimeter-wave radar obstacle 4 (referred to as radar 4), etc., each target-level millimeter-wave radar obstacle has its own corresponding target-level millimeter-wave radar obstacle information, including target-level millimeter-wave radar obstacle motion information (referred to as historical information, or historical motion information), and target-level millimeter-wave radar obstacle information also includes, for example, the aforementioned speed and distance measurement information.

[0060] Furthermore, the target-level visual obstacle motion information can include multiple visual obstacle motion information sets, which are also multiple historical motion information sets. For example, the multiple historical motion information sets included in the target-level visual obstacle motion information set can be, for instance, the N historical motion information sets whose generation time is closest to the target fusion time. When N is 20, such as... Figure 2B As shown in section (a), the target-level visual obstacle motion information includes: the visual obstacle motion information obtained at time k (i.e., the information at time k) and the historical visual obstacle motion information obtained at times k-1…k-18, k-19 (i.e., historical time data: k-1…k-18, k-19). Here, time k can be understood as the current time.

[0061] Furthermore, the target-level millimeter-wave radar obstacle motion information can include multiple millimeter-wave radar obstacle motion information sets, which are also multiple historical motion information sets. For example, the multiple historical motion information sets included in the target-level millimeter-wave radar obstacle motion information can be, for instance, the N historical motion information sets whose generation time is closest to the target fusion time. When N is 20, such as... Figure 2B As shown in section (b), the target-level millimeter-wave radar obstacle motion information includes: millimeter-wave radar obstacle motion information obtained at time k (i.e., time k information) and historical millimeter-wave radar obstacle motion information obtained at times k-1…k-18, k-19 (i.e., historical time data: k-1…k-18, k-19). Here, time k can be understood as the current time.

[0062] Based on this, the resulting target obstacle matching pair is a matching pair that includes information about the corresponding target-level visual obstacle and the target-level millimeter-wave radar obstacle itself.

[0063] Furthermore, motion information can include the obstacle's position, speed, and corresponding timestamp when it is collected.

[0064] In one embodiment of the obstacle fusion method provided in this application, step S14 may include: inputting the obtained multiple target obstacle matching pairs into a target matching model, performing inference analysis processing on target-level visual obstacles and target-level millimeter-wave radar obstacles, and obtaining corresponding inference results. Further, the inference results are used to indicate whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions. Therefore, the inference results may, for example, indicate whether the target obstacle matching pair is fusionable or not fusionable. Of course, the inference results may also be other information used to achieve the aforementioned indication.

[0065] Whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions, for example, whether the historical motion information included in the target obstacle matching pair corresponds to the motion information of the same obstacle. If yes, the fusion conditions are met; if no, the fusion conditions are not met.

[0066] In other words, if the obstacle corresponding to the target-level visual obstacle and the obstacle corresponding to the target-level millimeter-wave radar obstacle are the same obstacle, the inference result is that the target-obstacle matching pair can be fused. Conversely, if the obstacle corresponding to the target-level visual obstacle and the obstacle corresponding to the target-level millimeter-wave radar obstacle are not the same obstacle, the inference result is that the target-obstacle matching pair cannot be fused.

[0067] This improves the matching degree between target-level visual obstacles and target-level millimeter-wave radar obstacles, thereby improving the accuracy of obstacle fusion information and thus improving the accuracy of obstacle detection.

[0068] Of course, whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions can refer to whether other conditions are met, such as whether the timestamp information of the target-level visual obstacles included in the target obstacle matching pair is consistent with the timestamp information of the target-level millimeter-wave radar obstacles, etc., which can be set as needed.

[0069] Furthermore, in one embodiment of the obstacle fusion method provided in this application, step S14 may further include: the target matching model performs reasoning analysis on the aforementioned target-level visual obstacle information corresponding to the target-level visual obstacle and the aforementioned target-level millimeter-wave radar obstacle information corresponding to the target-level millimeter-wave radar obstacle in the target obstacle matching pair to obtain the corresponding reasoning result.

[0070] In one embodiment of the obstacle fusion method provided in this application, step S15 may include: when the reasoning result is that the target obstacle matching pair can be fused, that is, when it is determined that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair meet the fusion conditions, the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are fused to obtain obstacle fusion information.

[0071] Furthermore, in one embodiment of the obstacle fusion method provided in this application, the method further includes: if the inference result indicates that the target obstacle matching pair cannot be fused, i.e., if it is determined that the target obstacle matching pair includes target-level visual obstacles and target-level millimeter-wave radar obstacles that are not the same obstacle, then, for example, the unpaired target-level visual obstacles and target-level millimeter-wave radar obstacles can be paired, or newly obtained target-level visual obstacles and target-level millimeter-wave radar obstacles can be paired to obtain new target obstacle matching pairs. The new target obstacle matching pairs are then input into the target matching model to obtain the corresponding inference result. That is, if the inference result corresponding to the current frame indicates that the target obstacle matching pair cannot be fused, the relevant inference processing for the next frame is performed.

[0072] Furthermore, the obstacle fusion method provided in this application can be used for detecting obstacles near a vehicle. Furthermore, the aforementioned visual camera can be an in-vehicle camera, and the aforementioned millimeter-wave radar can be an in-vehicle millimeter-wave radar.

[0073] In one embodiment of the obstacle fusion method provided in this application, the method further includes obtaining target-level visual obstacles based on visual information acquired by a visual camera in the following manner: obtaining corresponding visual obstacles based on visual information acquired by a visual camera, removing visual obstacles that are not within the target detection range from the visual obstacles, and obtaining target-level visual obstacles.

[0074] By employing the aforementioned obstacle fusion method, visual obstacles outside the target detection range are removed, preventing the presence of visual obstacles outside the target detection range in the target-level visual obstacles. This ensures that the generated target-level visual obstacles are more accurate, thereby guaranteeing a better match between the target-level visual obstacles and the target-level millimeter-wave radar obstacles.

[0075] In one embodiment of the obstacle fusion method provided in this application, a target detection range is preset in the vehicle. In the process of determining target-level visual obstacles based on visual information, visual obstacles that are not within the target detection range in the visual information are determined according to the preset target detection range. Visual obstacles that are not within the target detection range in the visual information are filtered out to obtain target-level visual obstacles.

[0076] Furthermore, the target detection range can be limited based on information such as the vehicle's condition and user needs, but no limit is set here.

[0077] Furthermore, in one embodiment of the obstacle fusion method provided in this application, steps S11 and S12 can be performed simultaneously, or step S11 can be performed first, followed by step S12, or step S12 can be performed first, followed by step S11. Specifically, the execution order of steps S11 and S12 can be determined based on the specific application scenario, and is not limited here.

[0078] In one embodiment of the obstacle fusion method provided in this application, the method further includes obtaining target-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by millimeter-wave radar in the following manner: obtaining point cloud-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by millimeter-wave radar; performing point cloud clustering processing on point cloud-level millimeter-wave radar obstacles to obtain multiple point cloud clustering results, and representing each point cloud clustering result with an anchor point; performing tracking and matching processing on the anchor points to obtain multiple tracked target points, where the target points are target-level millimeter-wave radar obstacles.

[0079] Using the obstacle fusion method described above, point cloud-level millimeter-wave radar obstacles are obtained based on the millimeter-wave radar point cloud information. Point cloud-level millimeter-wave radar obstacles are then subjected to point cloud clustering processing to accurately obtain the point cloud clustering results. Each point cloud clustering result is represented by an anchor point, and the anchor point is tracked and matched to accurately identify multiple target points. The target points are the target-level millimeter-wave radar obstacles, thus ensuring that the obtained target-level millimeter-wave radar obstacles are more accurate.

[0080] In one embodiment of the obstacle fusion method provided in this application, when point cloud-level millimeter-wave radar obstacles are obtained, each cluster of point clouds in the point cloud-level millimeter-wave radar obstacles can be clustered according to a clustering algorithm to obtain the clustering result corresponding to each cluster of point clouds. An anchor point is selected from the clustering results to represent the clustering result of each cluster of point clouds. The anchor point is then tracked and matched according to a tracking and matching algorithm to obtain multiple tracked target points, which are the target-level millimeter-wave radar obstacles.

[0081] Furthermore, in one embodiment of the obstacle fusion method provided in this application, when multiple target points are obtained after tracking, target points that are not within the preset target detection range can be removed to ensure the accuracy of obtaining target-level millimeter-wave radar obstacles.

[0082] In one embodiment of the obstacle fusion method provided in this application, the target-level visual obstacle motion information corresponding to each target-level visual obstacle may include multiple visual obstacle motion information. These multiple visual obstacle motion information can be stored in a target-level visual obstacle historical information array corresponding to each target-level visual obstacle, and the multiple visual obstacle motion information constitutes multiple historical motion information. Similarly, the target-level millimeter-wave radar obstacle motion information corresponding to each target-level millimeter-wave radar obstacle may include multiple millimeter-wave radar obstacle motion information. These multiple millimeter-wave radar obstacle motion information are stored in a target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle, and the multiple millimeter-wave radar obstacle motion information constitutes multiple historical motion information. The number of multiple historical motion information included in the target-level visual obstacle motion information corresponding to each target-level visual obstacle may be the same as or different from the number of multiple historical motion information included in the target-level millimeter-wave radar obstacle operation information corresponding to each target-level millimeter-wave radar obstacle.

[0083] The obstacle fusion method described above involves matching and fusing obstacle motion information from multiple frames. This effectively avoids matching and fusion errors caused by matching and fusing single-frame obstacle motion information, resulting in inaccurate fused information. Therefore, it effectively ensures the accuracy of the final fused obstacle information.

[0084] Furthermore, the time information of the visual obstacle motion information and the millimeter-wave radar obstacle motion information stored at the same time is consistent, which can ensure that the timestamps of the target-level visual obstacle and the target-level millimeter-wave radar obstacle are aligned, so as to ensure that the target-level visual obstacle and the target-level millimeter-wave radar obstacle can be matched, thereby further ensuring the accuracy of the fused obstacle information.

[0085] In one embodiment of the obstacle fusion method provided in this application, a visual camera acquires the visual obstacle motion information of visual obstacles in real time, and stores the acquired visual obstacle motion information in the target-level visual obstacle historical information array of the corresponding target-level visual obstacle for easy retrieval.

[0086] Furthermore, in one embodiment of the obstacle fusion method provided in this application, the visual obstacle motion information stored in the target-level visual obstacle historical information array corresponding to each target-level visual obstacle can be obtained as follows: When the first visual obstacle motion information of a certain target-level visual obstacle is initially acquired, the target-level visual obstacle historical information array corresponding to that target-level visual obstacle is empty. At this time, the visual obstacle motion information corresponding to that target-level visual obstacle acquired in real time is continuously stored in the corresponding target-level visual obstacle historical information array until the number of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is N. Furthermore, during the storage process, the corresponding visual obstacle motion information is obtained based on the visual information acquired in real time by the visual camera, and it is determined whether the number of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is greater than or equal to N. If the number of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is less than N, the acquired visual obstacle motion information is stored in the corresponding target-level visual obstacle historical information array. If the number of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is equal to or greater than N, delete the earliest stored visual obstacle motion information in the corresponding target-level visual obstacle historical information array until the number of visual obstacle motion information stored in the corresponding target-level visual obstacle historical information array is N-1, so that the newly obtained visual obstacle motion information can be stored in the corresponding target-level visual obstacle historical information array.

[0087] That is, at the initial moment, the target-level visual obstacle historical information array (i.e., the target-level visual obstacle historical information array) corresponding to each target-level visual obstacle is empty. When the first frame of visual obstacle motion information corresponding to a certain target-level visual obstacle is obtained, the first frame of visual obstacle motion information (i.e., the first frame of data) is stored at the beginning of the target-level visual obstacle historical information array corresponding to that target-level visual obstacle, and the position of the first frame of visual obstacle motion information in the target-level visual obstacle historical information array is recorded. This is so that when the number of visual obstacle motion information stored in the target-level visual obstacle historical information array reaches N, when the next frame of visual obstacle motion information is stored, if the array is full, the old data (i.e., the visual obstacle motion information stored at the earliest time) is removed, so as to ensure that the multiple visual obstacle motion information stored in the target-level visual obstacle historical information array are the N visual obstacle motion information whose generation time is closest to the target fusion time.

[0088] Furthermore, in one embodiment of the obstacle fusion method provided in this application, the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle can be obtained in the following way: When the first millimeter-wave radar obstacle motion information of a certain target-level millimeter-wave radar obstacle is initially acquired, the target-level millimeter-wave radar obstacle historical information array corresponding to that target-level millimeter-wave radar obstacle is empty. At this time, the millimeter-wave radar obstacle motion information corresponding to that target-level millimeter-wave radar obstacle acquired in real time is continuously stored in the corresponding target-level millimeter-wave radar obstacle historical information array until the number of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle historical information array is equal to N. Furthermore, during the storage process, the corresponding millimeter-wave radar obstacle motion information is obtained based on the millimeter-wave radar point cloud information acquired in real time by the millimeter-wave radar, and it is determined whether the number of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle historical information array is greater than or equal to N. If the number of millimeter-wave radar obstacle motion information entries stored in the target-level millimeter-wave radar obstacle history information array is less than N, the acquired millimeter-wave radar obstacle motion information is stored in the corresponding target-level millimeter-wave radar obstacle history information array. If the number of millimeter-wave radar obstacle motion information entries stored in the target-level millimeter-wave radar obstacle history information array is equal to or greater than N, the earliest stored millimeter-wave radar obstacle motion information in the corresponding target-level millimeter-wave radar obstacle history information array is deleted until the number of stored millimeter-wave radar obstacle motion information entries in the corresponding target-level millimeter-wave radar obstacle history information array is N-1, so that newly obtained millimeter-wave radar obstacle motion information can be stored in the corresponding target-level millimeter-wave radar obstacle history information array.

[0089] That is, at the initial moment, the target-level millimeter-wave radar obstacle historical information array (i.e., the target-level millimeter-wave radar obstacle historical information array) corresponding to each target-level millimeter-wave radar obstacle is empty. When the first frame of millimeter-wave radar obstacle motion information of a certain target-level millimeter-wave radar obstacle is obtained, the first frame of millimeter-wave radar obstacle motion information (i.e., the first frame of data) is stored at the beginning of the corresponding target-level millimeter-wave radar obstacle historical information array, and the position of the first frame of millimeter-wave radar obstacle motion information in the corresponding target-level millimeter-wave radar obstacle historical information array is recorded. This is so that when the number of millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle historical information array reaches N, when the next frame of millimeter-wave radar obstacle motion information is stored, if the array is full, the old data (i.e., the millimeter-wave radar obstacle motion information stored at the earliest time) is removed, so as to ensure that the multiple millimeter-wave radar obstacle motion information stored in the corresponding target-level millimeter-wave radar obstacle historical information array are the N millimeter-wave radar obstacle motion information whose generation time is closest to the target fusion time.

[0090] Furthermore, in one embodiment of the obstacle fusion method provided in this application, when N is 20, as described above, such as Figure 2B As shown in section (a), the visual obstacle motion information stored in the target-level visual obstacle historical information array includes, for example, motion information obtained at time k and historical motion information obtained at times k-1…k-18, k-19. Here, time k can be understood as the current time.

[0091] Furthermore, in one embodiment of the obstacle fusion method provided in this application, when N is 20, as described above, such as Figure 2B As shown in section (b), the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array includes, for example, motion information obtained at time k and historical motion information obtained at times k-1…k-18, k-19. Here, time k can be understood as the current time.

[0092] In one embodiment of the obstacle fusion method provided in this application, the target obstacle matching pair includes the motion trajectory information of the corresponding target-level visual obstacle and the motion trajectory information of the target-level millimeter-wave radar obstacle.

[0093] The obstacle fusion method described above includes motion trajectory information of the corresponding target-level visual obstacle and the target-level millimeter-wave radar obstacle in the target obstacle matching pair. This allows for further determination of whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle match based on their respective motion trajectory information, ensuring more accurate obstacle fusion information. Furthermore, when fusing the target-level visual obstacle and the target-level millimeter-wave radar obstacle, the corresponding motion trajectory information is also fused to further guarantee the accuracy of the obtained obstacle fusion information.

[0094] In one embodiment of the obstacle fusion method provided in this application, when matching target-level visual obstacles and target-level millimeter-wave radar obstacles to obtain target-obstacle matching pairs, the target obstacle matching includes: motion trajectory information corresponding to the target-level visual obstacles and motion trajectory information corresponding to the target-level millimeter-wave radar obstacles. When the target obstacle matching pairs are input into the target matching model, the target matching model infers the motion trajectory information corresponding to the target-level visual obstacles and the motion trajectory information corresponding to the target-level millimeter-wave radar obstacles to output corresponding inference results.

[0095] In one embodiment of the obstacle fusion method provided in this application, a real vehicle can record, for example, one hour of data. During the recording process, motion information of target-level visual obstacles and motion information of target-level millimeter-wave radar obstacles can be recorded simultaneously, and the corresponding information can be visualized using visualization tools. Figure 3 The image shows information displayed at a certain moment, where trajectory 1 is the motion trajectory information (i.e., visual trajectory point) corresponding to the target-level visual obstacle, and trajectory 2 is the motion trajectory information (i.e., radar trajectory point) corresponding to the target-level millimeter-wave radar obstacle. Furthermore, given trajectory 1 and trajectory 2, they can be fused to obtain the motion trajectory information corresponding to the target obstacle matching pair, which is then used as the target obstacle matching pair.

[0096] In one embodiment of the obstacle fusion method provided in this application, point cloud clustering processing is performed based on point cloud-level millimeter-wave radar obstacles, including: performing point cloud clustering processing based on point cloud-level millimeter-wave radar obstacles using a density-based clustering algorithm; and anchor point tracking and matching processing is performed, including: matching anchor points using a Hungarian matching algorithm and an identifier information matching algorithm. The identifier information is, for example, an ID.

[0097] By employing the aforementioned obstacle fusion method and using a density-based clustering algorithm for point cloud clustering, the processing efficiency of point cloud-level millimeter-wave radar obstacles can be effectively improved. Furthermore, the point cloud clustering results obtained through this process are more accurate. Using the Hungarian matching algorithm and an identifier matching algorithm to match anchor points effectively improves the matching efficiency and reduces the number of mismatched anchor points.

[0098] like Figure 4 As shown, in one embodiment of the obstacle fusion method provided in this application, obtaining target-level millimeter-wave radar obstacles based on millimeter-wave radar point cloud information acquired by millimeter-wave radar includes the following steps.

[0099] Step S121: Obtain point cloud-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar.

[0100] Further, step S121 can be to determine the point cloud-level millimeter-wave radar obstacle based on the vehicle-mounted radar, so as to obtain the point cloud-level millimeter-wave radar obstacle.

[0101] Step S122: The point cloud level millimeter-wave radar obstacle is clustered using the density-based clustering algorithm (DBSCAN) to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point.

[0102] Further, step S122 involves clustering using the density-based clustering algorithm (DBSCAN).

[0103] Step S123: The anchor point is tracked and matched using the Hungarian matching algorithm and the ID matching algorithm to obtain the target-level millimeter-wave radar obstacle.

[0104] Further, step S123 involves tracking using the Hungarian matching algorithm and the ID matching algorithm to identify obstacles for the target-level millimeter-wave radar.

[0105] In one embodiment of the obstacle fusion method provided in this application, the target matching model is a fully connected matching model.

[0106] By employing the obstacle fusion method described above, using the fully connected matching model as the target matching model enables global feature extraction, and the fully connected matching model has high nonlinear expression capability, thereby effectively improving the accuracy of the matching results obtained based on the fully connected matching model.

[0107] Furthermore, such as Figure 5 and Figure 6As shown, in one embodiment of the obstacle fusion method provided in this application, taking the number of both target-level visual obstacles and target-level millimeter-wave radar obstacles as N (i.e., the number of visual obstacles and millimeter-wave radar obstacles are both N, although the number of the two can be different), True indicates that the target obstacle matching pair can be fused, and False indicates that the target obstacle matching pair cannot be fused, this example illustrates whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair in the above obstacle fusion method can be fused. Specifically, it includes the following process and steps.

[0108] The visual camera acquires multiple visual information in real time or periodically. Based on the visual information acquired by the visual camera, N target-level visual obstacles are obtained. The target-level visual obstacle historical information array corresponding to each target-level visual obstacle stores, for example, 20 frames of visual obstacle motion information.

[0109] Further, N target-level visual obstacles are obtained, as well as 20 frames of visual obstacle motion information stored in the target-level visual obstacle historical information array corresponding to each target-level visual obstacle. Specifically, the steps are as follows.

[0110] Step S100: The visual camera acquires visual information.

[0111] Step S101: Based on the visual information acquired by the visual camera, obtain the corresponding visual obstacles, remove the visual obstacles that are not within the target detection range, and obtain N target-level visual obstacles.

[0112] Furthermore, the vision camera can be an in-vehicle camera. The target detection range can be the area the vehicle is to reach or the area within a preset distance around the vehicle, determined based on vehicle driving information, to facilitate the detection of obstacles in the area the vehicle is to reach or the area within the preset distance around the vehicle. Furthermore, obstacles can include: traffic signs, pedestrians, surrounding vehicles, and other obstacles that hinder the vehicle's movement or parking.

[0113] Step S102: Store or save the visual obstacle motion information of the target-level visual obstacle obtained at the current moment into the historical information storage array corresponding to the target-level visual obstacle, that is, the historical information array of the target-level visual obstacle.

[0114] Step S103: Determine whether the number of visual obstacle motion information stored in the target-level visual obstacle historical information array is greater than or equal to 20. If yes, proceed to step S104; otherwise, proceed to step S105.

[0115] That is, step S103 can be used to determine whether the historical information array is greater than 20.

[0116] Step S104: If yes, delete the earliest visual obstacle motion information stored in the target-level visual obstacle historical information array, and store the visual obstacle motion information obtained at the current moment into the target-level visual obstacle historical information array.

[0117] Further, in step S104, if it is determined that the number of visual obstacle motion information stored in the target-level visual obstacle historical information array is greater than or equal to 20, the visual obstacle motion information stored in the target-level visual obstacle historical information array needs to be updated. At this time, one frame of historical data in the target-level visual obstacle historical information array is removed in order to store new data.

[0118] Step S105: If not, store the motion information of the visual obstacle obtained at the current moment into the target-level visual obstacle historical information array in sequence.

[0119] Furthermore, the millimeter-wave radar acquires multiple millimeter-wave radar point cloud information in real time or periodically, obtains N target-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information, and obtains, for example, 20 frames of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle.

[0120] Furthermore, obtaining the motion information of 20 frames of millimeter-wave radar obstacles stored in the target-level millimeter-wave radar obstacle historical information array corresponding to each of the N target-level millimeter-wave radar obstacles specifically includes the following steps.

[0121] Step S106: Obtain point cloud-level millimeter-wave radar obstacles based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar.

[0122] Further, step S106 can be to determine the point cloud-level millimeter-wave radar obstacle based on the vehicle-mounted millimeter-wave radar, so as to obtain the point cloud-level millimeter-wave radar obstacle.

[0123] Step S107: The point cloud level millimeter-wave radar obstacle is clustered using the density-based clustering algorithm (DBSCAN) to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point.

[0124] Further, step S107 involves clustering using the density-based clustering algorithm (DBSCAN).

[0125] In step S108, the anchor points are tracked and matched using the Hungarian matching algorithm and the ID matching algorithm to obtain the target-level millimeter-wave radar obstacle.

[0126] Further, step S108 involves tracking based on the Hungarian matching algorithm and the ID matching algorithm to identify obstacles for the target-level millimeter-wave radar.

[0127] Step S109: Store the obtained millimeter-wave radar obstacle motion information corresponding to the target-level millimeter-wave radar obstacle into the historical information of the target-level millimeter-wave radar obstacle.

[0128] Step S110: Determine whether the number of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle historical information array corresponding to the target-level millimeter-wave radar obstacle is greater than or equal to 20. If yes, proceed to step S111; otherwise, proceed to step S112.

[0129] Step S111: If yes, delete the earliest millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array, and store the millimeter-wave radar obstacle motion information obtained at the current moment into the target-level millimeter-wave radar obstacle history information array.

[0130] Further, in step S111, if it is determined that the number of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array is greater than or equal to 20, the millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle history information array needs to be updated. At this time, one frame of historical data is removed from the target-level millimeter-wave radar obstacle history information array to apply the corresponding stored data.

[0131] Step S112: If not, store the current millimeter-wave radar obstacle motion information in the target-level millimeter-wave radar obstacle history information array in sequence.

[0132] Furthermore, the target-level millimeter-wave radar obstacle historical information array for each target-level millimeter-wave radar obstacle can be a historical information array of the motion information of that target-level millimeter-wave radar obstacle. Furthermore, in step S112, the currently obtained millimeter-wave radar obstacle motion information can be stored in the corresponding target-level millimeter-wave radar obstacle historical information array according to the order in which the millimeter-wave radar obstacle motion information is obtained.

[0133] Based on the steps described above, the process involves determining multiple target-level visual obstacles and the 20 frames of visual obstacle motion information stored in the target-level visual obstacle historical information array corresponding to each target-level visual obstacle, as well as determining multiple target-level millimeter-wave radar obstacles and the 20 frames of millimeter-wave radar obstacle motion information stored in the target-level millimeter-wave radar obstacle historical information array corresponding to each target-level millimeter-wave radar obstacle. The process of calling the above information to determine whether the corresponding target obstacle matching pairs can be fused specifically includes the following steps.

[0134] Step S113: Pair N target-level visual obstacles with N target-level millimeter-wave radar obstacles in pairs to obtain multiple corresponding target-obstacle matching pairs.

[0135] Since target-level visual obstacles and target-level millimeter-wave radar obstacles both have corresponding target-level visual obstacle information, pairing target-level visual obstacles and target-level millimeter-wave radar obstacles can be understood as pairing target-level visual obstacle information and target-level millimeter-wave radar obstacle information.

[0136] Step S114: Input each target obstacle matching pair into the target matching model so that the target matching model outputs a reasoning result that the target obstacle matching pair can be fused (i.e., True) or the target obstacle matching pair cannot be fused (i.e., False) based on the target obstacle matching pair.

[0137] Furthermore, in step S114, the target matching model can be a fully connected matching model.

[0138] In step S115, if the target matching model outputs that the target obstacle matching pair can be fused (i.e., True), then it can be determined that the target-level visual obstacles and the target-level millimeter-wave radar obstacles included in the target obstacle matching pair can be fused.

[0139] Further, after performing step S115, the target obstacle matching pair, including target-level visual obstacles and target-level millimeter-wave radar obstacles, is fused to obtain obstacle fusion information.

[0140] In step S116, if the target matching model outputs that the target obstacle matching pair cannot be fused (i.e., False), it can be determined that the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair cannot be fused.

[0141] Furthermore, when pairing target-level visual obstacles and target-level millimeter-wave radar obstacles, for example, when the target-level visual obstacles and target-level millimeter-wave radar obstacles correspond to the same obstacle at the same time, it can be considered that the target-level visual obstacles and target-level millimeter-wave radar obstacles can be fused; otherwise, it can be considered that the corresponding target-level visual obstacles and target-level radar point cloud obstacles cannot be fused.

[0142] By employing the aforementioned obstacle fusion method, target obstacle matching pairs, including target-level visual obstacles and target-level millimeter-wave radar obstacles, are input into the target matching model to determine whether the target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions. This method can improve the matching degree between the target-level visual obstacles and target-level millimeter-wave radar obstacles being fused, thereby improving the accuracy of obstacle fusion information and ultimately enhancing the accuracy of obstacle detection.

[0143] Furthermore, while ensuring that the target-level visual obstacles and the target-level millimeter-wave radar obstacles meet the fusion conditions, fusing the corresponding target-level visual obstacles and target-level millimeter-wave radar obstacles to generate corresponding obstacle fusion information can ensure that the generated obstacle fusion information is more accurate.

[0144] like Figure 7 As shown, an embodiment of this application also discloses a model training method, which includes the following steps.

[0145] Step S31: Determine the training dataset. The training dataset includes multiple positive training samples and multiple negative training samples. Positive training samples are obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that meet the fusion conditions. Negative training samples are obstacle matching pairs formed by target-level visual obstacles and target-level millimeter-wave radar obstacles that do not meet the fusion conditions.

[0146] Step S32: Train the model based on the training dataset to obtain the target matching model. The target matching model is used to detect whether the target-level visual obstacles and target-level millimeter-wave radar obstacles included in the target obstacle matching pair meet the fusion conditions.

[0147] Specifically, motion information of target-level visual obstacles with trajectories and motion information of target-level millimeter-wave radar obstacles with trajectories can be recorded step-by-step or simultaneously to generate a training dataset in the manner described above. This training dataset is then input into, for example, a fully connected matching model for training, generating a target matching model as the training result, which can then be used for real-vehicle inference.

[0148] In real-vehicle reasoning, as described in the obstacle fusion method section above, there will be N target-level visual obstacles and N target-level millimeter-wave radar obstacles during the target fusion time. These are paired up to form matching pairs, which are then input into the target matching model to obtain the corresponding reasoning results.

[0149] Using the above model training method, a target matching model is trained based on a training dataset including positive and negative training samples. This effectively improves the accuracy of detecting whether target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions based on the trained target matching model.

[0150] Furthermore, in one embodiment of the model training method provided in this application, the positive training samples in the training dataset can be obstacle matching pairs formed by manually selected obstacles, such as target-level visual obstacles and target-level millimeter-wave radar obstacles corresponding to the same obstacle, which meet the fusion conditions. The negative samples can be obstacle matching pairs formed by randomly generated obstacles, such as target-level visual obstacles and target-level millimeter-wave radar obstacles that do not meet the fusion conditions, which do not meet the fusion conditions.

[0151] Furthermore, in one embodiment of the model training method provided in this application, the ratio of the number of positive training samples to the number of negative training samples in the training dataset can be, for example, 1:3.

[0152] Furthermore, the above is merely an example of the ratio of positive training samples to negative training samples in the training dataset, and no limitation is imposed here. Users can determine it according to their actual needs.

[0153] Furthermore, the obtained camera-related data and millimeter-wave radar-related data can be saved to tools used for recording and replaying data, such as ROSBATO, for presentation and use. The corresponding camera-related data and millimeter-wave radar-related data in ROSBATO can be converted into an easy-to-read and write file format, such as a JSON file. The information in the JSON file is then input into a fully connected matching model for training. The weights of the trained model can be saved as a TXT file and deployed to a domain controller for inference. Multiple frames of millimeter-wave point cloud information and multiple frames of camera obstacle information are input into the model. The model outputs true or false to determine whether the input point cloud information and camera information constitute a single obstacle, thus determining whether the fusion conditions are met.

[0154] like Figure 8 As shown, embodiments of this application also disclose an electronic device, which includes: a transceiver 121, a processor 122, and a memory 123 communicatively connected to the processor 122; the memory 123 stores computer execution instructions; the processor 122 executes the computer execution instructions stored in the memory 123 to enable the electronic device to implement the obstacle fusion method provided in any of the above embodiments, or to implement the model training method provided in any of the above embodiments.

[0155] Using the aforementioned electronic equipment, a target matching model can determine whether target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions. This ensures that the target obstacle matching pair includes a more accurate match of target-level visual obstacles and target-level millimeter-wave radar obstacles, thereby guaranteeing more accurate obstacle fusion information. Based on the target-level visual obstacles and target-level millimeter-wave radar obstacles that meet the fusion conditions, corresponding obstacle fusion information is generated to ensure greater accuracy of the generated obstacle fusion information.

[0156] Furthermore, the aforementioned electronic device can train a target matching model based on a training dataset including positive and negative training samples, thereby effectively improving the accuracy of detecting whether target-level visual obstacles and target-level millimeter-wave radar obstacles meet the fusion conditions based on the trained target matching model.

[0157] Furthermore, the processor 122 executes the computer execution instructions stored in the memory 123, causing the processor 122 to execute the obstacle fusion method or model training method in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0158] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0159] Furthermore, in one embodiment of the electronic device of this application, the electronic device may be a vehicle, computer, mobile phone, or other electronic device.

[0160] In one embodiment of this application, the electronic device may further include a transceiver 121. The transceiver 121 can be used to acquire the task to be run and its configuration information.

[0161] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0162] Another embodiment of this application discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the obstacle fusion method as provided in any of the above embodiments, or implements the model training method as provided in any of the above embodiments.

[0163] Another embodiment of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the obstacle fusion method provided in any of the above embodiments, or implements the model training method provided in any of the above embodiments.

[0164] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this application are limited to these embodiments. On the contrary, the purpose of describing the application in conjunction with the embodiments is to cover other options or modifications that may be derived based on the claims of this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0165] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0166] In the description of this embodiment, it should be noted that the terms "upper", "lower", "inner", "bottom", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0167] The terms “first”, “second”, etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0168] In the description of this embodiment, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this embodiment based on the specific circumstances.

[0169] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific embodiments, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. An obstacle fusion method, characterized in that, The method includes: Multiple target-level visual obstacles are identified, which are obtained based on visual information acquired by a visual camera. Each target-level visual obstacle has its own corresponding target-level visual obstacle information, which includes motion information of the target-level visual obstacle. The motion information of the target-level visual obstacle includes multiple motion information of visual obstacles, and each motion information of visual obstacle is historical motion information of the obstacle obtained based on the visual information. Multiple target-level millimeter-wave radar obstacles are identified. These obstacles are obtained based on millimeter-wave radar point cloud information acquired by the millimeter-wave radar. Each obstacle possesses its own corresponding target-level millimeter-wave radar obstacle information, which includes obstacle motion information. This motion information comprises multiple obstacle motion information sets, and each obstacle's motion information is historical motion information obtained based on the millimeter-wave radar point cloud information. The target-level millimeter-wave radar obstacles are obtained from the millimeter-wave radar point cloud information using the following method: Based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar, point cloud-level millimeter-wave radar obstacles are obtained; Based on the point cloud-level millimeter-wave radar obstacles, point cloud clustering processing is performed to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point; The anchor points are tracked and matched to obtain multiple target points after tracking, and the target points are the obstacles of the target-level millimeter-wave radar. and, Based on the point cloud-level millimeter-wave radar obstacles, point cloud clustering processing is performed, including: Point cloud clustering is performed based on the point cloud level millimeter-wave radar obstacles using a density-based clustering algorithm. The tracking and matching process for the anchor points includes: The anchor points are matched using the Hungarian matching algorithm and the identifier information matching algorithm. Each of the target-level visual obstacles is paired with each of the target-level millimeter-wave radar obstacles to obtain multiple corresponding target-obstacle matching pairs. Each target-obstacle matching pair includes the motion trajectory information of the corresponding target-level visual obstacle and the motion trajectory information of the target-level millimeter-wave radar obstacle. Each of the target obstacle matching pairs is input into the target matching model so that the target matching model obtains a corresponding inference result based on the target obstacle matching pair. The inference result is whether the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair can be fused or not fused. If the inference result indicates that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair can be fused, then the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are fused to obtain obstacle fusion information. If the inference result indicates that the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair cannot be fused, then the target-level visual obstacle and the target-level millimeter-wave radar obstacle included in the target obstacle matching pair are not fused.

2. The method according to claim 1, characterized in that, The method further includes obtaining the target-level visual obstacle based on the visual information acquired by the visual camera in the following manner: Based on the visual information acquired by the visual camera, corresponding visual obstacles are obtained, and visual obstacles that are not within the target detection range are removed to obtain the target-level visual obstacles.

3. The method according to claim 2, characterized in that, The target matching model is a fully connected matching model.

4. The method according to any one of claims 1-3, characterized in that, The vision camera is a vehicle-mounted camera, and the millimeter-wave radar is a vehicle-mounted millimeter-wave radar.

5. A model training method, characterized in that, The method includes: A training dataset is determined, comprising multiple positive training samples and multiple negative training samples. The positive training samples are obstacle matching pairs formed by fusionable target-level visual obstacles and target-level millimeter-wave radar obstacles. The negative training samples are obstacle matching pairs formed by non-fusionable target-level visual obstacles and target-level millimeter-wave radar obstacles. The target-level visual obstacles are obtained based on visual information acquired by a visual camera. Each target-level visual obstacle possesses corresponding target-level visual obstacle information, including target-level visual obstacle motion information. This motion information comprises multiple visual obstacle motion information sets, each of which is historical motion information of the obstacle obtained based on the visual information. The target-level millimeter-wave radar... The obstacles are obtained based on millimeter-wave radar point cloud information acquired by millimeter-wave radar. Each target-level millimeter-wave radar obstacle has its own corresponding target-level millimeter-wave radar obstacle information, which includes target-level millimeter-wave radar obstacle motion information. The target-level millimeter-wave radar obstacle motion information includes multiple millimeter-wave radar obstacle motion information, and each millimeter-wave radar obstacle motion information is the historical motion information of the obstacle obtained based on the millimeter-wave radar point cloud information. The obstacle matching pair includes the motion trajectory information of the corresponding target-level visual obstacle and the motion trajectory information of the target-level millimeter-wave radar obstacle. The target-level millimeter-wave radar obstacles are obtained based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar in the following manner: Based on the millimeter-wave radar point cloud information acquired by the millimeter-wave radar, point cloud-level millimeter-wave radar obstacles are obtained; Based on the point cloud-level millimeter-wave radar obstacles, point cloud clustering processing is performed to obtain multiple point cloud clustering results, and each point cloud clustering result is represented by an anchor point; The anchor points are tracked and matched to obtain multiple target points after tracking, and the target points are the obstacles of the target-level millimeter-wave radar. and, Based on the point cloud-level millimeter-wave radar obstacles, point cloud clustering processing is performed, including: Point cloud clustering is performed based on the point cloud level millimeter-wave radar obstacles using a density-based clustering algorithm. The tracking and matching process for the anchor points includes: The anchor points are matched using the Hungarian matching algorithm and the identifier information matching algorithm. The model is trained based on the training dataset to obtain a target matching model. The target matching model is applied to the obstacle fusion method as described in any one of claims 1-4 to obtain inference results on whether the target-level visual obstacles and the target-level millimeter-wave radar obstacles included in the target obstacle matching pair can be fused or not.

6. An electronic device, characterized in that, The electronic device includes: a processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to enable the electronic device to implement the obstacle fusion method as described in any one of claims 1-4, or the model training method as described in claim 5.

Images