Obstacle association method and device in multi-sensor data and electronic equipment

Abstract

The present application relates to the field of multi-sensor data fusion, and particularly relates to a multi-sensor data obstacle correlation method, device and electronic equipment. The method comprises: obtaining sensor data at the current time and obtaining historical fusion tracking obstacles; predicting the predicted positions of each historical fusion tracking obstacle; determining a first obstacle with the same obstacle identification information in the historical fusion tracking obstacles and the observation obstacles, and correlating the first obstacle; calculating the Euclidean distance cost matrix between each observation obstacle and each historical fusion tracking obstacle in the top-down perspective according to the predicted positions of each historical fusion tracking obstacle, using the Hungarian matching algorithm for correlation, and updating the correlation result; and correlating the radar obstacles and the camera obstacles in the perspective projection perspective according to the predicted positions of each historical fusion tracking obstacle, and updating the correlation result. The accuracy of the correlation result is ensured, and the safety of driving is ensured.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor data fusion, and specifically to a method, apparatus, and electronic device for associating obstacles in multi-sensor data. Background Technology

[0002] In current autonomous driving, multi-sensor fusion technology is often used to compensate for the performance deficiencies of single sensors. This technology mainly includes a multi-sensor data association module and a state update module. The role of the data association module is to establish the relationship between the tracking results of different sensors, providing a data source for the subsequent state update module.

[0003] Currently, a common data association scheme involves calculating the Euclidean or Mahalanobis distance cost matrix from a 3D top-down view, and then using the Hungarian matching algorithm to associate targets. Another approach is to project 3D obstacles onto a pixel coordinate system, set the cost matrix on the 2D image, and then perform association matching.

[0004] However, the cost matrices calculated by the above methods are not comprehensive and there are cases of cost matrix calculation errors, which may result in obstacles not being associated or being mistakenly associated, thus posing a driving hazard. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method for associating obstacles in multi-sensor data, aiming to solve the problem in the prior art where inaccurate obstacle association in multi-sensor data leads to potential driving hazards.

[0006] According to a first aspect, embodiments of the present invention provide a method for associating obstacles in multi-sensor data, including:

[0007] Acquire sensor data at the current moment and acquire historical fusion-tracked obstacles; historical fusion-tracked obstacles are used to characterize obstacles after associating them with obstacles in the historical sensor data of the previous moment.

[0008] Predict the predicted positions of each historically fused tracking obstacle within a preset time period using a preset prediction method;

[0009] When the sensor data at the current moment is the radar sensor data at the current moment, the observed obstacles in the radar sensor data at the current moment are compared with the historical fusion tracked obstacles to determine the first obstacle in the historical fusion tracked obstacles that has the same obstacle identification information as the observed obstacles, and the first obstacle is associated.

[0010] When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle under the top-down view is calculated. Based on the Euclidean distance cost matrix, the Hungarian matching algorithm is used to perform association and update the association results.

[0011] When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated.

[0012] The obstacle association method in multi-sensor data provided in this invention acquires current-time sensor data and historically fused-tracked obstacles; it uses a preset prediction method to predict the predicted positions of each historically fused-tracked obstacle within a preset time period, ensuring the accuracy of the predicted positions. When the current-time sensor data is radar sensor data, the observed obstacles in the current-time radar sensor data are compared with the historically fused-tracked obstacles to identify the first obstacle among the historically fused-tracked obstacles that has the same obstacle identification information as the observed obstacles, and then associates the first obstacle, thereby achieving fast and accurate association of the first obstacle. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle is calculated from the top-down perspective. Based on the Euclidean distance cost matrix, the Hungarian matching algorithm is used for association, and the association results are updated. This achieves the association of unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle from the top-down perspective. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historically fused tracked obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historically fused tracked obstacles from the perspective projection view, and the association results are updated. This method achieves accurate association between radar obstacles and camera obstacles from a perspective projection viewpoint. It employs a multi-layered association mechanism, fully utilizing the multi-dimensional information of obstacles to improve association accuracy and reduce false and missed associations. Furthermore, the multi-layered association mechanism associates as many camera and radar obstacles as possible, providing more accurate position and velocity information for fusion state updates. This ensures the accuracy of the association results, thereby guaranteeing driving safety.

[0013] In conjunction with the first aspect, in the first embodiment of the first aspect, after acquiring the sensor data at the current moment and acquiring the historical fusion tracking obstacles, the method further includes:

[0014] Obtain the location information of each candidate obstacle from the sensor data at the current moment;

[0015] Based on the location information of each candidate obstacle, candidate obstacles outside the target range are deleted, and the observed obstacles included in the sensor data at the current moment are obtained.

[0016] The obstacle association method in multi-sensor data provided in this invention obtains the location information of each candidate obstacle in the sensor data at the current moment; based on the location information of each candidate obstacle, candidate obstacles outside the target range are deleted to obtain the observed obstacles included in the sensor data at the current moment, ensuring the accuracy of the obtained observed obstacles. The above method uses target range-based filtering of candidate obstacles, reducing the time consumption and false associations in obstacle association.

[0017] In conjunction with the first aspect, in the second embodiment of the first aspect, the method further includes:

[0018] When the sensor data at the current moment is not the radar sensor data at the current moment, and when the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle from the top-down perspective. Based on the Euclidean distance cost matrix, use the Hungarian matching algorithm to perform association and update the association results.

[0019] When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated;

[0020] When the sensor data at the current moment is the camera sensor data at the current moment, the identification information of the unassociated camera obstacles in the historical fusion tracking obstacles is compared with the identification information of the obstacles in the observed obstacles, and the second obstacle with the same identification information is associated.

[0021] The obstacle association method in multi-sensor data provided in this invention addresses the following situations: when the current sensor data is not the current radar sensor data, and when the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused tracked obstacles is also greater than zero, the method calculates the Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle from a top-down perspective, based on the predicted positions of the unassociated obstacles in each historically fused tracked obstacle within a preset time period. Based on the Euclidean distance cost matrix, the method uses a Hungarian matching algorithm for association, updates the association results, and ensures the accuracy of the calculated Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle from a top-down perspective. This, in turn, ensures the accuracy of using the Hungarian matching algorithm to associate the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle. Then, when the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unassociated obstacles in each historically fused tracked obstacle within a preset time period, the unassociated obstacles in the observed obstacles and the radar obstacles among the unassociated obstacles in the historically fused tracked obstacles are associated with the camera obstacles among the unassociated obstacles in the observed obstacles and the camera obstacles among the unassociated obstacles in the historically fused tracked obstacles under the perspective projection view. The association results are updated, ensuring the accuracy of associating radar obstacles and camera obstacles under the perspective projection view. When the sensor data at the current moment is the camera sensor data at the current moment, the identification information of the unassociated camera obstacles in the historically fused tracked obstacles is compared with the identification information of the obstacles in the observed obstacles. The second obstacle with the same identification information is associated, ensuring the accuracy of associating the second obstacle. The above method uses a multi-layer association mechanism to make full use of the multi-dimensional information of obstacles, improve the association accuracy, and reduce false associations and missed associations. Furthermore, a multi-layered association mechanism is used to associate as many camera and radar obstacles as possible, providing more accurate position and velocity information during fusion state updates. This ensures the accuracy of the association results, thereby guaranteeing driving safety.

[0022] In conjunction with the first aspect or the second embodiment of the first aspect, in the third embodiment of the first aspect, based on the predicted positions of unrelated obstacles in each historically fused tracked obstacle within a preset time period, the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle under the top-down view is calculated, including:

[0023] Based on the predicted positions of unrelated obstacles in each historical fusion tracking obstacle within a preset time period, obtain the corresponding LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data for the unrelated obstacles in each historical fusion tracking obstacle.

[0024] From the lidar prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data, respectively, determine the corresponding lidar prediction position, millimeter-wave radar prediction position, camera prediction position, and V2X prediction position of each unrelated obstacle in the historical fusion tracking data at the most recent moment.

[0025] Calculate the candidate distance cost between the current position of the unrelated obstacle in each observed obstacle and the predicted positions of the lidar, millimeter-wave radar, camera, and V2X in each historically fused tracked obstacle;

[0026] Based on each candidate distance cost, calculate the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle from the top-down view.

[0027] The obstacle association method in multi-sensor data provided in this invention obtains LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data corresponding to unrelated obstacles in each historically fused tracked obstacle set within a preset time period, based on the predicted positions of these unrelated obstacles. This ensures the accuracy of the obtained LiDAR, millimeter-wave radar, camera, and V2X prediction data. Furthermore, it determines the most recent LiDAR prediction position, millimeter-wave radar prediction position, camera prediction position, and V2X prediction position of the unrelated obstacles in each historically fused tracked obstacle set from the LiDAR, millimeter-wave radar, camera, and V2X prediction data, respectively. This ensures the accuracy of the determined most recent LiDAR, millimeter-wave radar, camera, and V2X prediction positions of the unrelated obstacles in each historically fused tracked obstacle set. Candidate distance costs are calculated between the current positions of unrelated obstacles in each observed obstacle and the predicted positions of LiDAR, millimeter-wave radar, camera, and V2X obstacles in each historically fused and tracked obstacle, ensuring the accuracy of the calculated candidate distance costs. Based on the candidate distance costs, Euclidean distance cost matrices are calculated between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused and tracked obstacle from the top-down view. This ensures the accuracy of the calculated Euclidean distance cost matrices between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused and tracked obstacle from the top-down view, thereby ensuring the accuracy of associating unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused and tracked obstacle from the top-down view.

[0028] In conjunction with the third implementation of the first aspect, in the fourth implementation of the first aspect, based on each candidate distance cost, the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle under the top-view perspective is calculated, including:

[0029] For unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, obtain the candidate distance cost between the current position of the unrelated obstacle in the observed obstacle and the predicted position of the lidar, millimeter-wave radar, camera, and V2X corresponding to the unrelated obstacle in the historical fusion tracking obstacle.

[0030] The minimum distance is selected from the candidate distance costs and determined as the target distance cost between unrelated obstacles in the observed obstacles and unrelated obstacles in the historical fusion tracking obstacles;

[0031] Based on the target distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, determine the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle from the top-down view.

[0032] The obstacle association method in multi-sensor data provided in this invention obtains candidate distance costs between the current position of the unassociated obstacle in each observed obstacle and the predicted positions of the unassociated obstacles in each historically fused and tracked obstacle, including the predicted positions of LiDAR, millimeter-wave radar, camera, and V2X, ensuring the accuracy of the calculated candidate distance costs. The minimum distance is selected from the candidate distance costs and determined as the target distance cost between the unassociated obstacle in the observed obstacle and the unassociated obstacle in the historically fused and tracked obstacle, ensuring the accuracy of the determined target distance cost. Based on the target distance costs between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused and tracked obstacle, a Euclidean distance cost matrix is ​​determined between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused and tracked obstacle from a top-down perspective, ensuring the accuracy of the determined Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused and tracked obstacle from a top-down perspective.

[0033] In conjunction with the first aspect or the second embodiment of the first aspect, in the fifth embodiment of the first aspect, based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles among the unrelated obstacles in the historical fusion-tracked obstacles are associated with the camera obstacles among the unrelated obstacles in the observed obstacles and the camera obstacles among the unrelated obstacles in the historical fusion-tracked obstacles under a perspective projection view, including:

[0034] Based on the predicted positions of unrelated obstacles in each historical fusion tracking obstacle within a preset time period, traverse the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles, and calculate the position distance cost, shape distance cost, and motion distance cost of the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle respectively.

[0035] Based on the relationship between location distance cost, shape distance cost, and motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle;

[0036] Based on the magnitude of the comprehensive distance cost, under the perspective projection view, unrelated obstacles in the observed obstacles and radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with camera obstacles in the unrelated obstacles in the observed obstacles and camera obstacles in the unrelated obstacles in the historical fusion tracking obstacles.

[0037] The obstacle association method in multi-sensor data provided in this invention calculates the positional distance cost, shape distance cost, and motion distance cost between unrelated obstacles in each observed obstacle and each unrelated obstacle in each historically fused tracked obstacle within a preset time period, based on the predicted positions of unrelated obstacles in each observed obstacle and each unrelated obstacle in each historically fused tracked obstacle. This ensures the accuracy of the calculated positional distance cost, shape distance cost, and motion distance cost. Furthermore, based on the relationship between the positional distance cost, shape distance cost, and motion distance cost, the comprehensive distance cost between unrelated obstacles in each observed obstacle and each unrelated obstacle in each historically fused tracked obstacle is calculated, ensuring the accuracy of the calculated comprehensive distance cost between unrelated obstacles in each observed obstacle and each unrelated obstacle in each historically fused tracked obstacle. Based on the magnitude of the comprehensive distance cost, under the perspective projection view, unrelated obstacles in the observed obstacles and radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with camera obstacles in the observed obstacles and camera obstacles in the unrelated obstacles in the historical fusion tracking obstacles. This ensures the accuracy of associating unrelated obstacles in the observed obstacles and radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles with camera obstacles in the perspective projection view.

[0038] In conjunction with the fifth embodiment of the first aspect, in the sixth embodiment of the first aspect, based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion-tracked obstacles are traversed, and the position distance cost, shape distance cost, and motion distance cost of the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion-tracked obstacle are calculated respectively, including:

[0039] Obtain the first center point of the radar obstacle and the second center point of the camera obstacle;

[0040] Calculate the first angle between the projection line segment of the line connecting the first center point to the second center point on the vertical projection plane of the vehicle and the central axis of the camera, and the second angle between the projection line segment of the line connecting the first center point to the second center point on the horizontal projection plane of the vehicle and the central axis of the camera.

[0041] When the first included angle is less than half of the horizontal field of view of the camera, the second included angle is less than half of the vertical field of view of the camera, and the Euclidean distance between the first center point and the second center point is less than the set distance threshold, the world coordinate system corresponding to the radar obstacle is converted into the pixel coordinate system according to the predicted position of the unrelated obstacle in each historical fusion tracking obstacle within the preset time period.

[0042] Project the vertices of the 3D bounding box of the LiDAR corresponding to the radar obstacle and the downsampled point cloud within the box, or the vertices of the 3D bounding box of the millimeter-wave radar corresponding to the radar obstacle, onto the pixel coordinate system, and take the smallest envelope rectangle as the projected 2D target box.

[0043] Obtain the width ratio between the pixel deviation in the width direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the width of the camera 2D frame; and the height ratio between the pixel deviation in the height direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the height of the camera 2D frame.

[0044] Determine the maximum value of the width ratio and height ratio as l1;

[0045] Calculate the ratio l2 of the number of vertices in the projected 2D target box corresponding to each radar obstacle to the total number of vertices of the radar obstacle;

[0046] Calculate the cosine distance l3 of the heading angle of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle;

[0047] Calculate the first ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame, and the second ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the camera 2D frame;

[0048] Calculate the average value l4 between the first ratio and the second ratio;

[0049] Calculate the location distance cost based on the relationship between l1, l2, l3, and l4;

[0050] Get the ratio s1 of the width of the projected 2D target frame to the width of the camera 2D frame; the ratio s2 of the height of the projected 2D target frame to the height of the camera 2D frame; and the ratio s3 of the area of ​​the projected 2D target frame to the area of ​​the camera 2D frame.

[0051] Calculate the shape distance cost based on the relationship between s1, s2, and s3;

[0052] The ratio of the velocities of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the x-direction is m1; the ratio of the velocities of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the y-direction is m2; and the ratio of the absolute velocity difference of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the camera data to the absolute velocity in the radar data is m3.

[0053] Calculate the cost of movement distance based on the relationship between m1, m2, and m3.

[0054] The obstacle association method in multi-sensor data provided in this invention obtains a first center point of radar obstacles and a second center point of camera obstacles; calculates the first angle between the projection of the line segment connecting the first and second center points onto the vehicle's vertical projection plane and the camera's central axis, and the second angle between the projection of the line segment connecting the first and second center points onto the vehicle's horizontal projection plane and the camera's central axis, ensuring the accuracy of the calculated first and second angles. When the first angle is less than half of the camera's horizontal field of view, the second angle is less than half of the camera's vertical field of view, and the Euclidean distance between the first and second center points is less than a set distance threshold, the world coordinate system corresponding to the radar obstacle is converted to a pixel coordinate system based on the predicted positions of unassociated obstacles in each historical fusion-tracked obstacle within a preset time period, ensuring the accuracy of the conversion from the world coordinate system to the pixel coordinate system. The vertices of the 3D bounding box corresponding to the radar obstacle and the downsampled point cloud within the box, or the vertices of the 3D bounding box corresponding to the millimeter-wave radar obstacle, are projected onto the pixel coordinate system. The smallest envelope rectangle is taken as the projected 2D target box, ensuring the accuracy of the selected projected 2D target box. The width ratio between the center point of the projected 2D target box and the center point of the camera 2D bounding box in the width direction and the width of the camera 2D bounding box, and the height ratio between the center point of the projected 2D target box and the center point of the camera 2D bounding box in the height direction and the height of the camera 2D bounding box, are obtained, ensuring the accuracy of the obtained width and height ratios. The maximum value of the width and height ratios is determined as l1, ensuring the accuracy of the determined l1. The ratio l2 of the number of vertices in the projected 2D target box corresponding to each radar obstacle to the total number of vertices of the radar obstacle is calculated, ensuring the accuracy of the calculated l2. The cosine distance l3 of the heading angle of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle is calculated, ensuring the accuracy of the calculated l3. The accuracy of the calculated ratios is ensured by calculating a first ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame, and a second ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the camera 2D frame. The average value l4 between the first and second ratios is also calculated, ensuring the accuracy of the calculated l4. Based on the relationships between l1, l2, l3, and l4, the positional distance cost is calculated, ensuring the accuracy of the calculated positional distance cost. The ratios s1 (width of the projected 2D target frame) to s2 (width of the camera 2D frame), s2 (height of the projected 2D target frame) to s2 (height of the camera 2D frame), and s3 (area of ​​the projected 2D target frame) to s3 are obtained. Based on the relationships between s1, s2, and s3, the shape distance cost is calculated, ensuring the accuracy of the calculated shape distance cost.The system obtains the velocity ratio m1 of unrelated obstacles in the x-direction among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles, the velocity ratio m2 of unrelated obstacles in the y-direction among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles, and the ratio m3 of the absolute velocity difference of unrelated obstacles in the camera data to the absolute velocity in the radar data among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles. Based on the relationship between m1, m2, and m3, the motion distance cost is calculated, ensuring the accuracy of the calculated motion distance cost.

[0055] In conjunction with the fifth embodiment of the first aspect, in the seventh embodiment of the first aspect, based on the relationship between position distance cost, shape distance cost, and motion distance cost, a comprehensive distance cost is calculated between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle, including:

[0056] Obtain the IOU reward factor and category reward factor;

[0057] Based on the relationship between IOU reward factor, category reward factor, location distance cost, shape distance cost, and motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle.

[0058] The obstacle association method in multi-sensor data provided in this invention obtains the IOU reward factor and the category reward factor; based on the relationship between the IOU reward factor, the category reward factor, the location distance cost, the shape distance cost, and the motion distance cost, it calculates the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle, thus ensuring the accuracy of the calculated comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle.

[0059] According to a second aspect, embodiments of the present invention also provide an obstacle association device in multi-sensor data, comprising:

[0060] The acquisition module is used to acquire sensor data at the current moment and acquire historical fusion tracked obstacles; historical fusion tracked obstacles are used to represent obstacles after associating them with obstacles in the historical sensor data of the previous moment.

[0061] The prediction module is used to predict the predicted position of each historically fused tracked obstacle within a preset time period using a preset prediction method;

[0062] The determination module is used to compare the observed obstacles in the current radar sensor data with the historical fusion-tracked obstacles when the current sensor data is the current radar sensor data, determine the first obstacle in the historical fusion-tracked obstacles that has the same obstacle identification information as the observed obstacles, and associate the first obstacle;

[0063] The first calculation module is used to calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle when the number of unrelated obstacles in the observed obstacle is greater than zero and the number of unrelated obstacles in the historically fused tracked obstacle is also greater than zero. Based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the module calculates the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle from the top-down perspective. Based on the Euclidean distance cost matrix, the module uses the Hungarian matching algorithm to perform association and update the association results.

[0064] The first association module is used to associate the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated ...

[0065] The multi-sensor data target association device provided in this invention acquires sensor data at the current moment and acquires historically fused tracked obstacles. It uses a preset prediction method to predict the predicted positions of each historically fused tracked obstacle within a preset time period, ensuring the accuracy of the predicted positions. When the current sensor data is radar sensor data, the observed obstacles in the current radar sensor data are compared with the historically fused tracked obstacles to identify the first obstacle whose obstacle identification information is the same as that in the observed obstacles. This first obstacle is then associated, thus achieving rapid and accurate association of the first obstacle. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle is calculated from the top-down perspective. Based on the Euclidean distance cost matrix, the Hungarian matching algorithm is used for association, and the association results are updated. This achieves the association of unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle from the top-down perspective. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles among the unrelated obstacles in the historically fused tracked obstacles are associated with the camera obstacles among the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historically fused tracked obstacles from the perspective projection view, and the association results are updated. This device achieves accurate association between radar obstacles and camera obstacles from a perspective projection viewpoint. It employs a multi-layer association mechanism, fully utilizing multi-dimensional obstacle information to improve association accuracy and reduce false and missed associations. Furthermore, the multi-layer association mechanism associates as many camera and radar obstacles as possible, providing more accurate position and velocity information for fusion state updates. This ensures the accuracy of the association results, thereby guaranteeing driving safety.

[0066] According to a third aspect, embodiments of the present invention provide an electronic device, including a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the obstacle association method in multi-sensor data in the first aspect or any embodiment of the first aspect.

[0067] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the multi-sensor data target association method of the first aspect or any embodiment of the first aspect. Attached Figure Description

[0068] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0069] Figure 1 This is a flowchart of the obstacle association method in multi-sensor data provided in the embodiments of the present invention;

[0070] Figure 2 This is a flowchart of an obstacle association method in multi-sensor data provided by another embodiment of the present invention;

[0071] Figure 3 This is a flowchart of an obstacle association method in multi-sensor data provided by another embodiment of the present invention;

[0072] Figure 4 This is a flowchart of an obstacle association method in multi-sensor data provided by another embodiment of the present invention;

[0073] Figure 5 This is a flowchart of an obstacle association method in multi-sensor data provided by another embodiment of the present invention;

[0074] Figure 6 This is a functional block diagram of the obstacle association device in multi-sensor data provided in the embodiments of the present invention;

[0075] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] It should be noted that the obstacle association method in multi-sensor data provided in this application embodiment can be executed by a device for obstacle association in multi-sensor data. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The electronic device can be a processor in the control system of an intelligent driving vehicle, or a processor independent of the intelligent driving vehicle. The processor can be a server or a terminal. In this application embodiment, the server can be a single server or a server cluster composed of multiple servers. The terminal in this application embodiment can be a smartphone, personal computer, tablet computer, wearable device, or other intelligent hardware device such as an intelligent robot. The following method embodiments will use an electronic device as an example for explanation.

[0078] In one embodiment of this application, such as Figure 1 As shown, a method for associating obstacles in multi-sensor data is provided. Taking the application of this method to an electronic device as an example, the method includes the following steps:

[0079] S11. Obtain the sensor data at the current moment and obtain historical fusion tracking obstacles.

[0080] Among them, historical fusion tracking obstacles are used to characterize obstacles after associating them with obstacles in the historical sensor data of the previous moment.

[0081] Optionally, the electronic device can receive sensor data at the current time input by the user, or it can receive sensor data at the current time sent by the sensor device based on the connection with the sensor device, or it can receive sensor data at the current time sent by other devices. This application embodiment does not specifically limit the method by which the electronic device obtains sensor data at the current time.

[0082] The sensor device can be a lidar sensor, a millimeter-wave radar sensor, a camera, or other sensor devices. This application does not specifically limit the sensor device.

[0083] Optionally, the electronic device can read historical fusion data to obtain historical fusion-tracked obstacles.

[0084] S12. Use a preset prediction method to predict the predicted position of each historical fusion tracking obstacle within a preset time period.

[0085] Specifically, electronic devices can use a uniform motion model to predict the predicted positions of various historically fused tracking obstacles within a preset time period.

[0086] The uniform motion model can be received by the electronic device from the user or other devices, or it can be obtained by the electronic device through training data. This application does not specifically limit the way the electronic device obtains the uniform motion model.

[0087] S13. When the sensor data at the current moment is the radar sensor data at the current moment, compare the observed obstacles in the radar sensor data at the current moment with the historical fusion tracked obstacles, determine the first obstacle in the historical fusion tracked obstacles that has the same obstacle identification information as the observed obstacles, and associate the first obstacle.

[0088] Optionally, the electronic device can acquire the data tag corresponding to the sensor data at the current moment, read the data tag, and determine whether the sensor data at the current moment is the radar sensor data at the current moment.

[0089] Optionally, the electronic device can also read the sensor data at the current moment and determine whether the sensor data at the current moment is the radar sensor data at the current moment based on the reading result.

[0090] When the sensor data at the current moment is the radar sensor data at the current moment, the electronic device can obtain the identification information corresponding to the observed obstacle in the radar sensor data at the current moment, as well as the identification information corresponding to the historically fused tracked obstacle. It compares the identification information of the observed obstacle in the radar sensor data at the current moment with the identification information of the historically fused tracked obstacle to determine the first obstacle with the same obstacle identification information as the observed obstacle, and associates the first obstacle.

[0091] S14. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle from the top-down view. Based on the Euclidean distance cost matrix, use the Hungarian matching algorithm to perform association and update the association results.

[0092] Specifically, after associating the first obstacle, the electronic device can obtain the number of unassociated obstacles in the observed obstacles and the number of unassociated obstacles in the historically fused and tracked obstacles. When the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused and tracked obstacles is also greater than zero, the electronic device can obtain the current position of the unassociated obstacles in the observed obstacles. Then, based on the predicted positions of the unassociated obstacles in each historically fused and tracked obstacle within a preset time period and the current positions of the unassociated obstacles in the observed obstacles, the Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused and tracked obstacle is calculated.

[0093] Then, the Hungarian matching algorithm is used to associate unrelated obstacles in the observed obstacles with unrelated obstacles in each historical fusion tracking obstacle, and the association results are updated.

[0094] This step will be explained in detail below.

[0095] S15. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated.

[0096] Specifically, after using the Hungarian matching algorithm to associate unrelated obstacles in the observed obstacles with unrelated obstacles in each historical fusion-tracked obstacle, and updating the association results, the electronic device again obtains the number of unrelated obstacles in the observed obstacles and the number of unrelated obstacles in the historical fusion-tracked obstacles.

[0097] When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, the electronic device, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, associates the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and updates the association results.

[0098] This step will be explained in detail below.

[0099] The obstacle association method in multi-sensor data provided in this invention acquires current-time sensor data and historically fused-tracked obstacles; it uses a preset prediction method to predict the predicted positions of each historically fused-tracked obstacle within a preset time period, ensuring the accuracy of the predicted positions. When the current-time sensor data is radar sensor data, the observed obstacles in the current-time radar sensor data are compared with the historically fused-tracked obstacles to identify the first obstacle among the historically fused-tracked obstacles that has the same obstacle identification information as the observed obstacles, and then associates the first obstacle, thereby achieving fast and accurate association of the first obstacle. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle is calculated from the top-down perspective. Based on the Euclidean distance cost matrix, the Hungarian matching algorithm is used for association, and the association results are updated. This achieves the association of unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle from the top-down perspective. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles among the unrelated obstacles in the historically fused tracked obstacles are associated with the camera obstacles among the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historically fused tracked obstacles from the perspective projection view, and the association results are updated. This method achieves accurate association between radar obstacles and camera obstacles from a perspective projection viewpoint. It employs a multi-layered association mechanism, fully utilizing the multi-dimensional information of obstacles to improve association accuracy and reduce false and missed associations. Furthermore, the multi-layered association mechanism associates as many camera and radar obstacles as possible, providing more accurate position and velocity information for fusion state updates. This ensures the accuracy of the association results, thereby guaranteeing driving safety.

[0100] In one embodiment of this application, such as Figure 2 As shown, a method for associating obstacles in multi-sensor data is provided. Taking the application of this method to an electronic device as an example, the method includes the following steps:

[0101] S21. Obtain the sensor data at the current moment and obtain historical fusion tracking obstacles.

[0102] Among them, historical fusion tracking obstacles are used to characterize obstacles after associating them with obstacles in the historical sensor data of the previous moment.

[0103] For details on this step, please refer to [link / reference]. Figure 1 The details of S11 will not be elaborated here.

[0104] S22. Obtain the position information of each candidate obstacle from the sensor data at the current moment.

[0105] Specifically, electronic devices can identify each candidate obstacle in the sensor data at the current moment, and determine the location information corresponding to each candidate obstacle based on the identification results.

[0106] S23. Based on the location information of each candidate obstacle, delete the candidate obstacles that are outside the target range to obtain the observed obstacles included in the sensor data at the current moment.

[0107] Specifically, after obtaining the location information of each candidate obstacle, the electronic device can identify candidate obstacles within the target range and candidate obstacles outside the target range, and then delete the candidate obstacles outside the target range to obtain the observed obstacles included in the sensor data at the current moment. The target range can be a drivable area.

[0108] S24. Use a preset prediction method to predict the predicted position of each historical fusion tracking obstacle within a preset time period.

[0109] For details on this step, please refer to [link / reference]. Figure 1 The details of S12 will not be elaborated here.

[0110] S25. When the sensor data at the current moment is the radar sensor data at the current moment, compare the observed obstacles in the radar sensor data at the current moment with the historical fusion tracked obstacles, determine the first obstacle in the historical fusion tracked obstacles that has the same obstacle identification information as the observed obstacles, and associate the first obstacle.

[0111] For details on this step, please refer to [link / reference]. Figure 1 The details of S13 will not be elaborated here.

[0112] S26. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle from the top-down view. Based on the Euclidean distance cost matrix, use the Hungarian matching algorithm to perform association and update the association results.

[0113] For details on this step, please refer to [link / reference]. Figure 1 The details of S14 will not be elaborated here.

[0114] S27. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated.

[0115] For details on this step, please refer to [link / reference]. Figure 1 The details of S15 will not be elaborated here.

[0116] The obstacle association method in multi-sensor data provided in this invention obtains the location information of each candidate obstacle in the sensor data at the current moment; based on the location information of each candidate obstacle, candidate obstacles outside the target range are deleted to obtain the observed obstacles included in the sensor data at the current moment, ensuring the accuracy of the obtained observed obstacles. The above method uses target range-based filtering of candidate obstacles, reducing the time consumption and false associations in obstacle association.

[0117] In one embodiment of this application, such as Figure 3 As shown, the obstacle association method in the above multi-sensor data may further include the following steps:

[0118] S31. When the sensor data at the current moment is not the radar sensor data at the current moment, and when the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, according to the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle from the top-down perspective. Based on the Euclidean distance cost matrix, use the Hungarian matching algorithm to perform association and update the association results.

[0119] Optionally, the electronic device can acquire the data tag corresponding to the sensor data at the current moment, read the data tag, and determine whether the sensor data at the current moment is the radar sensor data at the current moment.

[0120] Optionally, the electronic device can also read the sensor data at the current moment and determine whether the sensor data at the current moment is the radar sensor data at the current moment based on the reading result.

[0121] When the sensor data at the current moment is not the radar sensor data at the current moment, the electronic device can obtain the number of unrelated obstacles in the observed obstacles and the number of unrelated obstacles in the historically fused tracked obstacles. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, the electronic device can obtain the current position of the unrelated obstacles in the observed obstacles. Then, based on the predicted position of each unrelated obstacle in the historically fused tracked obstacles within a preset time period and the current position of the unrelated obstacles in the observed obstacles, the Euclidean distance cost matrix between each unrelated obstacle in the observed obstacles and each unrelated obstacle in the historically fused tracked obstacles is calculated.

[0122] Then, the Hungarian matching algorithm is used to associate unrelated obstacles in the observed obstacles with unrelated obstacles in each historical fusion tracking obstacle, and the association results are updated.

[0123] This step will be explained in detail below.

[0124] S32. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated.

[0125] Specifically, after using the Hungarian matching algorithm to associate unrelated obstacles in the observed obstacles with unrelated obstacles in each historical fusion-tracked obstacle, and updating the association results, the electronic device again obtains the number of unrelated obstacles in the observed obstacles and the number of unrelated obstacles in the historical fusion-tracked obstacles.

[0126] When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, the electronic device, based on the predicted positions of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, associates the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles with the camera obstacles in the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles under the perspective projection view, and updates the association results.

[0127] This step will be explained in detail below.

[0128] S33. When the sensor data at the current moment is the camera sensor data at the current moment, compare the identification information of the unassociated camera obstacles in the historical fusion tracking obstacles with the identification information of the obstacles in the observed obstacles, and associate the second obstacle with the same identification information.

[0129] Optionally, the electronic device can acquire the data tag corresponding to the sensor data at the current moment, read the data tag, and determine whether the sensor data at the current moment is the camera sensor data at the current moment.

[0130] Optionally, the electronic device can also read the sensor data at the current moment and determine whether the sensor data at the current moment is the camera sensor data at the current moment based on the reading result.

[0131] When the sensor data at the current moment is the camera sensor data at the current moment, the electronic device can obtain the identification information of the unassociated camera obstacles in the historical fusion tracking obstacles and the identification information of the obstacles in the observed obstacles in the current moment camera sensor data. It compares the identification information of the unassociated camera obstacles in the historical fusion tracking obstacles with the identification information of the obstacles in the observed obstacles, and associates the second obstacle with the same identification information.

[0132] The obstacle association method in multi-sensor data provided in this invention addresses the following situations: when the current sensor data is not the current radar sensor data, and when the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused tracked obstacles is also greater than zero, the method calculates the Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle from a top-down perspective, based on the predicted positions of the unassociated obstacles in each historically fused tracked obstacle within a preset time period. Based on the Euclidean distance cost matrix, the method uses a Hungarian matching algorithm for association, updates the association results, and ensures the accuracy of the calculated Euclidean distance cost matrix between the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle from a top-down perspective. This, in turn, ensures the accuracy of using the Hungarian matching algorithm to associate the unassociated obstacles in each observed obstacle and the unassociated obstacles in each historically fused tracked obstacle. Then, when the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unassociated obstacles in each historically fused tracked obstacle within a preset time period, the unassociated obstacles in the observed obstacles and the radar obstacles among the unassociated obstacles in the historically fused tracked obstacles are associated with the camera obstacles among the unassociated obstacles in the observed obstacles and the camera obstacles among the unassociated obstacles in the historically fused tracked obstacles under the perspective projection view. The association results are updated, ensuring the accuracy of associating radar obstacles and camera obstacles under the perspective projection view. When the sensor data at the current moment is the camera sensor data at the current moment, the identification information of the unassociated camera obstacles in the historically fused tracked obstacles is compared with the identification information of the obstacles in the observed obstacles. The second obstacle with the same identification information is associated, ensuring the accuracy of associating the second obstacle. The above method uses a multi-layer association mechanism to make full use of the multi-dimensional information of obstacles, improve the association accuracy, and reduce false associations and missed associations. Furthermore, a multi-layered association mechanism is used to associate as many camera and radar obstacles as possible, providing more accurate position and velocity information during fusion state updates. This ensures the accuracy of the association results, thereby guaranteeing driving safety.

[0133] In an optional embodiment of this application, such as Figure 4 As shown, the step S14 and step S31 above, "calculating the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle within a preset time period based on the predicted positions of unrelated obstacles in each historically fused tracked obstacle", may include the following steps:

[0134] S41. Based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, obtain the corresponding LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data for the unrelated obstacles in each historical fusion-tracked obstacle.

[0135] Specifically, after acquiring the predicted positions of each historically fused tracked obstacle within a preset time period, the electronic device acquires the corresponding LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data of each historically fused tracked obstacle based on the predicted positions of unrelated obstacles within the preset time period.

[0136] S42. From the lidar prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data, respectively determine the corresponding lidar prediction position, millimeter-wave radar prediction position, camera prediction position, and V2X prediction position of each unrelated obstacle in the historical fusion tracking data at the most recent moment.

[0137] Specifically, after acquiring the lidar prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data corresponding to unrelated obstacles in each historical fusion-tracked obstacle, the electronic device can determine the corresponding lidar prediction position, millimeter-wave radar prediction position, camera prediction position, and V2X prediction position of the unrelated obstacles in each historical fusion-tracked obstacle at the most recent moment from the lidar prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data.

[0138] S43. Calculate the candidate distance cost between the current position of the unrelated obstacle in each observed obstacle and the predicted position of the lidar, millimeter-wave radar, camera, and V2X in each historical fused tracked obstacle.

[0139] Specifically, after acquiring the predicted positions of LiDAR, millimeter-wave radar, camera, and V2X for unrelated obstacles in the most recent historical fusion-tracked obstacles, the electronic device can calculate the candidate distance cost between the current position of the unrelated obstacle in each observed obstacle and the predicted positions of LiDAR, millimeter-wave radar, camera, and V2X for the unrelated obstacles in each historical fusion-tracked obstacle using a preset method, based on the current position of the unrelated obstacle in each observed obstacle.

[0140] The preset method can be Euclidean distance calculation method, Manhattan distance calculation method, Chebyshev distance calculation method, Minkowski distance calculation method, or standardized Euclidean distance calculation method.

[0141] The embodiments of this application do not specifically limit the preset methods such as Mahalanobis distance calculation method and cosine angle calculation method.

[0142] S44. Based on each candidate distance cost, calculate the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle from the top-down view.

[0143] In one optional embodiment of this application, the electronic device can calculate the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle from a top-down perspective, based on each candidate distance cost.

[0144] In an optional embodiment of this application, the above-mentioned S44 "calculating the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle under the top-view perspective, based on each candidate distance cost" may include the following steps:

[0145] S441. For unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, obtain the candidate distance cost between the current position of the unrelated obstacle in the observed obstacle and the predicted position of the lidar, millimeter-wave radar, camera, and V2X in the unrelated obstacle in the historical fusion tracking obstacle.

[0146] Specifically, for unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, the electronic device can obtain the candidate distance cost between the current position of the unrelated obstacle in the observed obstacle and the predicted position of the lidar, millimeter-wave radar, camera, and V2X in the unrelated obstacle in the historical fusion tracking obstacle.

[0147] S442. Select the minimum distance from the candidate distance costs and determine it as the target distance cost between the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles.

[0148] Specifically, for unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, the electronic device compares the current position corresponding to the unrelated obstacle in the observed obstacle with the candidate distance cost between the lidar predicted position, millimeter-wave radar predicted position, camera predicted position and V2X predicted position corresponding to the unrelated obstacle in the historical fusion tracking obstacle, and selects the minimum distance from the candidate distance cost to determine the target distance cost between the unrelated obstacle in the observed obstacle and the unrelated obstacle in the historical fusion tracking obstacle.

[0149] S443. Based on the target distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle, determine the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle from the top-down view.

[0150] Specifically, after determining the target distance cost between unrelated obstacles in the observed obstacles and unrelated obstacles in the historical fusion tracking obstacles, the electronic device can determine the Euclidean distance cost matrix between unrelated obstacles in the observed obstacles and unrelated obstacles in the historical fusion tracking obstacles from the top-down perspective, based on the target distance cost between unrelated obstacles in the observed obstacles and unrelated obstacles in the historical fusion tracking obstacles.

[0151] The obstacle association method in multi-sensor data provided in this invention obtains the predicted positions of unassociated obstacles in each historically fused tracked obstacle set within a preset time period, including LiDAR prediction data, millimeter-wave radar prediction data, camera prediction data, and V2X prediction data. This ensures the accuracy of the obtained LiDAR, millimeter-wave radar, camera, and V2X prediction data for unassociated obstacles in each historically fused tracked obstacle set. Candidate distance costs are calculated between the current position of the unassociated obstacle in each observed obstacle set and the predicted positions of the LiDAR, millimeter-wave radar, camera, and V2X obstacles in each historically fused tracked obstacle set, ensuring the accuracy of the calculated candidate distance costs. For unrelated obstacles in each observed obstacle and each unrelated obstacle in each historically fused tracking obstacle, candidate distance costs are obtained between the current position of the unrelated obstacle in the observed obstacle and the predicted positions of the LiDAR, millimeter-wave radar, camera, and V2X in the historically fused tracking obstacle, ensuring the accuracy of the calculated candidate distance costs. The minimum distance is selected from the candidate distance costs and determined as the target distance cost between the unrelated obstacle in the observed obstacle and the unrelated obstacle in the historically fused tracking obstacle, ensuring the accuracy of the determined target distance cost. Based on the target distance costs between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracking obstacle, a Euclidean distance cost matrix is ​​determined between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracking obstacle from the top-down view, ensuring the accuracy of the determined Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracking obstacle from the top-down view.

[0152] In one optional embodiment of this application, such as Figure 5 As shown, the step S15 or / and step S32 described above, "based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, associating unrelated obstacles in the observed obstacles and radar obstacles in the historical fusion-tracked obstacles with camera obstacles in the observed obstacles and camera obstacles in the historical fusion-tracked obstacles under a perspective projection view," may include the following steps:

[0153] S51. Based on the predicted positions of unrelated obstacles in each historical fusion tracking obstacle within a preset time period, traverse the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles, and calculate the position distance cost, shape distance cost, and motion distance cost of the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle.

[0154] In an optional embodiment of this application, the above step S51, "based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, traversing the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion-tracked obstacles, and calculating the position distance cost, shape distance cost, and motion distance cost between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion-tracked obstacle," may include the following steps:

[0155] S511, Obtain the first center point of the radar obstacle and the second center point of the camera obstacle.

[0156] Specifically, electronic devices can identify radar obstacles and camera obstacles, and determine the first center point of the radar obstacle and the second center point of the camera obstacle based on the identification results.

[0157] S512. Calculate the first angle between the projection line segment of the line connecting the first center point to the second center point on the vertical projection plane of the vehicle and the central axis of the camera, and the second angle between the projection line segment of the line connecting the first center point to the second center point on the horizontal projection plane of the vehicle and the central axis of the camera.

[0158] Specifically, after determining the first center and the second center, the electronic device can generate a line connecting the first center point and the second center point. Then, the electronic device calculates the first angle between the projection of the line connecting the first center point to the second center point onto the vehicle's vertical projection plane and the camera's central axis, and the second angle between the projection of the line connecting the first center point to the second center point onto the vehicle's horizontal projection plane and the camera's central axis.

[0159] S513. When the first included angle is less than half of the horizontal field of view of the camera, the second included angle is less than half of the vertical field of view of the camera, and the Euclidean distance between the first center point and the second center point is less than a set distance threshold, the world coordinate system corresponding to the radar obstacle is converted into the pixel coordinate system according to the predicted position of the unrelated obstacle in each historical fusion tracking obstacle within a preset time period.

[0160] Specifically, after calculating the first and second included angles, the electronic device can compare the first included angle with the horizontal field of view of the camera, compare the second included angle with the vertical field of view of the camera, and compare the Euclidean distance between the first and second center points with a set distance threshold. When the first included angle is less than half of the horizontal field of view of the camera, the second included angle is less than half of the vertical field of view of the camera, and the Euclidean distance between the first and second center points is less than the set distance threshold, the electronic device can convert the world coordinate system corresponding to the radar obstacle into the obstacle coordinate system corresponding to the radar obstacle, and convert the obstacle coordinate system corresponding to the radar obstacle into the pixel coordinate system, based on the predicted positions of unrelated obstacles in each historical fusion tracking obstacle within a preset time period.

[0161] S514. Project the vertices of the 3D bounding box of the LiDAR corresponding to the radar obstacle and the downsampled point cloud within the box, or the vertices of the 3D bounding box of the millimeter-wave radar corresponding to the radar obstacle, onto the pixel coordinate system, and take the smallest envelope rectangle as the projected 2D target box.

[0162] Specifically, the electronic device can project the vertices of the 3D bounding box of the LiDAR corresponding to the radar obstacle and the downsampled point cloud within the box, or the vertices of the 3D bounding box of the millimeter-wave radar corresponding to the radar obstacle, onto the pixel coordinate system. Then, the electronic device can take the smallest envelope rectangle as the projected 2D target box.

[0163] S515. Obtain the width ratio between the pixel deviation in the width direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the width of the camera 2D frame, and the height ratio between the pixel deviation in the height direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the height of the camera 2D frame.

[0164] Specifically, the electronic device can acquire the center point of the projected 2D target frame and the center point of the camera 2D frame, and then calculate the width ratio between the pixel deviation in the width direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the width of the camera 2D frame. Then, it calculates the height ratio between the pixel deviation in the height direction between the center point of the projected 2D target frame and the center point of the camera 2D frame and the height of the camera 2D frame.

[0165] S516. Determine the maximum value of the width ratio and height ratio as l1.

[0166] Specifically, the electronic device can compare the width ratio and the height ratio, and then determine the maximum value of the width ratio and the height ratio as l1.

[0167] S517. Calculate the ratio l2 of the number of vertices in the projected 2D target frame corresponding to each radar obstacle to the total number of vertices of the radar obstacle.

[0168] Specifically, the electronic device can calculate the number of vertices in the projected 2D target frame corresponding to each radar obstacle, and then calculate the ratio l2 of the number of vertices in the projected 2D target frame corresponding to each radar obstacle to the total number of vertices of the radar obstacle.

[0169] S518. Calculate the cosine distance l3 of the heading angle of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle.

[0170] Specifically, the electronic device can also acquire the heading angles of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle, and then calculate the cosine distance l3 of the heading angles of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle.

[0171] S519. Calculate the first ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame, and the second ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the camera 2D frame.

[0172] Specifically, the electronic device can obtain the intersection area of ​​the projected 2D target frame and the camera 2D frame, as well as the area of ​​the projected 2D target frame, and then calculate a first ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame.

[0173] The electronic device can also acquire the area of ​​the camera 2D frame and calculate a second ratio of the intersection area of ​​the projected 2D target frame and the camera 2D frame to the area of ​​the camera 2D frame.

[0174] S5110, Calculate the average value l4 between the first ratio and the second ratio.

[0175] Specifically, after calculating the first ratio and the second ratio, the electronic device can calculate the average value l4 between the first ratio and the second ratio.

[0176] S5111. Calculate the location distance cost based on the relationship between l1, l2, l3, and l4.

[0177] In one optional embodiment of this application, the electronic device can add l1, l2, l3, and l4 together and then calculate the average value to obtain the location distance cost.

[0178] In one optional embodiment of this application, the electronic device can further obtain the first weight q1 corresponding to l1, the first weight q2 corresponding to l2, the first weight q3 corresponding to l3, and the first weight q4 corresponding to l4, respectively. Then, the electronic device can multiply l1 by q1, l2 by q2, l3 by q3, and l4 by q4, add them together, and then divide by the sum of q1, q2, q3, and q4 to calculate the location distance cost.

[0179] S5112. Obtain the ratio s1 of the width of the projected 2D target frame to the width of the camera 2D frame; the ratio s2 of the height of the projected 2D target frame to the height of the camera 2D frame; and the ratio s3 of the area of ​​the projected 2D target frame to the area of ​​the camera 2D frame.

[0180] Specifically, the electronic device can acquire the width of the projected 2D target frame and the width of the camera 2D frame, and calculate the ratio s1 between the width of the projected 2D target frame and the width of the camera 2D frame. Then, the electronic device can also acquire the height of the projected 2D target frame and the height of the camera 2D frame, and calculate the ratio s2 between the height of the projected 2D target frame and the height of the camera 2D frame. The electronic device can also acquire the area of ​​the projected 2D target frame and the area of ​​the projected 2D target frame, and calculate the ratio s3 between the area of ​​the projected 2D target frame and the area of ​​the camera 2D frame.

[0181] S5113. Calculate the shape distance cost based on the relationship between s1, s2, and s3.

[0182] In one optional embodiment of this application, the electronic device can add s1, s2, and s3 together and then calculate the average value to obtain the shape distance cost.

[0183] In one optional embodiment of this application, the electronic device can further obtain the fifth weight q5 corresponding to s1, the sixth weight q6 corresponding to s2, and the seventh weight q7 corresponding to s3. Then, the electronic device can multiply s1 by q5, s2 by q6, and s3 by q7, add them together, and then divide by the sum of q5, q6, and q7 to calculate the shape distance cost.

[0184] S5114. Obtain the velocity ratio m1 of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the x direction, the velocity ratio m2 of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the y direction, and the ratio m3 of the absolute velocity difference of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the camera data to the absolute velocity in the radar data.

[0185] Specifically, the electronic device can acquire the velocities of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the x-direction, and then calculate the ratio m1 of the velocities of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle in the x-direction.

[0186] The electronic device can also acquire the velocities of unrelated obstacles in the observed obstacles and unrelated obstacles in each historical fusion tracked obstacles in the y-direction, and then calculate the ratio m2 of the velocities of unrelated obstacles in the observed obstacles and unrelated obstacles in each historical fusion tracked obstacles in the y-direction.

[0187] The electronic device can also acquire the ratio m3 of the absolute velocity difference between unrelated obstacles in camera data and the absolute velocity in radar data for unrelated obstacles in each observed obstacle and each historical fusion tracked obstacle.

[0188] S5115. Calculate the movement distance cost based on the relationship between m1, m2, and m3.

[0189] In one optional embodiment of this application, the electronic device can add m1, m2, and m3 together and then calculate the average value to obtain the motion distance cost.

[0190] In one optional embodiment of this application, the electronic device can further obtain the eighth weight q8 corresponding to m1, the ninth weight q9 corresponding to m2, and the tenth weight q10 corresponding to m3. Then, the electronic device can multiply m1 by q8, m2 by q9, and m3 by q10, add them together, and then divide by the sum of q8, q9, and q10 to calculate the motion distance cost.

[0191] S52. Based on the relationship between position distance cost, shape distance cost, and motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle.

[0192] In an optional embodiment of this application, step S52, "calculating the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle based on the relationship between position distance cost, shape distance cost, and motion distance cost," may include the following steps:

[0193] S521, Obtain IOU reward factors and category reward factors.

[0194] Specifically, the electronic device can receive IOU reward factors and category reward factors input by the user, and can also receive IOU reward factors and category reward factors sent by other devices. This application embodiment does not specifically limit the way the electronic device obtains IOU reward factors and category reward factors.

[0195] S522. Based on the relationship between IOU reward factor, category reward factor, location distance cost, shape distance cost, and motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle.

[0196] In one optional embodiment of this application, the electronic device can calculate the sum of the location distance cost, shape distance cost, and motion distance cost, and then multiply the sum of the location distance cost, shape distance cost, and motion distance cost by the IOU reward factor, and then multiply by the category reward factor to calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle.

[0197] In one optional embodiment of this application, the electronic device can obtain the eleventh weight q11 corresponding to the position distance cost, the twelfth weight q12 corresponding to the shape distance cost, and the thirteenth weight q13 corresponding to the motion distance cost. The electronic device can multiply the position distance cost by the eleventh weight q11, the shape distance cost by the twelfth weight q12, and the motion distance cost by the thirteenth weight q13, and then add them together to obtain a first distance cost sum. Then, the electronic device adds the eleventh weight q11, the twelfth weight q12, and the thirteenth weight q13 together to calculate the first weight sum. Then, the electronic device uses the quotient of the first distance cost sum divided by the first weight sum, multiplies it by the IOU reward factor, and then multiplies it by the category reward factor to calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle.

[0198] S53. Based on the magnitude of the comprehensive distance cost, under the perspective projection view, associate the unrelated obstacles in the observed obstacles and the radar obstacles in the unrelated obstacles in the historical fusion tracking obstacles with the camera obstacles in the unrelated obstacles in the observed obstacles and the camera obstacles in the unrelated obstacles in the historical fusion tracking obstacles.

[0199] Specifically, after calculating the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle, the electronic device can associate the unrelated obstacles in the observed obstacle and the radar obstacles in the unrelated obstacles in the historically fused tracked obstacle with the camera obstacles in the unrelated obstacles in the observed obstacle and the unrelated obstacles in the historically fused tracked obstacle under the perspective projection view, based on the magnitude of the comprehensive distance cost.

[0200] The obstacle association method in multi-sensor data provided in this invention obtains a first center point of radar obstacles and a second center point of camera obstacles; calculates the first angle between the projection of the line segment connecting the first and second center points onto the vehicle's vertical projection plane and the camera's central axis, and the second angle between the projection of the line segment connecting the first and second center points onto the vehicle's horizontal projection plane and the camera's central axis, ensuring the accuracy of the calculated first and second angles. When the first angle is less than half of the camera's horizontal field of view, the second angle is less than half of the camera's vertical field of view, and the Euclidean distance between the first and second center points is less than a set distance threshold, the world coordinate system corresponding to the radar obstacle is converted to a pixel coordinate system based on the predicted positions of unassociated obstacles in each historical fusion-tracked obstacle within a preset time period, ensuring the accuracy of the conversion from the world coordinate system to the pixel coordinate system. The vertices of the 3D bounding box corresponding to the radar obstacle and the downsampled point cloud within the box, or the vertices of the 3D bounding box corresponding to the millimeter-wave radar obstacle, are projected onto the pixel coordinate system. The smallest envelope rectangle is taken as the projected 2D target box, ensuring the accuracy of the selected projected 2D target box. The width ratio between the center point of the projected 2D target box and the center point of the camera 2D bounding box in the width direction and the width of the camera 2D bounding box, and the height ratio between the center point of the projected 2D target box and the center point of the camera 2D bounding box in the height direction and the height of the camera 2D bounding box, are obtained, ensuring the accuracy of the obtained width and height ratios. The maximum value of the width and height ratios is determined as l1, ensuring the accuracy of the determined l1. The ratio l2 of the number of vertices in the projected 2D target box corresponding to each radar obstacle to the total number of vertices of the radar obstacle is calculated, ensuring the accuracy of the calculated l2. The cosine distance l3 of the heading angle of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle is calculated, ensuring the accuracy of the calculated l3. The accuracy of the calculated ratios is ensured by calculating a first ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame, and a second ratio between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the camera 2D frame. The average value l4 between the first and second ratios is also calculated, ensuring the accuracy of the calculated l4. Based on the relationships between l1, l2, l3, and l4, the positional distance cost is calculated, ensuring the accuracy of the calculated positional distance cost. The ratios s1 (width of the projected 2D target frame) to s2 (width of the camera 2D frame), s2 (height of the projected 2D target frame) to s2 (height of the camera 2D frame), and s3 (area of ​​the projected 2D target frame) to s3 are obtained. Based on the relationships between s1, s2, and s3, the shape distance cost is calculated, ensuring the accuracy of the calculated shape distance cost.The system obtains the velocity ratio m1 of unrelated obstacles in the x-direction among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles, the velocity ratio m2 of unrelated obstacles in the y-direction among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles, and the ratio m3 of the absolute velocity difference of unrelated obstacles in the camera data to the absolute velocity in the radar data among all observed obstacles and unrelated obstacles in the fusion tracking of all historical obstacles. Based on the relationship between m1, m2, and m3, the motion distance cost is calculated, ensuring the accuracy of the calculated motion distance cost.

[0201] Then, the IOU reward factor and category reward factor are obtained. Based on the relationship between the IOU reward factor, category reward factor, position distance cost, shape distance cost, and motion distance cost, the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle is calculated, ensuring the accuracy of the calculated comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historically fused tracked obstacle. Based on the magnitude of the comprehensive distance cost, under the perspective projection view, unrelated obstacles in the observed obstacles and radar obstacles in the historically fused tracked obstacles are associated with camera obstacles in the observed obstacles and unrelated obstacles in the historically fused tracked obstacles, ensuring the accuracy of the association between unrelated obstacles in the observed obstacles and radar obstacles in the historically fused tracked obstacles and camera obstacles in the historically fused tracked obstacles under the perspective projection view.

[0202] It should be understood that, although Figures 1-5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-5 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0203] like Figure 6 As shown, this embodiment provides a multi-sensor data target association device, including:

[0204] The first acquisition module 61 is used to acquire sensor data at the current moment and acquire historical fusion tracked obstacles; the historical fusion tracked obstacles are used to represent the obstacles after associating them with the obstacles in the historical sensor data of the previous moment.

[0205] Prediction module 62 is used to predict the predicted position of each historical fusion-tracked obstacle within a preset time period using a preset prediction method;

[0206] The determination module 63 is used to compare the observed obstacles in the current radar sensor data with the historical fusion tracking obstacles when the current sensor data is the current radar sensor data, determine the first obstacle in the historical fusion tracking obstacles that has the same obstacle identification information as the observed obstacles, and associate the first obstacle;

[0207] The first calculation module 64 is used to calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle when the number of unrelated obstacles in the observed obstacle is greater than zero and the number of unrelated obstacles in the historical fusion tracking obstacle is also greater than zero. Based on the predicted position of the unrelated obstacles in each historical fusion tracking obstacle within a preset time period, the module calculates the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion tracking obstacle from the top-down view. Based on the Euclidean distance cost matrix, the module uses the Hungarian matching algorithm to perform association and update the association result.

[0208] The first association module 65 is used to associate the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated ...

[0209] In one embodiment of this application, such as Figure 6 As shown, the obstacle association device in the above-mentioned multi-sensor data also includes:

[0210] The second acquisition module 66 is used to acquire the position information of each candidate obstacle in the sensor data at the current moment;

[0211] The deletion module 67 is used to delete candidate obstacles outside the target range based on the position information of each candidate obstacle, so as to obtain the observed obstacles included in the sensor data at the current moment.

[0212] In one embodiment of this application, such as Figure 6As shown, the obstacle association device in the above-mentioned multi-sensor data also includes:

[0213] The second calculation module 68 is used to calculate the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle when the sensor data at the current moment is not the radar sensor data at the current moment, and when the number of unrelated obstacles in the observed obstacle is greater than zero and the number of unrelated obstacles in the historically fused tracked obstacle is also greater than zero. Based on the predicted position of the unrelated obstacles in each historically fused tracked obstacle within a preset time period, the module calculates the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historically fused tracked obstacle from the top-down perspective. Based on the Euclidean distance cost matrix, the module uses the Hungarian matching algorithm to perform association and update the association results.

[0214] The second association module 69 is used to associate the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated ...

[0215] The third association module 610 is used to compare the identification information of the camera obstacle that is not associated in the historical fusion tracking obstacle with the identification information of the obstacle in the observed obstacle when the sensor data at the current time is the camera sensor data at the current time, and associate the second obstacle with the same identification information.

[0216] In one embodiment of this application, the first calculation module 64 or the second calculation module 68 is specifically used to obtain, based on the predicted positions of unrelated obstacles in each historically fused tracked obstacle within a preset time period, the corresponding LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data for the unrelated obstacles in each historically fused tracked obstacle; and to determine, from the LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data, the corresponding LiDAR prediction positions of the unrelated obstacles in each historically fused tracked obstacle within the LiDAR prediction data at the most recent moment, and the corresponding millimeter-wave radar obstacle prediction positions for the unrelated obstacles in each historically fused tracked obstacle within the LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data, respectively. The system calculates the corresponding millimeter-wave radar predicted position in the radar obstacle prediction data, the corresponding camera predicted position in the camera prediction data, and the corresponding V2X predicted position in the V2X prediction data; it also calculates the candidate distance cost between the current position of the unrelated obstacle in each observed obstacle and the predicted positions of the lidar, millimeter-wave radar, camera, and V2X obstacles in each historical fusion-tracked obstacle; and based on the candidate distance cost, it calculates the Euclidean distance cost matrix between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion-tracked obstacle from the top-down perspective.

[0217] In one embodiment of this application, the first calculation module 64 or the second calculation module 68 is specifically used to obtain, for each unrelated obstacle in the observed obstacles and each unrelated obstacle in the historical fusion tracking obstacles, the candidate distance cost between the current position of the unrelated obstacle in the observed obstacles and the predicted position of the lidar, millimeter-wave radar, camera, and V2X corresponding to the unrelated obstacle in the historical fusion tracking obstacles; select the minimum distance from the candidate distance costs and determine it as the target distance cost between the unrelated obstacle in the observed obstacles and the unrelated obstacle in the historical fusion tracking obstacles; and determine the Euclidean distance cost matrix between the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles from the top-down perspective based on the target distance cost between the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles.

[0218] In one embodiment of this application, the first association module 65 or the second association module 69 is specifically configured to, based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, traverse the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion-tracked obstacles, respectively calculate the position distance cost, shape distance cost, and motion distance cost between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion-tracked obstacle; calculate the comprehensive distance cost between the unrelated obstacles in each observed obstacle and the unrelated obstacles in each historical fusion-tracked obstacle based on the relationship between the position distance cost, shape distance cost, and motion distance cost; and, based on the magnitude of the comprehensive distance cost, determine the unrelated obstacles in the observed obstacles from the perspective projection viewpoint.

[0219] The obstacle and historical fusion tracking method associates radar obstacles among unassociated obstacles with camera obstacles among unassociated obstacles in the observed obstacles.

[0220] In one embodiment of this application, the first association module 65 or the second association module 69 is specifically used to acquire the first center point of the radar obstacle and the second center point of the camera obstacle; calculate the first angle between the projection line segment of the line connecting the first center point to the second center point on the vertical projection plane of the vehicle and the central axis of the camera, and the angle between the first center point and the second center point.

[0221] The second angle between the projection line segment of the line connecting the center points onto the vehicle's horizontal projection plane and the camera's central axis; when the first angle θ is less than half of the camera's horizontal field of view, the second angle is less than half of the camera's vertical field of view, and the first center point and...

[0222] When the Euclidean distance to the second center point is less than a set distance threshold, based on the predicted positions of unrelated obstacles in each historical fusion-tracked obstacle within a preset time period, the world coordinate system corresponding to the radar obstacle is converted to a pixel coordinate system; the vertices of the lidar 3D bounding box corresponding to the radar obstacle and the downsampled point cloud within the bounding box, or the millimeter-scale data corresponding to the radar obstacle, are then converted.

[0223] The vertices of the radar 3D bounding box are projected onto the pixel coordinate system, and the smallest envelope rectangle is taken as the projected 2D target bounding box. The width between the pixel deviation in the width direction between the center point of the projected 2D target bounding box and the center point of the camera 2D bounding box and the width of the camera 2D bounding box is obtained.

[0224] The proportions, and the height ratio between the pixel deviation in the height direction between the center point of the projected 2D target frame and the center point of the camera 2D frame, and the height of the camera 2D frame; determine the maximum value of the width and height proportions as l1; calculate the proportion l2 of the number of vertices in the projected 2D target frame corresponding to each radar obstacle to the total number of vertices of the radar obstacle; calculate the proportion l2 of each observation...

[0225] The distance l3 is calculated by taking the cosine of the heading angle of unrelated obstacles in the obstacle course and unrelated obstacles in each historical fusion-tracked obstacle course; the first ratio between the intersection area of ​​the projected 2D target box and the camera 2D box and the area of ​​the projected 2D target box is calculated.

[0226] The values, and the second ratio of the intersection area of ​​the projected 2D target frame and the camera 2D frame to the area of ​​the camera 2D frame; calculate the average value l4 between the first and second ratios; calculate the positional distance cost based on the relationship between l1, l2, l3, and l4; obtain the ratio s1 of the width of the projected 2D target frame to the width of the camera 2D frame; the ratio of the height of the projected 2D target frame to the area of ​​the camera 2D frame.

[0227] The ratio of the frame height s2; the ratio of the area of ​​the projected 2D target frame to the area of ​​the camera 2D frame s3; calculate the shape distance cost based on the relationship between s1, s2, and s3; obtain unrelated obstacles in each observed obstacle and each historical obstacle.

[0228] The ratio of the velocities of unrelated obstacles in the x-direction (m1) of the fused tracked obstacles; the ratio of the velocities of unrelated obstacles in the y-direction of each observed obstacle and each historical fused tracked obstacle; and the absolute ratio of the velocities of unrelated obstacles in the camera data of each observed obstacle and each historical fused tracked obstacle.

[0229] The ratio m3 of the velocity difference to the absolute velocity in the radar data is used; the distance cost of movement is calculated based on the relationship between m1, m2, and m3.

[0230] In one embodiment of this application, the first association module 65 or the second association module 69 is specifically used to obtain the IOU reward factor and the category reward factor; and to calculate the comprehensive distance cost between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle based on the relationship between the IOU reward factor, the category reward factor, the location distance cost, the shape distance cost and the motion distance cost.

[0231] 5. For specific limitations and beneficial effects of obstacle association devices in multi-sensor data, please refer to the section above.

[0232] The limitations of obstacle association methods in multi-sensor data will not be elaborated here. Each module in the obstacle association device described above can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0233] This invention also provides an electronic device having the above-described features. Figure 6 The obstacle association device shown is used in multi-sensor data analysis. For example... Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, memory 74, and at least one communication bus 72. The communication bus 72 is used to enable communication between these components. The communication interface 73 may include a display screen or a keyboard; optionally, the communication interface 73 may also include a standard wired interface or a wireless interface. The memory 74 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 74 may also be at least one storage device located remotely from the aforementioned processor 71. The processor 71 may be combined with... Figure 6 The described apparatus has an application program stored in memory 74, and the processor 71 calls the program code stored in memory 74 to perform any of the above method steps.

[0234] The communication bus 72 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 72 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0235] The memory 74 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 74 may also include a combination of the above types of memory.

[0236] The processor 71 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0237] The processor 71 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PlD), or a combination thereof. The PLD may be a complex programmable logic device (CPID), a field-programmable gate array (FPGA), a generic array logic (GAI), or any combination thereof.

[0238] Optionally, memory 74 is also used to store program instructions. Processor 71 can call the program instructions to implement the functions described in this application. Figures 1 to 5 The obstacle association method in multi-sensor data shown in the embodiment.

[0239] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the multi-sensor data target association method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0240] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for associating obstacles in multi-sensor data, characterized in that, include: Acquire current sensor data and retrieve historical data for obstacle tracking; The historical fusion tracking obstacle is used to characterize the obstacles after associating them with the obstacles in the historical sensor data of the previous moment; The predicted positions of each of the historically fused tracking obstacles within a preset time period are predicted using a preset prediction method; When the current sensor data is the current radar sensor data, the observed obstacles in the current radar sensor data are compared with the historical fusion tracking obstacles to determine the first obstacle in the historical fusion tracking obstacles that has the same obstacle identification information as the observed obstacles, and the first obstacle is associated with it. When the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each of the historical fusion tracking obstacles within a preset time period, the Euclidean distance cost matrix between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historical fusion tracking obstacles is calculated from the top-down perspective. Based on the Euclidean distance cost matrix, the Hungarian matching algorithm is used to perform association and update the association results. When the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unassociated obstacles in each of the historical fusion tracking obstacles within a preset time period, the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated obstacles in the historical fusion tracking obstacles are associated with the unassociated obstacles in the observed obstacles and the camera obstacles in the unassociated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated; Specifically, based on the predicted positions of unrelated obstacles in each of the historically fused tracked obstacles within a preset time period, the Euclidean distance cost matrix between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historically fused tracked obstacles is calculated from a top-down perspective, including: Based on the predicted positions of unrelated obstacles in each of the historical fusion tracking obstacles within a preset time period, obtain the corresponding LiDAR prediction data, millimeter-wave radar obstacle prediction data, camera prediction data, and V2X prediction data for each of the historical fusion tracking obstacles. From the lidar prediction data, the millimeter-wave radar obstacle prediction data, the camera prediction data, and the V2X prediction data, respectively, determine the corresponding lidar prediction position, the corresponding millimeter-wave radar prediction position, the corresponding camera prediction position, and the corresponding V2X prediction position of each unrelated obstacle in the historical fusion tracking obstacle at the most recent moment. Calculate the candidate distance cost between the current position of the unrelated obstacle in each of the observed obstacles and the predicted positions of the lidar, millimeter-wave radar, camera, and V2X in each of the historically fused tracked obstacles; Based on each candidate distance cost, calculate the Euclidean distance cost matrix between unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracked obstacle from the top-down view.

2. The method according to claim 1, characterized in that, After acquiring the current sensor data and acquiring historical fusion tracking of obstacles, the method further includes: The sensor data at the current moment includes the location information corresponding to each candidate obstacle; Based on the location information of each candidate obstacle, the candidate obstacles outside the target range are deleted to obtain the observed obstacles included in the sensor data at the current moment.

3. The method according to claim 1, characterized in that, The method further includes: When the current sensor data is not the current radar sensor data, and when the number of unrelated obstacles in the observed obstacles is greater than zero, and the number of unrelated obstacles in the historically fused tracked obstacles is also greater than zero, based on the predicted positions of the unrelated obstacles in each of the historically fused tracked obstacles within a preset time period, calculate the Euclidean distance cost matrix between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historically fused tracked obstacles from the top-down perspective. Based on the Euclidean distance cost matrix, use the Hungarian matching algorithm to perform association and update the association results. When the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historical fusion tracking obstacles is also greater than zero, based on the predicted positions of the unassociated obstacles in each of the historical fusion tracking obstacles within a preset time period, the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated obstacles in the historical fusion tracking obstacles are associated with the unassociated obstacles in the observed obstacles and the camera obstacles in the unassociated obstacles in the historical fusion tracking obstacles under the perspective projection view, and the association results are updated; When the current sensor data is the current camera sensor data, the identification information of the unassociated camera obstacles in the historical fusion tracking obstacles is compared with the identification information of the obstacles in the observed obstacles, and the second obstacle with the same identification information is associated.

4. The method according to claim 1, characterized in that, The step of calculating the Euclidean distance cost matrix between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historically fused tracked obstacles from the top-down view, based on each of the candidate distance costs, includes: For unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles, obtain the candidate distance cost between the current position of the unrelated obstacle in the observed obstacles and the predicted position of the lidar, the predicted position of the millimeter-wave radar, the predicted position of the camera, and the predicted position of the unrelated obstacle in the historical fusion tracking obstacles. The minimum distance is selected from the candidate distance costs and determined as the target distance cost between the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historical fusion tracking obstacles; Based on the target distance cost between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles, the Euclidean distance cost matrix between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles is determined from the top-down perspective.

5. The method according to any one of claims 1 or 3, characterized in that, The step of associating, from a perspective projection view, the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated obstacles in the historically fused tracked obstacles with the camera obstacles in the unassociated obstacles in the observed obstacles and the unassociated obstacles in the historically fused tracked obstacles, based on the predicted positions of the unassociated obstacles in each of the historically fused tracked obstacles within a preset time period, include: Based on the predicted positions of unrelated obstacles in each of the historically fused tracked obstacles within a preset time period, the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historically fused tracked obstacles are traversed, and the positional distance cost, shape distance cost, and motion distance cost between each unrelated obstacle in the observed obstacles and each unrelated obstacle in the historically fused tracked obstacles are calculated respectively; wherein, based on the predicted positions of unrelated obstacles in each of the historically fused tracked obstacles within a preset time period, the unrelated obstacles in the observed obstacles and the unrelated obstacles in the historically fused tracked obstacles are traversed, and the positional distance cost, shape distance cost, and motion distance cost between each unrelated obstacle in the observed obstacles and each unrelated obstacle in the historically fused tracked obstacles are calculated respectively. The historical fusion tracking of unrelated obstacles includes the following cost components: position distance cost, shape distance cost, and motion distance cost. This includes: acquiring the first center point of the radar obstacle and the second center point of the camera obstacle; calculating the first angle between the projection of the line segment connecting the first center point to the second center point onto the vehicle's vertical projection plane and the camera's central axis, and the second angle between the projection of the line segment connecting the first center point to the second center point onto the vehicle's horizontal projection plane and the camera's central axis; and determining the angle between the first angle and the second center point when the first angle is less than half the camera's horizontal field of view, the second angle is less than half the camera's vertical field of view, and the Euclidean distance between the first center point and the second center point is less than a set distance threshold. Based on the predicted positions of unrelated obstacles in the historical fusion tracking obstacles within a preset time period, the world coordinate system corresponding to the radar obstacle is converted into a pixel coordinate system; the vertices of the lidar 3D bounding box corresponding to the radar obstacle and the downsampled point cloud within the bounding box, or the vertices of the millimeter-wave radar 3D bounding box corresponding to the radar obstacle, are projected onto the pixel coordinate system, and the smallest envelope rectangle is taken as the projected 2D target bounding box; the width ratio between the pixel deviation in the width direction between the center point of the projected 2D target bounding box and the center point of the camera 2D bounding box and the width of the camera 2D bounding box, and the height ratio between the pixel deviation in the height direction between the center point of the projected 2D target bounding box and the center point of the camera 2D bounding box and the height of the camera 2D bounding box are obtained. The following calculations are performed: 1) Determine the maximum value of the width ratio and the height ratio as l1; 2) Calculate the ratio of the number of vertices in the projected 2D target frame corresponding to each radar obstacle to the total number of vertices in the radar obstacle; 3) Calculate the cosine distance l3 between the heading angles of unrelated obstacles in each observed obstacle and unrelated obstacles in each historical fusion tracking obstacle; 4) Calculate the first ratio l4 between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the projected 2D target frame, and the second ratio l4 between the intersection area of ​​the projected 2D target frame and the camera 2D frame and the area of ​​the camera 2D frame; 5) Calculate the average value l4 between the first ratio and the second ratio.Based on the relationship between l1, l2, l3, and l4, calculate the position distance cost; obtain the ratio s1 of the width of the projected 2D target frame to the width of the camera 2D frame; the ratio s2 of the height of the projected 2D target frame to the height of the camera 2D frame; the ratio s3 of the area of ​​the projected 2D target frame to the area of ​​the camera 2D frame; based on the relationship between s1, s2, and s3, calculate the shape distance cost; obtain the velocity ratio m1 of unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles in the x-direction, the velocity ratio m2 of unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles in the y-direction, and the ratio m3 of the absolute velocity difference of unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles in the camera data to the absolute velocity in the radar data; based on the relationship between m1, m2, and m3, calculate the motion distance cost; Based on the relationship between the location distance cost, the shape distance cost, and the motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles; Based on the magnitude of the comprehensive distance cost, under the perspective projection view, the unassociated obstacles in the observed obstacles and the radar obstacles in the unassociated obstacles in the historical fusion tracking obstacles are associated with the camera obstacles in the observed obstacles and the camera obstacles in the unassociated obstacles in the historical fusion tracking obstacles.

6. The method according to claim 5, characterized in that, The step of calculating the comprehensive distance cost between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historically fused tracked obstacles, based on the relationship between the location distance cost, the shape distance cost, and the motion distance cost, includes: Obtain the IOU reward factor and category reward factor; Based on the relationship between the IOU reward factor, the category reward factor, the location distance cost, the shape distance cost, and the motion distance cost, calculate the comprehensive distance cost between unrelated obstacles in each of the observed obstacles and unrelated obstacles in each of the historical fusion tracking obstacles.

7. A device for associating obstacles in multi-sensor data, characterized in that, include: The acquisition module is used to acquire sensor data at the current moment and to acquire historical fusion tracking obstacles; The historical fusion tracking obstacle is used to characterize the obstacles after associating them with the obstacles in the historical sensor data of the previous moment; The prediction module is used to predict the predicted position of each of the historical fusion-tracked obstacles within a preset time period using a preset prediction method; The determination module is used to, when the current sensor data is the current radar sensor data, compare the observed obstacles in the current radar sensor data with the historical fusion-tracked obstacles, determine the first obstacle in the historical fusion-tracked obstacles that has the same obstacle identification information as the observed obstacles, and associate the first obstacle; The first calculation module is used to calculate the Euclidean distance cost matrix between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historically fused tracking obstacles when the number of unrelated obstacles in the observed obstacles is greater than zero and the number of unrelated obstacles in the historically fused tracking obstacles is also greater than zero. Based on the predicted positions of the unrelated obstacles in each of the historically fused tracking obstacles within a preset time period, the module calculates the Euclidean distance cost matrix between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historically fused tracking obstacles from a top-down perspective. Based on the Euclidean distance cost matrix, the module uses a Hungarian matching algorithm to perform association and updates the association results. Specifically, calculating the Euclidean distance cost matrix between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historically fused tracking obstacles from a top-down perspective, based on the predicted positions of the unrelated obstacles in each of the historically fused tracking obstacles within a preset time period, includes: obtaining the predicted data of the lidar and the predicted data of the millimeter-wave radar corresponding to the unrelated obstacles in each of the historically fused tracking obstacles within a preset time period, based on the predicted positions of the unrelated obstacles in each of the historically fused tracking obstacles within a preset time period. Based on the LiDAR prediction data, the millimeter-wave radar obstacle prediction data, the camera prediction data, and the V2X prediction data, the corresponding LiDAR prediction positions, millimeter-wave radar prediction positions, camera prediction positions, and V2X prediction positions of the unrelated obstacles in each of the historically fused tracked obstacles at the most recent moment are determined from the LiDAR prediction data, the millimeter-wave radar obstacle prediction data, the camera prediction data, and the V2X prediction data. Candidate distance costs are calculated between the current position of the unrelated obstacle in each of the observed obstacles and the corresponding LiDAR prediction positions, millimeter-wave radar prediction positions, camera prediction positions, and V2X prediction positions of the unrelated obstacles in each of the historically fused tracked obstacles. Based on the candidate distance costs, a Euclidean distance cost matrix is ​​calculated between the unrelated obstacles in each of the observed obstacles and the unrelated obstacles in each of the historically fused tracked obstacles from the top-down perspective. The first association module is used to, when the number of unassociated obstacles in the observed obstacles is greater than zero, and the number of unassociated obstacles in the historically fused and tracked obstacles is also greater than zero, associate the unassociated obstacles in the observed obstacles and the radar obstacles among the unassociated obstacles in the historically fused and tracked obstacles with the camera obstacles among the unassociated obstacles in the observed obstacles and the camera obstacles among the unassociated obstacles in the historically fused and tracked obstacles in the predicted positions of the unassociated obstacles in each of the historically fused and tracked obstacles within a preset time period, and update the association results.

8. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores computer instructions, and the processor executes the computer instructions to perform the obstacle association method in multi-sensor data as described in any one of claims 1-6.

Images