Euclidean distance processing method based on multi-sensor observation data

Abstract

The embodiment of the present application relates to a kind of Euclidean distance processing method based on multi-sensor observation data, the method comprises: obtaining first observation data at any time t;And obtain the first historical motion trajectory and first historical observation queue of each known target at previous time t-1;According to first observation data, each first historical observation queue is carried out historical observation data selection and generates first historical observation data;Based on each first historical observation data and corresponding first historical motion trajectory, the observation data of each known target at current time t is estimated by Kalman filter, and first estimated observation data is obtained;The Euclidean distance of first observation data and each first estimated observation data is calculated to generate first Euclidean distance;The Euclidean distance set of current time t is formed by all the first Euclidean distance obtained. By the present application, it can be guaranteed that the problem of Euclidean distance set loss does not occur at any time.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for processing Euclidean distance based on multi-sensor observation data. Background Technology

[0002] When the perception module of an autonomous driving system performs target tracking based on Multi-Sensor Fusion (MSF) technology, one processing method is as follows: First, observation feature extraction is performed on the perception data (point cloud of LiDAR, image of camera, point cloud of millimeter-wave radar) from various sensors (LiDAR, camera, millimeter-wave radar). (The observation features of LiDAR point cloud include a polygonal convex hull and a target detection box; the observation features of camera image are a target detection box; and the observation features of millimeter-wave radar point cloud are an observation point). Then, spatiotemporal synchronization of various observation features is performed, using interpolation to synchronize all observation features to the same time t and using the same coordinate system to synchronize all observation features to the same space. Finally, the spatiotemporally synchronized observation features are fused to obtain... The system first generates a set of overall observation data. Then, based on the historical motion trajectories of multiple known targets and a Kalman filter, it estimates the observations of each target at time t to obtain multiple estimated observation data. Next, it calculates the Euclidean distance between the overall observation data and each estimated observation data to obtain multiple Euclidean distances forming a corresponding set. From the set, it selects the distances that meet the set requirements (e.g., the shortest distance requirement) as matching distances, and identifies the target corresponding to the matching distance as the associated target. Finally, based on the overall observation data, the target's motion state at the previous time t-1, and the corresponding Kalman filter, it estimates the target's motion state at the current time t and updates the target's motion trajectory based on the estimation results.

[0003] However, in practical applications, we found that the timing of various sensors is difficult to coordinate. The above scheme tends to produce significant delays when handling the spatiotemporal synchronization of multiple types of observation features, failing to meet the stringent delay control requirements of autonomous driving systems. Therefore, we adjusted the above scheme as follows: A historical observation queue is created for any target trajectory to store observation data from various sensors within a recent specified time period. A corresponding Kalman filter is created for any type of sensor, enabling it to predict the target's motion state based on the observation data of that type of sensor. Simultaneously, the motion models and state variable structures of the Kalman filters corresponding to various sensors are identical. Furthermore, the sensor data received at the current time t from sensor A is used to extract observation features according to sensor type, and the current observation features are uniformly converted to a bird's-eye view map coordinate system to complete spatial synchronization and generate the observation data M at the current time t. a Then, the most recent observation data from the same type of sensor is extracted from the historical observation queues of each known target as historical observation data M.h Then, based on the historical motion trajectories of each known target and the Kalman filter KF corresponding to the current sensor A. a and historical observation data M h The estimated observation data M' is obtained by estimating the observations of each known target by sensor A at the current time t. a Then, the observed data M a With each estimated observation data M' a The Euclidean distance is calculated to obtain a set of Euclidean distances. The distances that meet certain requirements (e.g., shortest distance) are selected from this set as matching distances, and the target corresponding to that matching distance is designated as the associated target. Then, based on the aforementioned observation data M... a The motion state of the target at the previous time t-1 and the Kalman filter KF corresponding to the current sensor A. a Estimate the target's motion state at the current time t, update the target's trajectory based on the estimation result, and then update the observation data M. a The data is stored in the historical observation queue corresponding to the target for future reference. This improved approach skips both the time synchronization and data fusion processes, significantly reducing target tracking processing time.

[0004] However, in practice, we found a flaw in the above-mentioned improved scheme: if there is no sensor observation data of the same type as the current sensor A in the historical observation queue, the Euclidean distance set for that instance cannot be obtained. In other words, the above-mentioned improved scheme may suffer from the problem of lost Euclidean distance set. This problem can easily lead to jumps in the target trajectory. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a method, electronic device, and computer-readable storage medium for processing Euclidean distances based on multi-sensor observation data. The method involves first acquiring sensor observation data at the current time t; then, when extracting historical observation data from the historical observation queues of various known targets, confirming whether observation data of the same type as the current sensor observation data exists in the queue. If so, this data is extracted as historical observation data by default; otherwise, observation data from other sensor types are extracted in a secondary order as historical observation data. Based on the historical motion trajectories of each known target, the aforementioned historical observation data, and their corresponding Kalman filters, the observations of each known target at the current time t based on the sensor type corresponding to the historical observation data are estimated to obtain the corresponding estimated observation data; finally, the Euclidean distances between the current sensor observation data and each estimated observation data are calculated to obtain a set of Euclidean distances. This invention provides auxiliary observation data from other sensors when there is no observation data of the same type as the current sensor observation data in the historical observation queue, ensuring that the Euclidean distance set is never lost at any time.

[0006] To achieve the above objectives, a first aspect of the present invention provides a method for processing Euclidean distance based on multi-sensor observation data, the method comprising:

[0007] At any time t, the sensor observation data is acquired and denoted as the corresponding first observation data M. t,c1 And obtain the first historical motion trajectory and the first historical observation queue of each known target at the previous time t-1; the first historical motion trajectory includes multiple first motion states S. i,j The first historical observation queue includes multiple first historical observation data M. i,j,c2 The first motion state S i,j Compared with the first historical observation data M i,j,c2 One-to-one correspondence; i is the target identifier, 1≤i; j is the time identifier, 0≤j≤t-1; c1 and c2 are both sensor type identifiers, which include lidar type identifiers, camera type identifiers and millimeter-wave radar type identifiers;

[0008] Based on the first observation data M t,c1 Historical observation data selection and processing are performed on each of the first historical observation queues to generate corresponding first historical observation data M. i,k,c2 k is the time identifier, 0≤k≤t-1;

[0009] Based on each of the first historical observation data M i,k,c2Using the corresponding first historical motion trajectory and Kalman filter, the observation data of each known target at the current time t are estimated to obtain the corresponding first estimated observation data M'. i,t,c2 ;

[0010] For the first observation data M t,c1 With each of the first estimated observation data M' i,t,c2 The corresponding first Euclidean distance d is generated by calculating the Euclidean distance. i,t ;

[0011] From all the first Euclidean distances d obtained i,t The set of Euclidean distances at the current time t.

[0012] Preferably, the step of basing the data on the first observation data M t,c1 Historical observation data selection and processing are performed on each of the first historical observation queues to generate corresponding first historical observation data M. i,k,c2 Specifically, it includes:

[0013] Step 21, for the first observation data M t,c1 The sensor type identifier c1 is used for identification; if the sensor type identifier c1 is a LiDAR type identifier, then the corresponding first, second, and third sequential identifiers are set to LiDAR type identifier, camera type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a camera type identifier, then the corresponding first, second, and third sequential identifiers are set to camera type identifier, LiDAR type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a millimeter-wave radar type identifier, then the corresponding first, second, and third sequential identifiers are set to millimeter-wave radar type identifier, LiDAR type identifier, and camera type identifier, respectively.

[0014] Step 22: Find the first historical observation data M in each of the first historical observation queues whose sensor type identifier c2 matches the first sequence identifier and whose time distance from the current time t is closest. i,j,c2 As the corresponding current matching data; if the current matching data is empty, then the first historical observation data M that is closest in time to the current time t to the sensor type identifier c2 and the second sequence identifier in the current first historical observation queue. i,j,c2 As the new current matching data; if the new current matching data is still empty, then the first historical observation data M that is closest in time to the current time t to the sensor type identifier c2 and the third sequence identifier in the current first historical observation queue. i,j,c2 As the new currently matched data;

[0015] Step 23: Use the obtained current matching data as the corresponding first historical observation data M. i,k,c2 The first historical observation data M i,k,c2 The target identifier i, time identifier k, and sensor type identifier c2 are matched with the first historical observation data M corresponding to the current matching data. i,j,c2 The target identifier i, time identifier j, and sensor type identifier c2 should be kept consistent.

[0016] Preferably, when the sensor type identifier c1 or c2 is a lidar type identifier, the corresponding observation data includes a polygonal convex hull and a target detection box, and the corresponding Kalman filter is a preset first Kalman filter;

[0017] When the sensor type identifier c1 or c2 is a camera type identifier, the corresponding observation data includes a target detection box, and the corresponding Kalman filter is a preset second Kalman filter;

[0018] When the sensor type identifier c1 or c2 is a millimeter-wave radar type identifier, the corresponding observation data includes one observation point, and the corresponding Kalman filter is the preset third Kalman filter.

[0019] Furthermore, the equations of motion and observation for the first Kalman filter are as follows:

[0020] X 1,t =A1X 1,t-1 +B 1,t-1 ,

[0021] Z 1,t =H1X 1,t +K 1,t ,

[0022] Among them, X 1,t-1 X 1,t Let Z be the state variables at times t-1 and t. 1,t Let X be the observation at time t, and A1 be X. 1,t-1 To X 1,t One-step prediction of the conversion amount, H1 is X 1,t With Z 1,t The state observation transition quantities between them; the state quantity X1 is the motion state S of the target's trajectory, and the observation quantity Z1 is composed of the corresponding polygonal convex hull and the target detection box; B 1,t-1 For state noise, k 1,t To observe noise;

[0023] The motion equations and observation equations of the second Kalman filter are as follows:

[0024] X 2,t =A2X 2,t-1 +B2,t-1 ,

[0025] Z 2,t =H2X 2,t +K 2,t ,

[0026] Among them, X 2,t-2 X 2,t Let Z be the state variables at times t-1 and t. 2,t Let X be the observation at time t, and A2 be X. 2,t-1 To X 2,t One-step prediction of the conversion amount, H2 is X 2,t With Z 2,t The state observation transition quantities between them; state quantity X2 is the motion state S of the target trajectory, and observation quantity Z2 is the corresponding target detection box; B 2,t-1 For state noise, k 2,t To observe noise;

[0027] The motion equations and observation equations of the third Kalman filter are as follows:

[0028] X 3,t =A3X 3,t-1 +B 3,t-1 ,

[0029] Z 3,t =H3X 3,t +K 3,t ,

[0030] Among them, X 3,t-1 X 3,t Let Z be the state variables at times t-1 and t. 3,t Let A3 be the observation at time t, and let X be the value of X. 3,t-1 To X 3,t One-step prediction of conversion amount, H3 is X 3,t With Z 3,t The state observation transition quantities between them; state quantity X3 is the motion state S of the target trajectory, and observation quantity Z3 is the corresponding observation point; B 3,t-1 For state noise, k 3,t To observe noise.

[0031] Preferably, the step based on each of the first historical observation data M i,k,c2 Using the corresponding first historical motion trajectory and Kalman filter, the observation data of each known target at the current time t are estimated to obtain the corresponding first estimated observation data M'. i,t,c2 Specifically, it includes:

[0032] The first historical observation data M i,k,c2The sensor type identifier c2 is recorded as the corresponding current sensor type identifier; and the first historical observation data M is recorded as... i,k,c2 In the first historical motion trajectory of the corresponding known target, all the first motion states S with time identifier j≥k-1 i,j The extracted data forms the corresponding second historical motion trajectory, which is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1};

[0033] When the current sensor type identifier is a lidar type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the first Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The third historical trajectory generated by merging the second historical trajectory is {S} i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; and the third historical trajectory and the first historical observation data M i,k,c2 Substituting into the first Kalman filter, using each of the first motion states S in the third historical motion trajectory i,j As a known state quantity from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 ;

[0034] When the current sensor type identifier is a camera type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the second Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fourth historical trajectory generated by merging the second historical trajectory is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; The fourth historical trajectory and the first historical observation data M i,k,c2 Substituting into the second Kalman filter, each of the first motion states S in the fourth historical motion trajectory... i,j As a known state quantity from time k-1 to time t, the first historical observation data Mi,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 ;

[0035] When the current sensor type identifier is a millimeter-wave radar type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the third Kalman filter, a one-step prediction is performed to obtain the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fifth historical trajectory, generated by merging the second historical trajectory, is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; The fifth historical trajectory and the first historical observation data M i,k,c2 Substituting into the third Kalman filter, each of the first motion states S in the fifth historical motion trajectory i,j As a known state quantity from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 ;

[0036] Wherein, the first estimated observation data M' i,t,c2 The target identifier i and sensor type identifier c2 are compared with the first historical observation data M. i,k,c2 The target identifier i and the sensor type identifier c2 should be kept consistent.

[0037] Preferably, the first observation data M t,c1 With each of the first estimated observation data M' i,t,c2 The corresponding first Euclidean distance d is generated by calculating the Euclidean distance. i,t Specifically, it includes:

[0038] The first observation data M t,c1 The sensor type identifier c1 is recorded as the first type identifier; and the current first estimated observation data M' is recorded as the first type identifier. i,t,c2 The sensor type identifier c2 is designated as the second type identifier;

[0039] When the first type identifier is a lidar type identifier, the first observation data M t,c1The corresponding polygon convex hull is denoted as the first convex hull; and the first convex hull is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0040] When the first type identifier is a camera type identifier, the first observation data M t,c1 The corresponding target detection box is denoted as the corresponding first target detection box; and the first target detection box is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0041] When the first type identifier is a millimeter-wave radar type identifier, the first observation data M t,c1 The corresponding observation point is denoted as the corresponding first observation point; and the first observation point is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0042] Wherein, the first Euclidean distance d i,t The target identifier i and the current first estimated observation data M' i,t,c2 The target identifier i remains consistent.

[0043] Furthermore, the first convex hull and the current first estimated observation data M' are compared according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes:

[0044] When the second type identifier is a lidar type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding polygonal convex hull is denoted as the corresponding second convex hull; and the center points of the first and second convex hulls are calculated to generate the corresponding first and second center points respectively; the straight-line distance between the first and second center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0045] When the second type identifier is a camera type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding target detection box is denoted as the corresponding second target detection box; and the shortest distance between the first convex hull and the second target detection box is calculated, and the calculation result is used as the corresponding first Euclidean distance d.i,t ;

[0046] When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding observation point is denoted as the corresponding second observation point; the shortest distance between the first convex hull and the second observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

[0047] Furthermore, the first target detection box is compared with the current first estimated observation data M' based on the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes:

[0048] When the second type identifier is a lidar type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding polygonal convex hull is denoted as the third convex hull; the shortest distance between the first target detection box and the third convex hull is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0049] When the second type identifier is a camera type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding target detection box is denoted as the corresponding third target detection box; the center points of the first and third target detection boxes are denoted as the corresponding third and fourth center points; the straight-line distance between the third and fourth center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0050] When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding observation point is denoted as the third observation point; the shortest distance between the first target detection box and the third observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

[0051] Furthermore, the first observation point and the current first estimated observation data M' are compared according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes:

[0052] When the second type identifier is a lidar type identifier, the current first estimated observation data M' will be used. i,t,c2The corresponding polygonal convex hull is denoted as the fourth convex hull; and the shortest distance between the first observation point and the fourth convex hull is calculated, with the calculation result used as the corresponding first Euclidean distance d. i,t ;

[0053] When the second type identifier is a camera type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding target detection box is denoted as the fourth target detection box; and the shortest distance between the first observation point and the fourth target detection box is calculated, with the calculation result used as the corresponding first Euclidean distance d. i,t ;

[0054] When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' will be used. i,t,c2 The corresponding observation point is denoted as the fourth observation point; the straight-line distance between the first and fourth observation points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

[0055] A second aspect of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0056] The processor is used to couple with the memory, read and execute instructions in the memory to implement the steps of the method described in the first aspect above;

[0057] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0058] A third aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a computer, cause the computer to perform the instructions described in the first aspect.

[0059] This invention provides a method, electronic device, and computer-readable storage medium for processing Euclidean distances based on multi-sensor observation data. First, sensor observation data at the current time t is acquired. When extracting historical observation data from the historical observation queues of each known target, it is confirmed whether observation data of the same type as the current sensor observation data exists in the queue. If so, this data is extracted as historical observation data by default; otherwise, observation data from other types of sensors are extracted as historical observation data in a secondary selection order. Based on the historical motion trajectory of each known target, the aforementioned historical observation data, and their corresponding Kalman filters, the observation of each known target at the current time t based on the sensor type corresponding to the historical observation data is estimated to obtain the corresponding estimated observation data. Then, the Euclidean distances between the current sensor observation data and each estimated observation data are calculated to obtain a set of Euclidean distances. This invention provides observation data from other sensors as an aid when there is no observation data of the same type as the current sensor observation data in the historical observation queue, ensuring that the Euclidean distance set is never lost at any time. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of a method for processing Euclidean distance based on multi-sensor observation data provided in Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0063] Embodiment 1 of the present invention provides a method for processing Euclidean distance based on multi-sensor observation data, such as... Figure 1 The schematic diagram shows a method for processing Euclidean distance based on multi-sensor observation data provided in Embodiment 1 of the present invention. This method mainly includes the following steps:

[0064] Step 1: Acquire sensor observation data at any time t and record it as the corresponding first observation data M. t,c1 And obtain the first historical motion trajectory and the first historical observation queue of each known target at the previous time t-1;

[0065] The first historical trajectory includes multiple first motion states S.i,j The first historical observation queue includes multiple first historical observation data M. i,j,c2 First state of motion S i,j Compared with the first historical observation data M i,j,c2 One-to-one correspondence; i is the target identifier, 1≤i; j is the time identifier, 0≤j≤t-1; c1 and c2 are both sensor type identifiers, including lidar type identifier, camera type identifier and millimeter-wave radar type identifier.

[0066] Here, as can be seen from the improved scheme mentioned in the background section, at any given time, the historical motion trajectories of each known target from the previous moment (i.e., the first historical motion trajectory) and the historical observation queue (i.e., the first historical observation queue) can be obtained; the first historical motion trajectory consists of multiple motion states (i.e., the first motion state S). i,j Composition; The first historical observation queue consists of sensor observation data obtained from multiple previous moments, i.e., the first historical observation data M. i,j,c2 Composition; each observation data corresponds to a sensor type, namely sensor type identifiers c1 and c2. This embodiment of the invention supports three sensor types: lidar, camera and millimeter-wave radar.

[0067] Furthermore, as mentioned in the background section, the observation features of a lidar point cloud include a polygonal convex hull and a target detection box; the observation features of a camera image are a target detection box; and the observation features of a millimeter-wave radar point cloud are an observation point. Additionally, as mentioned in the background section, the improved scheme creates a corresponding Kalman filter for any type of sensor, enabling it to predict the target's motion state based on the observation data of that type of sensor. Therefore, in this embodiment of the invention:

[0068] When sensor type identifier c1 or c2 is a LiDAR type identifier, the corresponding observation data includes a polygonal convex hull and a target detection box, and the corresponding Kalman filter is a preset first Kalman filter; when sensor type identifier c1 or c2 is a camera type identifier, the corresponding observation data includes a target detection box, and the corresponding Kalman filter is a preset second Kalman filter; when sensor type identifier c1 or c2 is a millimeter-wave radar type identifier, the corresponding observation data includes an observation point, and the corresponding Kalman filter is a preset third Kalman filter.

[0069] As can be seen from the improved scheme mentioned in the background section above, all observation data will be pre-converted to the bird's-eye view map coordinate system to complete spatial synchronization. Therefore, the polygon convex hull, target detection box and observation point included in the above observation data should all be polygon convex hull, target detection box and observation point under the same bird's-eye view map coordinate system.

[0070] Based on the content of the above observation data and the known Kalman filter equations (equations of motion and observation equations), it can be further determined that:

[0071] The equations of motion and observation for the first Kalman filter are:

[0072] X 1,t =A1X 1,t-1 +B 1,t-1 ,

[0073] Z 1,t =H1X 1,t +K 1,t ,

[0074] Among them, X 1,t-1 X 1,t Let Z be the state variables at times t-1 and t. 1,t Let X be the observation at time t, and A1 be X. 1,t-1 To X 1,t One-step prediction of the conversion amount, H1 is X 1,t With Z 1,t The state observation transition quantities between them; the state quantity X1 is the motion state S of the target's trajectory, and the observation quantity Z1 is composed of the corresponding polygonal convex hull and the target detection box; B 1,t-1 For state noise, k 1,t To observe noise;

[0075] The equations of motion and observation for the second Kalman filter are:

[0076] X 2,t =A2X 2,t-1 +B 2,t-1 ,

[0077] Z 2,t =H2X 2,t +K 2,t ,

[0078] Among them, X 2,t-2 X 2,t Let Z be the state variables at times t-1 and t. 2,t Let X be the observation at time t, and A2 be X. 2,t-1 To X 2,t One-step prediction of the conversion amount, H2 is X 2,t With Z 2,t The state observation transition quantities between them; state quantity X2 is the motion state S of the target trajectory, and observation quantity Z2 is the corresponding target detection box; B 2,t-1 For state noise, k 2,t To observe noise;

[0079] The equations of motion and observation for the third Kalman filter are:

[0080] X 3,t =A3X 3,t-1 +B 3,t-1 ,

[0081] Z 3,t =H3X 3,t +K 3,t ,

[0082] Among them, X 3,t-1 X 3,t Let Z be the state variables at times t-1 and t. 3,t Let A3 be the observation at time t, and let X be the value of X. 3,t-1 To X 3,t One-step prediction of conversion amount, H3 is X 3,t With Z 3,t The state observation transition quantities between them; state quantity X3 is the motion state S of the target trajectory, and observation quantity Z3 is the corresponding observation point; B 3,t-1 For state noise, k 3,t To observe noise.

[0083] Step 2, based on the first observation data M t,c1 Historical observation data is selected and processed for each first historical observation queue to generate corresponding first historical observation data M. i,k,c2 ;

[0084] Where k is the time identifier, 0≤k≤t-1;

[0085] Here, in this embodiment of the invention, when performing historical observation data selection processing on each first historical observation queue, it checks whether there exists a data in the queue that matches the current sensor observation data, i.e., the first observation data M. t,c1 Observational data of the same type (i.e., those with the same sensor type) are confirmed; if such data exists, it is extracted as the first historical observation data M by default. i,k,c2 If not, then observation data from other types of sensors are extracted in a secondary order as the first historical observation data M. i,k,c2 ;

[0086] Specifically, this includes: Step 21, processing the first observation data M t,c1The sensor type identifier c1 is used for identification; if the sensor type identifier c1 is a LiDAR type identifier, then the corresponding first, second, and third sequential identifiers are set to LiDAR type identifier, camera type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a camera type identifier, then the corresponding first, second, and third sequential identifiers are set to camera type identifier, LiDAR type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a millimeter-wave radar type identifier, then the corresponding first, second, and third sequential identifiers are set to millimeter-wave radar type identifier, LiDAR type identifier, and camera type identifier, respectively.

[0087] Here, the first, second, and third sequence identifiers reflect the hierarchical selection order of the observation data, with the first sequence identifier having the highest priority and the third sequence identifier having the lowest priority; for the first observation data M of the sensor type being lidar... t,c1 In this regard, the corresponding hierarchical optimization order is: lidar type, camera type, millimeter-wave radar type; for the first observation data M with camera type sensor... t,c1 Specifically, the corresponding hierarchical optimization order is camera type, LiDAR type, and millimeter-wave radar type; for the first observation data M with a sensor type of millimeter-wave radar... t,c1 In this regard, the corresponding order of preference is millimeter-wave radar type, lidar type, and camera type;

[0088] Step 22: Find the first historical observation data M in each first historical observation queue whose sensor type identifier c2 is closest to the first sequence identifier at the current time t. i,j,c2 This serves as the corresponding current matching data; if the current matching data is empty, then the first historical observation data M, which is closest in time to the current time t, is selected from the current first historical observation queue and matches the sensor type identifier c2 with the second sequence identifier. i,j,c2 This will be used as the new current matching data; if the new current matching data is still empty, then the first historical observation data M that is closest in time to the current time t to the sensor type identifier c2 and the third sequence identifier in the current first historical observation queue will be used. i,j,c2 As the new current matching data;

[0089] Here, when extracting historical observation data, we first search for observation data in the first historical observation queue that matches the first sequence identifier. If there is, we extract the most recent data as the output. If not, we search for observation data in the first historical observation queue that matches the second sequence identifier. If there is, we extract the most recent data as the output. If not, we extract the most recent observation data in the first historical observation queue that matches the third sequence identifier as the output.

[0090] Step 23: Use the obtained current matching data as the corresponding first historical observation data M. i,k,c2 ;

[0091] Among them, the first historical observation data M i,k,c2 The target identifier i, time identifier k, and sensor type identifier c2 are matched with the first historical observation data M corresponding to the current data. i,j,c2 The target identifier i, time identifier j, and sensor type identifier c2 should be kept consistent.

[0092] Here, because the historical observation queue will include at least one sensor type supported by this embodiment of the invention, the first historical observation data M will not appear. i,k,c2 The case where it is empty; that is, step 2 ensures that the embodiment of the present invention will not lose the observation data M' used for subsequent calculation at any time. i,t,c2 First historical observation data M i,k,c2 This naturally ensures that the data used for subsequent calculations of the first Euclidean distance d will not be lost at any time. i,t First historical observation data M i,k,c2 This naturally ensures that the distance d from multiple first Euclidean distances will not be lost at any time. i,t The set of Euclidean distances formed by these distances.

[0093] Step 3, based on each first historical observation data M i,k,c2 Using the corresponding first historical trajectory and Kalman filter, the observation data of each known target at the current time t are estimated to obtain the corresponding first estimated observation data M'. i,t,c2 ;

[0094] Among them, the first estimated observation data M' i,t,c2 Target identifier i and sensor type identifier c2 with first historical observation data M i,k,c2 The target identifier i and the sensor type identifier c2 must be consistent;

[0095] Specifically, this includes: Step 31, transferring the first historical observation data M i,k,c2 The sensor type identifier c2 is recorded as the corresponding current sensor type identifier; and the first historical observation data M is recorded as... i,k,c2 In the first historical trajectory of the corresponding known target, all first motion states S with time marker j≥k-1 i,j The extracted data forms the corresponding second historical motion trajectory, which is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1};

[0096] Step 32, when the current sensor type identifier is a lidar type identifier, the first motion state S of the second historical motion trajectory is recorded. i,j＝t-1 Substituting the equation of motion into the first Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The third historical trajectory generated by merging the second historical trajectory is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; and combine the third historical trajectory with the first historical observation data M i,k,c2 Substituting into the first Kalman filter, using each of the first motion states S in the third historical motion trajectory i,j As the known state variables from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 ;

[0097] Here, the recursive formula for Kalman filtering is a publicly available technical implementation, consisting of a one-step state prediction formula, a one-step prediction variance formula, a Kalman filter gain formula, a prediction residual formula, a state filtering formula, and a filtering variance formula group. Specific details can be found in the publicly available technical implementations, and will not be elaborated upon here. In this embodiment of the invention, the first motion state S can be obtained based on the one-step state prediction formula. i,j＝t Then, based on the one-step state prediction formula, the one-step prediction variance formula, and the Kalman filter gain formula, the observations from time k+1 to the current time t are derived step by step. The specific derivation calculations are all standard operation methods, which can be obtained by consulting the publicly available technical implementations. They will not be elaborated here.

[0098] Step 33: When the current sensor type identifier is a camera type identifier, the first motion state S of the second historical motion trajectory is recorded. i,j＝t-1 Substituting the equation of motion into the second Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fourth historical trajectory, generated by merging the second historical trajectory, is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; Combine the fourth historical trajectory and the first historical observation data M i,k,c2 Substituting into the second Kalman filter, using each of the first motion states S in the fourth historical motion trajectory i,jAs the known state variables from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 ;

[0099] Here, the implementation principle of the derivation based on the second Kalman filter is similar to that of the derivation based on the first Kalman filter in step 32, and will not be elaborated further here;

[0100] Step 34: When the current sensor type identifier is millimeter-wave radar type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the third Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fifth historical trajectory, generated by merging the second historical trajectory, is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t}; Combine the fifth historical trajectory with the first historical observation data M i,k,c2 Substituting into the third Kalman filter, using each of the first motion states S in the fifth historical motion trajectory... i,j As the known state variables from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M'. i,t,c2 .

[0101] Here, the implementation principle of the derivation based on the third Kalman filter is similar to that of the derivation based on the first Kalman filter in step 32, and will not be elaborated further here.

[0102] In summary, step 3 allows us to estimate the first estimated observation data M' of any type of sensor (LiDAR, camera, millimeter-wave radar) on any known target at time t. i,t,c2 In other words:

[0103] If the first historical observation data M i,k,c2If sensor type identifier c2 is the lidar type identifier, then through the current step 3, the observation data of lidar on any known target at time t can be estimated. As mentioned above, the observation data of lidar is a polygonal convex hull and a target detection box. In other words, through the current step 3, a polygonal convex hull and a target detection box on any known target at time t in the same bird's-eye view map coordinate system can be estimated.

[0104] If the first historical observation data M i,k,c2 If sensor type identifier c2 is camera type identifier, then through the current step 3, the observation data of the camera on any known target at time t can be estimated. As mentioned above, the observation data of the corresponding camera is a target detection box. In other words, through the current step 3, a target detection box of any known target at time t in the same bird's-eye view map coordinate system can be estimated.

[0105] If the first historical observation data M i,k,c2 If sensor type identifier c2 is a millimeter-wave radar type identifier, then through the current step 3, the observation data of the millimeter-wave radar on any known target at time t can be estimated. As mentioned above, the observation data of the millimeter-wave radar is a single observation point. In other words, through the current step 3, an observation point of any known target at time t in the same bird's-eye view map coordinate system can be estimated.

[0106] Step 4, process the first observation data M t,c1 With each of the first estimated observation data M' i,t,c2 The corresponding first Euclidean distance d is generated by calculating the Euclidean distance. i,t ;

[0107] Wherein, the first Euclidean distance d i,t Target identifier i and current first estimated observation data M' i,t,c2 The target identifier i remains consistent;

[0108] Specifically, this includes: Step 41, transferring the first observation data M t,c1 The sensor type identifier c1 is designated as the first type identifier; and the current first estimated observation data M' is used as the first type identifier. i,t,c2 The sensor type identifier c2 is designated as the second type identifier;

[0109] Step 42: When the first type identifier is a lidar type identifier, the first observation data M is... t,c1 The corresponding polygon convex hull is denoted as the first convex hull; and the first convex hull is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0110] Here, the first observation data M t,c1 This actually corresponds to an unassociated target at the current time t. If the first type identifier is a lidar type identifier, it means the first observation data M t,c1 The current step uses the first observation data M, which is the observation data from the lidar. t,c1 The corresponding polygon convex hull, i.e., the first convex hull, is the location of the unassociated target, based on the first estimated observation data M' of each known target obtained in step 3. i,t,c2 The corresponding convex hull, target detection box, or observation point is the estimated position of the known target. The minimum distance between the two is calculated to obtain the corresponding Euclidean distance. As can be seen from the known target association technology, this Euclidean distance actually represents the degree of association or similarity between the unassociated target at the current time t and each known target at the previous time t-1.

[0111] Specifically, this includes: Step 421, transferring the first observation data M t,c1 The corresponding polygonal convex hull is denoted as the first convex hull.

[0112] Step 422: Based on the second type identifier, compare the first convex hull with the current first estimated observation data M'. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0113] Specifically, this includes: step 4221, when the second type identifier is a lidar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding polygonal convex hull is denoted as the corresponding second convex hull; the center points of the first and second convex hulls are calculated to generate the corresponding first and second center points respectively; the straight-line distance between the first and second center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0114] Here, if both objects for which the minimum distance is calculated are polygonal convex hulls, then the center points of the two convex hulls, i.e., the first and second center points, are used as reference points for the shortest distance calculation. The center of the convex hull can be obtained by averaging the x-coordinates and y-coordinates of all vertices of the convex hull. Since the straight-line distance between two points is the shortest, the straight-line distance between the first and second center points is taken as the first Euclidean distance d. i,t ;

[0115] Step 4222: When the second type identifier is a camera type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding target detection box is denoted as the corresponding second target detection box; the shortest distance between the first convex hull and the second target detection box is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t;

[0116] Here, if the two objects for which the minimum distance is calculated are a convex hull and a bounding box, then the shortest distance between the two polygons is taken as the first Euclidean distance d. i,t There are multiple algorithms for calculating the shortest distance between two polygons. This embodiment of the invention supports the calculation method based on the rotating caliper algorithm and the calculation method based on the GLK (Gilbert–Johnson–Keerthi) collision detection algorithm. In addition, other algorithms can be added according to specific implementation requirements. The specific algorithm implementations of the rotating caliper algorithm and the GLK collision detection algorithm can be found in the publicly available algorithm literature, and will not be elaborated further here.

[0117] Step 4223: When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding observation point is denoted as the corresponding second observation point; the shortest distance between the first convex hull and the second observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0118] Here, if the two objects for calculating the minimum distance are a convex hull and an observation point, then the shortest distance from the point to the polygon is taken as the first Euclidean distance d. i,t There are several algorithms for calculating the shortest distance from a point to a polygon. The simplest one is to use a point-by-point traversal method to find the distance from the observation point to each vertex of the convex hull when the number of vertices of the convex hull is limited, and then select the minimum value as the shortest distance between them. If the number of vertices of the convex hull is large, the GLK collision detection algorithm can be used to calculate the shortest distance between them, thereby reducing the overall computational load. Of course, other algorithms can be added according to the specific implementation requirements, which will not be listed here.

[0119] Step 43: When the first type identifier is a camera type identifier, the first observation data M is... t,c1 The corresponding target detection box is denoted as the corresponding first target detection box; and the first target detection box is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0120] Specifically, this includes: Step 431, transferring the first observation data M t,c1 The corresponding target detection box is denoted as the corresponding first target detection box;

[0121] Step 432: Match the first target detection box with the current first estimated observation data M' based on the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0122] Specifically, this includes: Step 4321, when the second type identifier is a lidar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding polygonal convex hull is denoted as the third convex hull; the shortest distance between the first target detection box and the third convex hull is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0123] Here, the calculation method of the current step is similar to that of step 4222 mentioned above, and will not be described again here;

[0124] Step 4322: When the second type identifier is a camera type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding target detection box is denoted as the third target detection box; the center points of the first and third target detection boxes are denoted as the third and fourth center points; the straight-line distance between the third and fourth center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0125] Here, if both objects for which the minimum distance is calculated are bounding boxes, the center points of the two bounding boxes, namely the third and fourth center points, are used as reference points for the shortest distance calculation. As is known from the characteristics of bounding boxes, the center point of the bounding box is an essential parameter and can be obtained without calculation. Since the straight-line distance between two points is known to be the shortest distance, the straight-line distance between the third and fourth center points is directly taken as the first Euclidean distance d. i,t ;

[0126] Step 4323: When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding observation point is denoted as the third observation point; the shortest distance between the first target detection box and the third observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0127] Here, if the two objects for calculating the minimum distance are a bounding box and an observation point, then the bounding box is considered as a polygon, and the shortest distance from the point to the polygon is taken as the first Euclidean distance d. i,t The calculation method for the shortest distance between a point and a polygon is similar to step 4223 above, and will not be described again here.

[0128] Step 44: When the first type identifier is a millimeter-wave radar type identifier, the first observation data M is... t,c1The corresponding observation point is denoted as the first observation point; and the first observation point is compared with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0129] Specifically, this includes: Step 441, transferring the first observation data M t,c1 The corresponding observation point is denoted as the first observation point.

[0130] Step 442: Match the first observation point with the current first estimated observation data M' according to the second type identifier. i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ;

[0131] Specifically, this includes: Step 4421, when the second type identifier is a lidar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding polygonal convex hull is denoted as the fourth convex hull; the shortest distance between the first observation point and the fourth convex hull is calculated, and the calculation result is taken as the corresponding first Euclidean distance d. i,t ;

[0132] Here, the calculation method of the current step is similar to that of step 4223 mentioned above, and will not be described again here;

[0133] Step 4422: When the second type identifier is a camera type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding target detection box is denoted as the fourth target detection box; the shortest distance between the first observation point and the fourth target detection box is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ;

[0134] Here, the calculation method of the current step is similar to that of step 4223 mentioned above, and will not be described again here;

[0135] Step 4423: When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M' is... i,t,c2 The corresponding observation point is denoted as the fourth observation point; the straight-line distance between the first and fourth observation points is calculated, and the result is used as the corresponding first Euclidean distance d. i,t .

[0136] Here, if both objects used to calculate the minimum distance are observation points, and the straight-line distance between the two points is known to be the shortest distance, then the straight-line distance between the first and fourth observation points is directly taken as the first Euclidean distance d. i,t .

[0137] Step 5, from all the obtained first Euclidean distances d i,t The set of Euclidean distances at the current time t.

[0138] In summary, this embodiment of the invention, in step 2 above, when extracting historical observation data from the historical observation queues of each known target, confirms whether there is observation data in the queue that is consistent with the current sensor observation data type. If so, this data is extracted by default as historical observation data; otherwise, observation data from other types of sensors are extracted as historical observation data in a secondary selection order. Then, in step 3 above, based on the historical motion trajectory of each known target, the aforementioned historical observation data, and their corresponding Kalman filters, the observation of each known target at the current time t based on the sensor type corresponding to the historical observation data is estimated to obtain the corresponding estimated observation data. Then, in step 4 above, the Euclidean distance between the current sensor observation data and each estimated observation data is calculated. Finally, in step 5 above, the set of Euclidean distances between the unassociated target corresponding to the observation data at the current time t and all known targets at the previous time t-1 is obtained. Through steps 1-5 above, it can be ensured that the Euclidean distance set will not be lost at any time.

[0139] In this embodiment of the invention, after step 5, target association processing is performed based on the obtained Euclidean distance set to obtain the observation data at the current time t, i.e., the first observation data M. t,c1 The corresponding unassociated target is associated with a known target from the previous time step, and then the first observation data M is used. t,c1 The data is stored in the first historical observation queue corresponding to the target as reference data for the next moment, and based on the first observation data M of the target... t,c1 The first historical trajectory and the corresponding sensor Kalman filter (first, second, or third Kalman filter) estimate the target's motion state at the current time t, and merge the latest estimation result into the first historical trajectory to complete the trajectory update; it should be noted that if the first observation data M is not provided during the above target association process, t,c1 If an unassociated target is associated with a known target from the previous time step, then the unassociated target is considered a new target, and a corresponding first historical trajectory and first historical observation queue are assigned to it. The first historical trajectory is initialized using a default initial motion state, and the first observation data M is used. t,c1 The first historical observation queue is initialized and used as reference data for the next moment.

[0140] Figure 2This is a schematic diagram of an electronic device provided in Embodiment 2 of the present invention. This electronic device can be the aforementioned terminal device or server, or it can be a terminal device or server connected to the aforementioned terminal device or server that implements the method of the embodiments of the present invention. Figure 2 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing method embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0141] exist Figure 2 The system bus 305 mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The symbol is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0142] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0143] It should be noted that the embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when run on a computer, cause the computer to perform the methods and processes provided in the above embodiments.

[0144] This invention also provides a chip for executing instructions, which is used to perform the processing steps described in the foregoing method embodiments.

[0145] This invention provides a method, electronic device, and computer-readable storage medium for processing Euclidean distances based on multi-sensor observation data. First, sensor observation data at the current time t is acquired. When extracting historical observation data from the historical observation queues of each known target, it is confirmed whether observation data of the same type as the current sensor observation data exists in the queue. If so, this data is extracted as historical observation data by default; otherwise, observation data from other types of sensors are extracted as historical observation data in a secondary selection order. Based on the historical motion trajectory of each known target, the aforementioned historical observation data, and their corresponding Kalman filters, the observation of each known target at the current time t based on the sensor type corresponding to the historical observation data is estimated to obtain the corresponding estimated observation data. Then, the Euclidean distances between the current sensor observation data and each estimated observation data are calculated to obtain a set of Euclidean distances. This invention provides observation data from other sensors as an aid when there is no observation data of the same type as the current sensor observation data in the historical observation queue, ensuring that the Euclidean distance set is never lost at any time.

[0146] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0147] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0148] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing Euclidean distance based on multi-sensor observation data, characterized in that, The method includes: At any time t, the sensor observation data is acquired and denoted as the corresponding first observation data M. t,c1 And obtain the first historical motion trajectory and the first historical observation queue of each known target at the previous time t-1; the first historical motion trajectory includes multiple first motion states S. i,j The first historical observation queue includes multiple first historical observation data M. i,j,c2 The first motion state S i,j Compared with the first historical observation data M i,j,c2 One-to-one correspondence; i is the target identifier, 1≤i; j is the time identifier, 0≤j≤t-1; c1 and c2 are both sensor type identifiers, which include lidar type identifiers, camera type identifiers and millimeter-wave radar type identifiers; Based on the first observation data M t,c1 Historical observation data selection and processing are performed on each of the first historical observation queues to generate corresponding first historical observation data M. i,k,c2 k is the time identifier, 0≤k≤t-1; Based on each of the first historical observation data M i,k,c2 Using the corresponding first historical motion trajectory and Kalman filter, the observation data of each known target at the current time t are estimated to obtain the corresponding first estimated observation data M. ’ i,t,c2 ; For the first observation data M t,c1 With each of the first estimated observation data M ’ i,t,c2 The corresponding first Euclidean distance d is generated by calculating the Euclidean distance. i,t ; From all the first Euclidean distances d obtained i,t The set of Euclidean distances at the current time t; Wherein, according to the first observation data M t,c1 Historical observation data selection and processing are performed on each of the first historical observation queues to generate corresponding first historical observation data M. i,k,c2 Specifically, it includes: Step 21, for the first observation data M t,c1 The sensor type identifier c1 is used for identification; if the sensor type identifier c1 is a LiDAR type identifier, then the corresponding first, second, and third sequential identifiers are set to LiDAR type identifier, camera type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a camera type identifier, then the corresponding first, second, and third sequential identifiers are set to camera type identifier, LiDAR type identifier, and millimeter-wave radar type identifier, respectively; if the sensor type identifier c1 is a millimeter-wave radar type identifier, then the corresponding first, second, and third sequential identifiers are set to millimeter-wave radar type identifier, LiDAR type identifier, and camera type identifier, respectively. Step 22: Find the first historical observation data M in each of the first historical observation queues whose sensor type identifier c2 is closest to the first sequence identifier at the current time t. i,j,c2 As the corresponding current matching data; if the current matching data is empty, then the first historical observation data M that is closest in time to the current time t to the sensor type identifier c2 and the second sequence identifier in the current first historical observation queue. i,j,c2 As the new current matching data; if the new current matching data is still empty, then the first historical observation data M that is closest in time to the current time t to the sensor type identifier c2 and the third sequence identifier in the current first historical observation queue. i,j,c2 As the new currently matched data; Step 23: Use the obtained current matching data as the corresponding first historical observation data M. i,k,c2 The first historical observation data M i,k,c2 The target identifier i, time identifier k, and sensor type identifier c2 are matched with the first historical observation data M corresponding to the current matching data. i,j,c2 The target identifier i, time identifier j, and sensor type identifier c2 are kept consistent; When sensor type identifier c1 or c2 is a LiDAR type identifier, the corresponding observation data includes a polygonal convex hull and a target detection box, and the corresponding Kalman filter is a preset first Kalman filter; when sensor type identifier c1 or c2 is a camera type identifier, the corresponding observation data includes a target detection box, and the corresponding Kalman filter is a preset second Kalman filter; when sensor type identifier c1 or c2 is a millimeter-wave radar type identifier, the corresponding observation data includes an observation point, and the corresponding Kalman filter is a preset third Kalman filter. The first historical observation data M i,k,c2 Using the corresponding first historical motion trajectory and Kalman filter, the observation data of each known target at the current time t are estimated to obtain the corresponding first estimated observation data M. ’ i,t,c2 Specifically, it includes: The first historical observation data M i,k,c2 The sensor type identifier c2 is recorded as the corresponding current sensor type identifier; and the first historical observation data M is recorded as... i,k,c2 In the first historical motion trajectory of the corresponding known target, all the first motion states S with time identifier j≥k-1 i,j The extracted data forms the corresponding second historical motion trajectory, which is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 }; When the current sensor type identifier is a lidar type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the first Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The third historical trajectory generated by merging the second historical trajectory is {S} i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t }; and the third historical trajectory and the first historical observation data M i,k,c2 Substituting into the first Kalman filter, using each of the first motion states S in the third historical motion trajectory i,j As a known state quantity from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M. ’ i,t,c2 ; When the current sensor type identifier is a camera type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the second Kalman filter and performing a one-step prediction yields the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fourth historical trajectory generated by merging the second historical trajectory is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t }; The fourth historical trajectory and the first historical observation data M i,k,c2 Substituting into the second Kalman filter, each of the first motion states S in the fourth historical motion trajectory... i,j As a known state quantity from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M. ’ i, t,c2; When the current sensor type identifier is a millimeter-wave radar type identifier, the first motion state S of the second historical motion trajectory is... i,j＝t-1 Substituting the equation of motion into the third Kalman filter, a one-step prediction is performed to obtain the corresponding first motion state S. i,j＝t ; and the first motion state S i,j＝t The fifth historical trajectory, generated by merging the second historical trajectory, is {S}. i,j＝k-1 ,S i,j＝k …S i,j＝t-1 ,S i,j＝t }; The fifth historical trajectory and the first historical observation data M i,k,c2 Substituting into the third Kalman filter, each of the first motion states S in the fifth historical motion trajectory i,j As a known state quantity from time k-1 to time t, the first historical observation data M i,k,c2 As known observations at time k, the observations from time k+1 to the current time t are derived step by step; and the derived observations at the last time t are used as the corresponding first estimated observation data M. ’ i,t,c2 ; Wherein, the first estimated observation data M ’ i,t,c2 The target identifier i and sensor type identifier c2 are compared with the first historical observation data M. i,k,c2 The target identifier i and the sensor type identifier c2 should be kept consistent.

2. The Euclidean distance processing method based on multi-sensor observation data according to claim 1, characterized in that, The motion equations and observation equations of the first Kalman filter are as follows: X 1,t =A1X 1,t-1 +B 1,t-1 , FROM 1,t =H1X 1,t +K 1,t , Among them, X 1,t-1 X 1,t Let Z be the state variables at times t-1 and t. 1,t Let X be the observation at time t, and A1 be X. 1,t-1 To X 1,t One-step prediction of the conversion amount, H1 is X 1,t With Z 1,t The state observation transition quantities between them; the state quantity X1 is the motion state S of the target's trajectory, and the observation quantity Z1 is composed of the corresponding polygonal convex hull and the target detection box; B 1,t-1 For state noise, k 1,t To observe noise; The motion equations and observation equations of the second Kalman filter are as follows: X 2,t < A2X 2,t-1 +B 2,t-1 , With 2,t =H2X 2,t +K 2,t , Among them, X 2,t-2 X 2,t Let Z be the state variables at times t-1 and t. 2,t Let X be the observation at time t, and A2 be X. 2,t-1 To X 2,t One-step prediction of the conversion amount, H2 is X 2,t With Z 2,t The state observation transition quantities between them; state quantity X2 is the motion state S of the target trajectory, and observation quantity Z2 is the corresponding target detection box; B 2,t-1 For state noise, k 2,t To observe noise; The motion equations and observation equations of the third Kalman filter are as follows: X 3,t =A3X 3,t-1 +B 3,t-1 , WITH 3,t =H3X 3,t +K 3,t , Among them, X 3,t-1 X 3,t Let Z be the state variables at times t-1 and t. 3,t Let A3 be the observation at time t, and let X be the value of X. 3,t-1 To X 3,t One-step prediction of conversion amount, H3 is X 3,t With Z 3,t The state observation transition quantities between them; state quantity X3 is the motion state S of the target trajectory, and observation quantity Z3 is the corresponding observation point; B 3,t-1 For state noise, k 3,t To observe noise.

3. The Euclidean distance processing method based on multi-sensor observation data according to claim 1, characterized in that, The first observation data M t,c1 With each of the first estimated observation data M ’ i,t,c2 The corresponding first Euclidean distance d is generated by calculating the Euclidean distance. i,t Specifically, it includes: The first observation data M t,c1 The sensor type identifier c1 is recorded as the first type identifier; and the current first estimated observation data M is recorded as the first type identifier. ’ i,t,c2 The sensor type identifier c2 is designated as the second type identifier; When the first type identifier is a lidar type identifier, the first observation data M t,c1 The corresponding polygon convex hull is denoted as the first convex hull; and the first convex hull is compared with the current first estimated observation data M according to the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ; When the first type identifier is a camera type identifier, the first observation data M t,c1 The corresponding target detection box is denoted as the corresponding first target detection box; and the first target detection box is compared with the current first estimated observation data M according to the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ; When the first type identifier is a millimeter-wave radar type identifier, the first observation data M t,c1 The corresponding observation point is denoted as the corresponding first observation point; and the first observation point is compared with the current first estimated observation data M according to the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t ; Wherein, the first Euclidean distance d i,t The target identifier i and the current first estimated observation data M ’ i, The target identifier i of t and c2 remains consistent.

4. The Euclidean distance processing method based on multi-sensor observation data according to claim 3, characterized in that, The first convex hull and the current first estimated observation data M are compared according to the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes: When the second type identifier is a lidar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding polygonal convex hull is denoted as the corresponding second convex hull; and the center points of the first and second convex hulls are calculated to generate the corresponding first and second center points respectively; the straight-line distance between the first and second center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a camera type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding target detection box is denoted as the corresponding second target detection box; and the shortest distance between the first convex hull and the second target detection box is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding observation point is denoted as the corresponding second observation point; the shortest distance between the first convex hull and the second observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

5. The Euclidean distance processing method based on multi-sensor observation data according to claim 3, characterized in that, The first target detection box is compared with the current first estimated observation data M based on the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes: When the second type identifier is a lidar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding polygonal convex hull is denoted as the third convex hull; the shortest distance between the first target detection box and the third convex hull is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a camera type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding target detection box is denoted as the corresponding third target detection box; the center points of the first and third target detection boxes are denoted as the corresponding third and fourth center points; the straight-line distance between the third and fourth center points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding observation point is denoted as the third observation point; the shortest distance between the first target detection box and the third observation point is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

6. The Euclidean distance processing method based on multi-sensor observation data according to claim 3, characterized in that, The first observation point and the current first estimated observation data M are compared according to the second type identifier. ’ i,t,c2 The Euclidean distance is estimated to generate the corresponding first Euclidean distance d. i,t Specifically, it includes: When the second type identifier is a lidar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding polygonal convex hull is denoted as the fourth convex hull; and the shortest distance between the first observation point and the fourth convex hull is calculated, with the calculation result used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a camera type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding target detection box is denoted as the fourth target detection box; and the shortest distance between the first observation point and the fourth target detection box is calculated, with the calculation result used as the corresponding first Euclidean distance d. i,t ; When the second type identifier is a millimeter-wave radar type identifier, the current first estimated observation data M will be used. ’ i,t,c2 The corresponding observation point is denoted as the fourth observation point; the straight-line distance between the first and fourth observation points is calculated, and the calculation result is used as the corresponding first Euclidean distance d. i,t .

7. An electronic device, characterized in that, include: Memory, processor, and transceiver; The processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method according to any one of claims 1-6; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1-6.

Images