A method, apparatus, vehicle, and medium for vehicle sensor data association

Abstract

The application relates to a vehicle sensor data association method and device, a vehicle and a medium. The method comprises the following steps: acquiring first sensor data collected by a first sensor, second sensor data collected by a second sensor and reference sensor data collected by a reference sensor; performing time alignment on the first sensor data and the second sensor data respectively with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set; performing matching on the sensor data in the first initial data frame pair set and the second initial data frame pair set respectively based on a preset target algorithm to obtain a first associated frame pair set and a second associated frame pair set; and determining a target associated frame pair set composed of the first sensor data and the second sensor data based on the reference sensor data in the first associated frame pair set and the second associated frame pair set. According to the embodiment of the application, the historical information from a target time to an association time is associated, and the association is robust and accurate.

Description

Technical Field

[0001] This invention relates to the field of vehicle sensor technology, and more specifically to a method, apparatus, vehicle, and medium for vehicle sensor data association. Background Technology

[0002] In the field of intelligent driving, there are two main perception solutions: single-sensor solutions and multi-sensor fusion solutions. Multi-sensor fusion solutions can be broadly divided into two levels: First, fusing raw information (point clouds or images) from different sensor modalities at the front end to obtain combined information before target detection and tracking, ultimately yielding an equivalent combined sensor track output, also known as pre-fusion; second, fusing the track results output independently from different sensors at the target level, also ultimately yielding an equivalent combined sensor track output, also known as post-fusion. In post-fusion, there are two key steps: first, associating and matching multiple targets output from different sensors to obtain a track group representing the same real target in different sensors, called the association step; second, fusing the information from the track group based on filtering or optimization methods to obtain an equivalent combined sensor track output, called the fusion step.

[0003] In post-fusion, the association step, which is at the forefront, directly affects the performance and robustness of the entire post-fusion scheme. Currently, the dominant methods in the field of track association are Nearest Neighbor Data Association (NNDA) and Probabilistic Data Association (PDA), which are based on statistical distance or statistical probability judgments. Both of them associate the states of the track between two consecutive frames, which satisfies the Markov process assumption. However, they do not incorporate historical frame information into the association judgment, which has its limitations. Summary of the Invention

[0004] In view of the above problems, a method, apparatus, vehicle, and medium for vehicle sensor data association are proposed to overcome or at least partially solve the above problems.

[0005] This invention provides a method for associating vehicle sensor data, wherein the vehicle is equipped with a first sensor, a second sensor, and a reference sensor. The method includes:

[0006] Acquire the first sensor data collected by the first sensor, the second sensor data collected by the second sensor, and the reference sensor data collected by the reference sensor;

[0007] The first sensor data and the second sensor data are time-aligned with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set.

[0008] Based on a preset target algorithm, the sensor data in the first initial data frame pair set and the second initial data frame pair set are matched respectively to obtain the first associated frame pair set and the second associated frame pair set.

[0009] Based on the reference sensor data in the first and second associated frame pairs, a target associated frame pair set composed of the first and second sensor data is determined.

[0010] The present invention also provides a device for associating vehicle sensor data, wherein the vehicle is equipped with a first sensor, a second sensor, and a reference sensor. The device includes:

[0011] The data acquisition module is used to acquire the first sensor data acquired by the first sensor, the second sensor data acquired by the second sensor, and the reference sensor data acquired by the reference sensor.

[0012] The time alignment module is used to time-align the first sensor data and the second sensor data with the reference sensor data respectively to obtain a first initial data frame pair set and a second initial data frame pair set.

[0013] The algorithm matching module is used to match the sensor data in the first initial data frame pair set and the second initial data frame pair set based on a preset target algorithm, respectively, to obtain the first associated frame pair set and the second associated frame pair set.

[0014] The data association module is used to determine a target associated frame pair set composed of the first sensor data and the second sensor data based on the reference sensor data in the first associated frame pair set and the second associated frame pair set.

[0015] The present invention also provides a vehicle, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the vehicle sensor data association method described above.

[0016] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the above-described method for associating vehicle sensor data.

[0017] The beneficial effects of this invention are:

[0018] In this invention, first sensor data collected by a first sensor, second sensor data collected by a second sensor, and reference sensor data collected by a reference sensor can be acquired. The first sensor data and second sensor data are time-aligned with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set. Then, based on a preset target algorithm, the sensor data in the first initial data frame pair set and the second initial data frame pair set are matched to obtain a first associated frame pair set and a second associated frame pair set. Based on the reference sensor data in the first associated frame pair set and the second associated frame pair set, a target associated frame pair set composed of the first sensor data and the second sensor data is determined. This realizes the association of historical information from the target time to the associated time. When applied to the association of multi-sensor data, it has good association robustness and accuracy.

[0019] Furthermore, in this invention, the matching and association efficiency is further improved by using the target algorithm twice during the association process.

[0020] Furthermore, in this invention, the associated frames can be used as the training set for the association model to train the model, thereby obtaining an association model with high association accuracy. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the steps of a method for associating vehicle sensor data according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating the steps of another method for associating vehicle sensor data according to an embodiment of the present invention;

[0023] Figure 3a This is a flowchart illustrating the steps of another method for associating vehicle sensor data according to an embodiment of the present invention;

[0024] Figure 3b This is a schematic diagram of the structure of an LSMT model according to an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a vehicle sensor data association device according to an embodiment of the present invention. Detailed Implementation

[0026] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0027] Reference Figure 1 The diagram illustrates a flowchart of a method for associating vehicle sensor data according to an embodiment of the present invention. The vehicle is equipped with a first sensor, a second sensor, and a reference sensor, and the method may include the following steps:

[0028] Step 101: Obtain the first sensor data collected by the first sensor, the second sensor data collected by the second sensor, and the reference sensor data collected by the reference sensor.

[0029] Among them, the first sensor and the second sensor are sensors in the vehicle to be associated, which can be any type of sensor such as lidar or ultrasonic radar. The reference sensor is a sensor with a higher measurement accuracy than the first sensor and the second sensor, and the sensor data collected by the reference sensor can be used as the true value for comparison between the first sensor and the second sensor.

[0030] In practical applications, two sensors to be associated (such as the first sensor and the second sensor) and a reference sensor can be installed on the target vehicle. Before the first sensor, the second sensor and the reference sensor are put into operation, they can be calibrated and tested to ensure that the sensors are in normal working condition and can record data.

[0031] After successful calibration and testing, once the vehicle is started, each of the three sensors collects and saves data during vehicle operation. In this embodiment of the invention, the data segment used for association can be multiple frames of data collected by the three sensors from the target time (a time before the association time) to the association time. Association is performed based on multiple frames of data collected from the target time to the association time; by incorporating historical data before the association time, the accuracy and robustness of the association can be improved.

[0032] In one example, the sensor is calibrated as follows: When calibrating the sensor, it is necessary to ensure that the reference system of the target output by the sensor is aligned (the origin and the coordinate axis are consistent) to avoid introducing reference system misalignment error, and it is necessary to ensure that the target attributes represented by the two sensors are consistent. For example, the first sensor and the second sensor both represent the coordinates of the center point of the rear bumper of the vehicle.

[0033] After sensor calibration, static testing is required to check whether the output attributes of the two sensors to be associated for the same stationary vehicle target are within the error range, in order to ensure that the sensors are in normal working condition.

[0034] In another example, before the sensors officially collect data, the time synchronization status between the sensors to be associated can be checked to ensure that the first and second sensors have the same time reference; and the time synchronization status between the sensors to be associated (i.e., the first and second sensors) and the reference sensor can be checked to ensure that all three sensors are under the same time reference.

[0035] When the time bases are different, the specific method for adjusting the time synchronization is as follows: use an external NTP server to synchronize the time of the sensor to be associated with the reference sensor. At this time, the data frames output by the sensor will carry the Coordinated Universal Time (UTC) timestamp, and the time base is Coordinated Universal Time.

[0036] In one example, the raw data collected by the reference sensor (such as the raw point cloud of lidar and the target of millimeter-wave radar) can be detected and tracked offline to obtain the true target data with higher measurement accuracy than the data of the sensor to be correlated. Since the true target data is obtained by offline processing, offline mapping and optimization algorithms are used, which has better accuracy and stability than real-time detection and tracking.

[0037] Step 102: Time-align the first sensor data and the second sensor data with the reference sensor data respectively to obtain the first initial data frame pair set and the second initial data frame pair set;

[0038] The collected first sensor data, second sensor data, and reference sensor data can all carry timestamps. Therefore, each frame of data in the first sensor data can be aligned with each frame of data in the reference sensor data based on the timestamps to obtain a first initial data frame pair. Similarly, each frame of data in the second sensor data can be aligned with each frame of data in the reference sensor data based on the timestamps to obtain a second initial data frame pair.

[0039] In this embodiment of the invention, the first sensor data and the second sensor data are time-aligned with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set, including:

[0040] Determine the first timestamp of each frame of data in the first sensor data, and determine the nearest first neighboring sensor data in the reference sensor data according to the first timestamp. Use a preset uniform acceleration motion model to predict the first predicted sensor data corresponding to the first neighboring sensor data at the first timestamp. Combine the first preset sensor data and the first sensor data to generate the first initial data frame pair set.

[0041] Determine the second timestamp of each frame of data in the second sensor data, and determine the nearest second neighbor sensor data in the reference sensor data according to the second timestamp. Use a preset uniform acceleration motion model to predict the second predicted sensor data corresponding to the second neighbor sensor data at the second timestamp. Combine the second preset sensor data and the second sensor data to generate a second initial data frame pair set.

[0042] In practical applications, the following operations are performed on each frame of the first sensor data or the second sensor data:

[0043] By traversing each frame of the true data from the reference sensor, the timestamp difference of the target data frame of the sensor to be associated is determined. Then, the frame with the smallest absolute value of the timestamp difference can be determined and recorded as the first neighboring sensor data or the second neighboring sensor data. The time interval between the true data frame and this data frame is also recorded.

[0044] Based on the time interval, the motion state attribute of each target in the first or second neighboring sensor data is predicted to the current time of the target data frame to be associated with the sensor (i.e., the first or second timestamp). The ground truth prediction method can be based on a uniformly accelerated motion model. Specifically, it can be: assuming that the target attribute changes with uniform acceleration during the corresponding time interval, substituting the time difference into the uniformly accelerated motion equation, calculating the first or second predicted sensor data to be predicted to the current target data frame time of the sensor to be associated with the sensor, and then saving the predicted ground truth data frame (such as the first predicted sensor data) and the current target data frame to be associated with the sensor as a data frame pair.

[0045] After performing the above operation on each frame in the target data frame set of the sensors to be associated, a set of true values ​​and data frame pairs of the sensors to be associated can be obtained. Each sensor to be associated corresponds to a frame of true data at the same time point after prediction. Since there are two sensors to be associated, there are also two sets of data frame pairs.

[0046] Step 103: Based on the preset target algorithm, the sensor data in the first initial data frame pair set and the second initial data frame pair set are matched respectively to obtain the first associated frame pair set and the second associated frame pair set.

[0047] The target algorithm may include, but is not limited to, the Hungarian algorithm, which is a matching algorithm. In this embodiment of the invention, the corresponding target algorithm can be set according to actual needs.

[0048] After obtaining the first and second initial data frame sets, the first initial data frame set includes multiple data frame pairs corresponding to first sensor data and reference sensor data, and the second initial data frame set contains multiple data frame pairs corresponding to second sensor data and reference sensor data. Then, the target algorithm is used to perform matching and association on the first initial data frame set, and the resulting associated frame pairs are written into the first associated set. Similarly, the target algorithm is used to perform matching and association on the second initial data frame set, and the resulting associated frame pairs are written into the second associated set.

[0049] Matching data separately using a preset algorithm helps improve the accuracy of the correlation between the data from the first sensor and the data from the second sensor.

[0050] Step 104: Based on the reference sensor data in the first and second associated frame pairs, determine the target associated frame pairs composed of the first sensor data and the second sensor data.

[0051] In step 103, after the first sensor data is associated with the reference sensor data, a first associated frame pair set is obtained. After the second sensor data is associated with the reference sensor data, a second associated frame pair set is obtained. Thus, the first sensor data and the second sensor data can be linked together through the reference sensor data, thereby obtaining an associated frame pair formed by the associated data frames of the first sensor data and the data frames of the second sensor data. The obtained associated frame pair is written into the target associated frame pair set.

[0052] In one example, the start times of the two offline matching results (i.e., the first and second associated frame pairs) can be obtained. The earlier associated frame pair is selected as the starting point of time. An equally spaced time series is constructed continuously at a certain fixed time interval. The two offline matching results are then divided according to this time series to obtain two sets of data consisting of segments of offline matching results with equal time intervals.

[0053] For example, the visual sensor data and the ground truth matching result start from 0 seconds, and the millimeter-wave radar sensor data and the ground truth matching result start from 1 second. The starting time of the visual matching result (0 seconds) is selected as the starting point of time. The visual matching result dataset and the millimeter-wave radar matching result dataset are divided into two parts every 5 seconds. According to time, they can be represented as: 0 to 5 seconds, 5 to 10 seconds, 10 to 15 seconds, etc.

[0054] In one example, each data frame can be assigned a corresponding data frame identifier, which is a unique identifier for the data frame. The initial data frame set and the associated data frame set can store data frames in the form of data frame identifier pairs.

[0055] When associating data from multiple sensors to be associated, the TrackID (i.e., the data frame identifier of the reference sensor data) of the previously associated true value can be used as a bridge to obtain the target TrackID combination and the corresponding data string combination that should be associated between the two sensors to be associated.

[0056] Specifically, offline matching result pairs within the same time period can be extracted from two sets of data (i.e., the first set of associated frame pairs and the second set of associated frame pairs). From the first sensor matching result segment to be associated in the matching result pair, a target TrackID is selected and denoted as the master TrackID. The associated true target TrackID is obtained and denoted as the true TrackID. Based on the true TrackID, another sensor target TrackID associated with the true TrackID is selected from the corresponding other sensor result segment to be associated and denoted as the guest TrackID. The complete multi-frame target data strings of the two selected master and guest targets within the current time period are saved as a set of associated data (i.e., the target associated frame pair set). The data string corresponding to the master TrackID is denoted as the master target string, and the data string corresponding to the guest TrackID is denoted as the guest target string.

[0057] Similarly, based on the ground value TrackID, another sensor target that is not associated with the ground value TrackID is randomly selected from the other sensor result segment to be associated. The complete multi-frame target data strings of the two selected targets within the current time period are saved as a set of non-associated data (i.e., a set of non-associated target frame pairs). The data that should be associated and the data that should not be associated are saved as training data respectively. Then, a target is selected again in the first sensor matching result segment to be associated, and the above operation is repeated until all target TrackIDs in the first result segment have been selected.

[0058] In this embodiment of the invention, the association of sensor data is illustrated using two sensor data sets to be associated as an example. The association process for more than two sensor data sets can be adapted according to this method.

[0059] In this embodiment of the invention, first sensor data collected by a first sensor, second sensor data collected by a second sensor, and reference sensor data collected by a reference sensor can be acquired. The first sensor data and second sensor data are time-aligned with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set. Then, based on a preset target algorithm, the sensor data in the first initial data frame pair set and the second initial data frame pair set are matched to obtain a first associated frame pair set and a second associated frame pair set. Based on the reference sensor data in the first associated frame pair set and the second associated frame pair set, a target associated frame pair set composed of the first sensor data and the second sensor data is determined. This realizes the association of historical information from the target time to the associated time. When applied to the association of multi-sensor data, it has good association robustness and accuracy.

[0060] Reference Figure 2 The diagram illustrates another step in vehicle sensor data association according to an embodiment of the present invention, which may include the following steps:

[0061] Step 201: Obtain the first sensor data collected by the first sensor, the second sensor data collected by the second sensor, and the reference sensor data collected by the reference sensor.

[0062] Step 202: Time-align the first sensor data and the second sensor data with the reference sensor data respectively to obtain the first initial data frame pair set and the second initial data frame pair set;

[0063] Step 203: Use a preset target algorithm to match the first sensor data and the reference sensor data in the first initial data frame pair set to obtain the third associated frame pair set; and use the target algorithm to match the second sensor data and the reference sensor data in the second initial data frame pair set to obtain the fourth associated frame pair set.

[0064] In one embodiment of the present invention, a preset target algorithm is used to match the first sensor data and the reference sensor data in the first initial data frame pair set to obtain a third associated frame pair set, including:

[0065] Sub-step S11: Calculate the cost matrix based on the sensor data in the set according to the first initial data frame, and reassign the costs of the cost matrix that are greater than the preset association threshold according to the association threshold.

[0066] In practical applications, the cost between sensor data can be determined based on spatial distance and / or velocity distance, thus forming a cost matrix. An association threshold is set for this cost matrix to control the cost value. Users can set the association threshold size according to common practical needs; this invention does not impose excessive restrictions on it. After determining the association threshold, the cost matrix can be adjusted using it. Specifically, each cost can be compared with the association threshold. When a cost exceeds the association threshold, that cost is reassigned, such as by reassigning a value according to the association threshold.

[0067] Sub-step S12: Based on the cost matrix, the Hungarian algorithm is used to perform minimum cost matching on the sensors in the initial data frame pair set to obtain preliminary matching results.

[0068] Among them, minimum cost matching is to select the matching scheme with the minimum total cost among all matching schemes.

[0069] Sub-step S13: When the cost of the matching data frame pair in the cost matrix in the preliminary matching result is less than the association threshold, the matching data frame pair is stored in the third association frame pair set.

[0070] After the initial matching is completed, the cost corresponding to the matched data frame pair obtained in the initial matching can be queried in the cost matrix, and the size of the cost and the association threshold can be judged. If the cost corresponding to the matched data frame pair is less than the association threshold, it means that the matching effect of the matched data frame pair is good, and the data frame pair can be stored in the third association frame pair set. If the cost corresponding to the matched data frame pair is not less than the association threshold, it means that the matching effect of the matched data frame pair is poor, and the matching result is discarded.

[0071] Similarly, the matching process of the second initial data frame can be adaptively adjusted by referring to the matching process of the first initial data frame in S11 to S13.

[0072] In one embodiment of the present invention, the method further includes: during the preset target life cycle, counting the multi-frame candidate reference sensor data associated with the target first sensor data in the third associated frame pair set, and the cumulative association number of each frame candidate reference sensor data; when the highest cumulative association number is not greater than a preset threshold, storing the associated frame pair composed of the target first sensor data and the candidate reference sensor data corresponding to the highest cumulative association number in the third associated frame pair set.

[0073] In practical applications, the association can be initiated multiple times within the target's lifecycle. During each association process, the target algorithm can be used for preliminary matching. Based on the obtained matching results, the target's entire lifecycle can be optimized to remove outliers and obtain association frame pair information that better matches the real-world scenario.

[0074] The optimization process for the initial matching results is as follows: Within the preset target lifecycle, the ground truth target TrackIDs and the cumulative number of associated frames associated with each target TrackID are calculated. For each target TrackID, the ground truth target TrackID with the highest cumulative number of associated frames is selected, and it is determined whether its cumulative frame count reaches a preset percentage (e.g., 80%) of the total frame count of the target's lifecycle. If so, the target TrackID and the ground truth target TrackID are saved as a pair in the optimized association pair information. The optimized association pair information stores highly reliable combinations of target TrackIDs and corresponding ground truth target TrackIDs representing the same real target.

[0075] Step 204: The target algorithm is used to match the first sensor data in the third associated frame pair set with the reference sensor data to obtain the first associated frame pair set; and the target algorithm is used to match the second sensor data in the fourth associated frame pair set with the reference sensor data to obtain the second associated frame pair set.

[0076] The target algorithm can be the Hungarian algorithm.

[0077] After the initial matching, the target algorithm can be used to perform a second matching based on the initial matching result as a priori condition. In this way, when the two targets have been confirmed to have a high probability of matching, noise that may be briefly misassociated with other targets during sudden changes in the track can be eliminated, resulting in a more accurate matching result.

[0078] In practical applications, after obtaining the third and fourth associated frame sets through the initial matching in step 203, the sensor targets to be tested with known association information in the third and fourth associated frame sets obtained in step 203, as well as the associated ground truth targets in the same frame, can be pre-confirmed and associated. Specifically, in the cost matrix calculation, the cost of the original cost that is less than the maximum association threshold can be directly assigned to 0, and then the Hungarian algorithm matching can be performed to minimize its total cost, thereby obtaining the first and second associated frame sets, which are groups of TrackIDs of the sensor to be associated and the corresponding ground truth targets representing the same ground truth target.

[0079] Step 205: Based on the reference sensor data in the first and second associated frame pairs, determine the target associated frame pairs composed of the first sensor data and the second sensor data.

[0080] In this embodiment of the invention, the data collected by multiple sensors to be associated are matched twice with the reference sensor before being associated. The second matching helps to improve the matching accuracy, thereby improving the association accuracy.

[0081] Reference Figure 3a The diagram illustrates another step in vehicle sensor data association according to an embodiment of the present invention, which may include the following steps:

[0082] Step 301: Obtain the first sensor data collected by the first sensor, the second sensor data collected by the second sensor, and the reference sensor data collected by the reference sensor.

[0083] Step 302: Time-align the first sensor data and the second sensor data with the reference sensor data to obtain the first initial data frame pair set and the second initial data frame pair set.

[0084] Step 303: Based on the preset target algorithm, the sensor data in the first initial data frame pair set and the second initial data frame pair set are matched respectively to obtain the first associated frame pair set and the second associated frame pair set.

[0085] Step 304: Based on the reference sensor data in the first and second associated frame pairs, determine the target associated frame pairs composed of the first sensor data and the second sensor data.

[0086] Step 305: Based on the reference sensor data in the first associated frame pair set and the second associated frame pair set, determine the target non-associated frame pair set composed of the first sensor data and the second sensor data.

[0087] Step 306: Extract the training set of the preset LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set;

[0088] In one example, steps 304 and 305 are performed on each result segment of the two datasets divided by time, resulting in a 1:1 ratio of positive to negative samples in the training dataset. Furthermore, the training dataset can be randomly split into a training set and a test set, and the ratio of the number of data sets in the training set and the test set can be set according to actual needs. For example, the ratio of the number of data sets in the training set and the test set can be 8:2. To ensure a better estimation of the generalization error, 80% of the data is usually used for training, while 20% is reserved for testing.

[0089] In one example, an LSTM (Long Short-Term Memory) neural network model is built, with the specific structure as follows: Figure 3bAs shown: The overall network structure is Y-shaped; the first layer is a Siamese LSTM layer, which has two parallel LSTM networks. The input layer size of these two LSTM networks is 5 (number of neurons), the hidden layer size is 20 (number of neurons), and the number of layers is 1. They share weights, meaning that the weight values ​​and bias values ​​in the two LSTM networks are always consistent; the second layer is a Flatten layer, which flattens the input tensor into a vector; after calculating the distance between the two vectors, it is input into the third layer, which is a fully connected layer (reducing the 20-dimensional vector to a 10-dimensional vector). The fourth layer is also a fully connected layer (converting the 10-dimensional vector to a 1-dimensional vector). Finally, the output is normalized by the Sigmoid function, which can normalize any value to a value between 0 and 1.

[0090] Step 307: Train the LSMT neural network model according to the training set to obtain the target association model.

[0091] After obtaining the training set, the target association model can be trained using the data in the training set. The training process is as follows:

[0092] (1) First, set the model training hyperparameters. For example, set batch_size (the number of data points passed to the program for training in a single batch) to 32 and epochs (the number of times all data is fed into the network for training) to 100; select the Adam optimizer and set the learning rate to 0.001.

[0093] (2) Randomly select training data (data strings that should be associated or data strings that should not be associated) from the training set without replacement and input them into the LSMT neural network model. The main data string is tensorized and passed to the first LSTM unit, and the guest data string is passed to the second LSTM unit. After forward propagation, the pairwise trajectory sequence association probability is output. The output association probability is compared with the "should be associated" or "should not be associated" flag (True=1, False=0) to calculate the binary classification cross-entropy loss and output the Loss. Then, backpropagation is performed through this Loss. The current Loss is output after every 10 batches to observe whether the training process is normal. If the Loss decreases with the increase of the batch number, it indicates that the training process is normal.

[0094] In one embodiment of the present invention, the method further includes: extracting a test set of the LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set; inputting the data in the test set into the target associated model for model testing to obtain the test results corresponding to the test set; and determining the learning effect of the target associated model based on the test results and the actual association results of the test set.

[0095] After the association model is trained, the test set is traversed, and each test data in the test set is fed into the model to obtain the association probability output. If the probability is greater than the preset association probability (e.g., 50%), it is considered to be confirmed as an association, and if the probability is less than the preset probability, it is considered to be confirmed as not an association. The test results are compared with the actual offline association results (i.e., the target association frame set), and the association accuracy is calculated. The association accuracy is equal to the number of correct associations divided by the total number of tests. The association accuracy is used to confirm the learning effect of the neural network. If the association accuracy is greater than the preset association accuracy threshold (e.g., 50%), it can be proved that the network has learned some patterns.

[0096] In practical applications, the target association model can be used for online association of multiple sensor data in a vehicle. In a vehicle equipped with a first sensor and a second sensor, sensor information can be collected in real time during vehicle operation, and the collected sensor data can be input into a trained target association model. The target association model can output association probabilities, and then the sensor data can be associated according to the association probabilities.

[0097] In this invention, a real-time track association method based on twin LSTM can be used to incorporate historical track information into the association judgment process. Compared with traditional association methods based on the statistical distance of a single frame or consecutive frames, this method utilizes more track information and has higher association accuracy and robustness.

[0098] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0099] Reference Figure 4 The diagram illustrates the structure of a vehicle sensor data association device according to an embodiment of the present invention, which may include the following modules:

[0100] Data acquisition module 401 is used to acquire first sensor data acquired by the first sensor, second sensor data acquired by the second sensor, and reference sensor data acquired by the reference sensor.

[0101] The time alignment module 402 is used to time-align the first sensor data and the second sensor data with the reference sensor data respectively to obtain a first initial data frame pair set and a second initial data frame pair set.

[0102] The algorithm matching module 403 is used to match the sensor data in the first initial data frame pair set and the second initial data frame pair set based on a preset target algorithm, respectively, to obtain the first associated frame pair set and the second associated frame pair set.

[0103] The data association module 404 is used to determine a target associated frame pair set composed of the first sensor data and the second sensor data based on the reference sensor data in the first associated frame pair set and the second associated frame pair set.

[0104] In one embodiment of the present invention, the algorithm matching module 403 includes:

[0105] The first algorithm matching submodule is used to match the first sensor data and the reference sensor data in the first initial data frame pair set with a preset target algorithm to obtain a third associated frame pair set, and to match the second sensor data and the reference sensor data in the second initial data frame pair set with the target algorithm to obtain a fourth associated frame pair set.

[0106] The second algorithm matching submodule is used to match the first sensor data and the reference sensor data in the third associated frame pair set with the target algorithm to obtain the first associated frame pair set, and to match the second sensor data and the reference sensor data in the fourth associated frame pair set with the target algorithm to obtain the second associated frame pair set.

[0107] In one embodiment of the present invention, the first algorithm matching submodule includes:

[0108] The first cost matrix unit is used to calculate the cost matrix based on the sensor data in the set according to the first initial data frame, and to reassign the cost of the cost matrix that is greater than the preset association threshold according to the association threshold.

[0109] The matching unit is used to perform minimum cost matching on the sensors in the initial data frame pair set according to the cost matrix and the Hungarian algorithm to obtain preliminary matching results;

[0110] The matching data frame pair writing unit is used to store the matching data frame pair in the third associated frame pair set when the cost of the matching data frame pair in the cost matrix in the preliminary matching result is less than the association threshold.

[0111] In one embodiment of the present invention, the first algorithm matching submodule further includes:

[0112] The matching statistics unit is used to count the number of candidate reference sensor data associated with the first sensor data of the set target in the third associated frame during the preset target life cycle, as well as the cumulative number of associations of each candidate reference sensor data frame.

[0113] The associated frame storage module is used to store the associated frame pair consisting of the target first sensor data and the candidate reference sensor data corresponding to the highest cumulative association number in the third associated frame pair set when the highest cumulative association number is not greater than a preset threshold.

[0114] In one embodiment of the present invention, the time alignment module 402 includes:

[0115] The first alignment submodule is used to determine the first timestamp of each frame of data in the first sensor data, and determine the nearest first neighboring sensor data in the reference sensor data according to the first timestamp, use a preset uniform acceleration motion model to predict the first predicted sensor data corresponding to the first neighboring sensor data at the first timestamp, and combine the first preset sensor data and the first sensor data to generate a first initial data frame pair set.

[0116] The second alignment submodule is used to determine the second timestamp of each frame of data in the second sensor data, and determine the nearest second neighboring sensor data in the reference sensor data according to the second timestamp, predict the second predicted sensor data corresponding to the second neighboring sensor data at the second timestamp using a preset uniform acceleration motion model, and combine the second preset sensor data and the second sensor data to generate a second initial data frame pair set.

[0117] In one embodiment of the present invention, the device further includes:

[0118] The target uncorrelated frame pair generation module is used to determine the target uncorrelated frame pair composed of the first sensor data and the second sensor data based on the reference sensor data in the first correlated frame pair and the second correlated frame pair;

[0119] The training set extraction module is used to extract a training set of a preset LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set;

[0120] The model training module is used to train the LSMT neural network model according to the training set to obtain the target association model.

[0121] In one embodiment of the present invention, the device further includes:

[0122] The test set extraction module is used to extract the test set of the LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set;

[0123] The model testing module is used to input the data in the test set into the target association model for model testing, and obtain the test results corresponding to the test set.

[0124] Model evaluation is used to determine the learning performance of the target association model based on the test results and the actual association results of the test set.

[0125] This invention also provides a vehicle that may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described method for associating vehicle sensor data.

[0126] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for associating vehicle sensor data.

[0127] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0128] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0129] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0134] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0135] The above provides a detailed description of a method, apparatus, vehicle, and medium for vehicle sensor data association. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for associating vehicle sensor data, characterized in that, The vehicle is equipped with a first sensor, a second sensor, and a reference sensor. The method includes: Acquire the first sensor data collected by the first sensor, the second sensor data collected by the second sensor, and the reference sensor data collected by the reference sensor; The first sensor data and the second sensor data are time-aligned with the reference sensor data to obtain a first initial data frame pair set and a second initial data frame pair set. A preset target algorithm is used to match the first sensor data and the reference sensor data in the first initial data frame pair set to obtain a third associated frame pair set; and the target algorithm is used to match the second sensor data and the reference sensor data in the second initial data frame pair set to obtain a fourth associated frame pair set. The target algorithm is used to match the first sensor data and the reference sensor data in the third associated frame pair set to obtain the first associated frame pair set; and the target algorithm is used to match the second sensor data and the reference sensor data in the fourth associated frame pair set to obtain the second associated frame pair set. Based on the reference sensor data in the first and second associated frame pairs, a target associated frame pair set composed of the first and second sensor data is determined.

2. The method according to claim 1, characterized in that, The step of matching the first sensor data and the reference sensor data in the first initial data frame pair set using a preset target algorithm to obtain the third associated frame pair set includes: The cost matrix is ​​calculated based on the sensor data in the set according to the first initial data frame, and the costs of the cost matrix that are greater than the preset association threshold are reassigned according to the association threshold. Based on the cost matrix, the Hungarian algorithm is used to perform minimum cost matching on the sensors in the initial data frame pair set to obtain preliminary matching results; When the cost of a matching data frame pair in the cost matrix is ​​less than the association threshold in the preliminary matching result, the matching data frame pair is stored in the third association frame pair set.

3. The method according to claim 1, characterized in that, Also includes: Throughout the preset target lifecycle, the third associated frame is used to count the number of candidate reference sensor data associated with the first sensor data of the target in the set, as well as the cumulative number of associations for each frame of candidate reference sensor data. When the highest cumulative correlation number is not greater than a preset threshold, the correlation frame pair consisting of the target first sensor data and the candidate reference sensor data corresponding to the highest cumulative correlation number is stored in the third correlation frame pair set.

4. The method according to any one of claims 1, characterized in that, The step of aligning the first sensor data and the second sensor data with the reference sensor data in time to obtain a first initial data frame pair set and a second initial data frame pair set includes: Determine the first timestamp of each frame of data in the first sensor data, and determine the nearest first neighboring sensor data in the reference sensor data according to the first timestamp. Use a preset uniform acceleration motion model to predict the first predicted sensor data corresponding to the first neighboring sensor data at the first timestamp. Combine the first predicted sensor data and the first sensor data to generate a first initial data frame pair set. The second timestamp of each frame of data in the second sensor data is determined, and the nearest second neighboring sensor data is determined in the reference sensor data according to the second timestamp. The second predicted sensor data corresponding to the second neighboring sensor data at the second timestamp is predicted using a preset uniform acceleration motion model. The second predicted sensor data and the second sensor data are combined to generate a second initial data frame pair set.

5. The method according to any one of claims 1 to 4, characterized in that, Also includes: Based on the reference sensor data in the first and second associated frame pairs, a target non-associated frame pair set composed of the first and second sensor data is determined. Extract the training set of the preset LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set; The LSMT neural network model is trained according to the training set to obtain the target association model.

6. The method according to claim 5, characterized in that, Also includes: Extract the test set of the LSMT neural network model from the target associated frame pair set and the target unassociated frame pair set; The data in the test set is input into the target association model for model testing, and the test results corresponding to the test set are obtained. The learning effect of the target association model is determined based on the test results and the actual association results of the test set.

7. A device for associating vehicle sensor data, characterized in that, The vehicle is equipped with a first sensor, a second sensor, and a reference sensor. The device includes: The data acquisition module is used to acquire the first sensor data acquired by the first sensor, the second sensor data acquired by the second sensor, and the reference sensor data acquired by the reference sensor. The time alignment module is used to time-align the first sensor data and the second sensor data with the reference sensor data respectively to obtain a first initial data frame pair set and a second initial data frame pair set. The first algorithm matching submodule is used to match the first sensor data and the reference sensor data in the first initial data frame pair set with a preset target algorithm to obtain a third associated frame pair set, and to match the second sensor data and the reference sensor data in the second initial data frame pair set with the target algorithm to obtain a fourth associated frame pair set. The second algorithm matching submodule is used to match the first sensor data and the reference sensor data in the third associated frame pair set with the target algorithm to obtain the first associated frame pair set, and to match the second sensor data and the reference sensor data in the fourth associated frame pair set with the target algorithm to obtain the second associated frame pair set. The data association module is used to determine a target associated frame pair set composed of the first sensor data and the second sensor data based on the reference sensor data in the first associated frame pair set and the second associated frame pair set.

8. A vehicle, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method for associating vehicle sensor data as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for associating vehicle sensor data as described in any one of claims 1 to 6.

Images