Multi-target tracking and trajectory prediction method for vehicle-mounted terminal, and electronic device and storage medium

Abstract

Disclosed in the embodiments of the present application is a multi-target tracking and trajectory prediction method for a vehicle-mounted terminal. The method comprises: acquiring a first detection box of each obstacle in a video frame at a first moment, and a first tracking trajectory of each obstacle in video frames within a first duration before the first moment; and inputting the first detection boxes and the first tracking trajectories into a tracking prediction model, and processing, by means of the model, the first detection boxes and the first tracking trajectories to obtain a matching relationship between the first detection boxes and the first tracking trajectories, and a motion trajectory of the obstacle in each first detection box within a second duration after the first moment, wherein the tracking prediction model is obtained by means of performing training on the basis of a second detection box of each obstacle in each frame of a sample video, a second tracking trajectory of each obstacle in video frames within the first duration before the moment of each frame, the actual matching relationship between the second detection boxes and the second tracking trajectories, and an actual motion trajectory of the obstacle in each second detection box within the second duration after the moment of each frame.

Description

Vehicle-mounted multi-target tracking and trajectory prediction method, electronic device and storage medium

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to the Chinese patent application No. 202410823239.2, filed on June 24, 2024, and entitled "Vehicle-mounted multi-target tracking and trajectory prediction method, electronic device and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of machine vision, and in particular to a vehicle-mounted multi-target tracking and trajectory prediction method, an electronic device and a storage medium. BACKGROUND

[0004] Intelligent driving technology is a technology that enables vehicles to travel autonomously without human control. It is based on advanced sensors, computer vision, artificial intelligence and machine learning, allowing vehicles to perceive the surrounding environment, make decisions and perform corresponding actions. As two important tasks of intelligent driving technology, multi-target tracking and trajectory prediction play a bridging role.

[0005] Currently, since the trajectory prediction task is a downstream task of the multi-target tracking task, the multi-target tracking and trajectory prediction tasks are combined, and a same model is used to output tracking results and prediction results. However, in the related art, the above model is trained by using a single-frame model training manner, which is quite different from the streaming scenario in actual deployment of the vehicle-mounted end, resulting in inaccurate tracking results and trajectory prediction results output by the model. SUMMARY

[0006] The present application provides a vehicle-mounted multi-target tracking and trajectory prediction method, an electronic device and a storage medium to solve the technical problem of inaccurate multi-target tracking results and trajectory prediction results of the vehicle-mounted end.

[0007] According to a first aspect of the present application, a vehicle-mounted multi-target tracking and trajectory prediction method is disclosed, the method comprising:

[0008] obtaining a first detection box of each obstacle in a video frame at a first time to be predicted and a first tracking trajectory of each obstacle in each video frame within a first time length before the first time;

[0009] input the first detection box and the first tracking trajectory into a tracking prediction model, process the first detection box and the first tracking trajectory through the tracking prediction model, obtain a matching relationship between the first detection box and the first tracking trajectory, and a motion trajectory of an obstacle in the first detection box within a second time length after the first time;

[0010] The tracking prediction model is trained based on a second detection box of each obstacle in each frame of a sample video, a second tracking trajectory of each obstacle in a video frame within a first time length before a time at which the each frame is located, a real matching relationship between the second detection box and the second tracking trajectory, and a real motion trajectory of an obstacle in the second detection box within a second time length after a time at which the each frame is located.

[0011] According to a second aspect of the present application, an electronic device is disclosed, which comprises a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the vehicle-mounted multi-target tracking and trajectory prediction method according to the first aspect.

[0012] According to a third aspect of the present application, a computer readable storage medium is disclosed, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the vehicle-mounted multi-target tracking and trajectory prediction method according to the first aspect.

[0013] According to a fourth aspect of the present application, a computer program product is disclosed, which comprises a computer program / instruction, and the computer program / instruction is executed by a processor to implement the vehicle-mounted multi-target tracking and trajectory prediction method according to the first aspect.

[0014] In the embodiments of the present application, a first detection box of each obstacle in a video frame at a first time to be predicted and a first tracking trajectory of each obstacle in each video frame within a first time length before the first time are obtained; the first detection box and the first tracking trajectory are input into a tracking prediction model, the first detection box and the first tracking trajectory are processed through the tracking prediction model, a matching relationship between the first detection box and the first tracking trajectory is obtained, and a motion trajectory of an obstacle in the first detection box within a second time length after the first time is obtained; the tracking prediction model is trained based on a second detection box of each obstacle in each frame of a sample video, a second tracking trajectory of each obstacle in a video frame within a first time length before a time at which the each frame is located, a real matching relationship between the second detection box and the second tracking trajectory, and a real motion trajectory of an obstacle in the second detection box within a second time length after a time at which the each frame is located.

[0015] It can be seen that in the embodiment of the present application, considering the streaming scenario when the vehicle-mounted end is actually deployed, the training mode of the video stream can be used to combine the multi-target tracking and trajectory prediction tasks, train the tracking prediction model, and perform multi-target tracking and trajectory prediction of the vehicle-mounted end through the tracking prediction model. Since the tracking prediction model is adapted to the streaming scenario when the vehicle-mounted end is actually deployed, and can correct the occlusion and missed detection results of the upstream module, the accuracy of the multi-target tracking result and the trajectory prediction result of the vehicle-mounted end can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG. 1 is a flowchart of a vehicle-mounted multi-target tracking and trajectory prediction method according to an embodiment of the present application;

[0017] FIG. 2 is a flowchart of an embodiment of step 102 according to an embodiment of the present application;

[0018] FIG. 3 is an example diagram of the network structure of the tracking prediction model according to an embodiment of the present application;

[0019] FIG. 4 is a structural schematic diagram of a vehicle-mounted multi-target tracking and trajectory prediction device according to an embodiment of the present application;

[0020] FIG. 5 is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] It should be noted that the image data obtained by the present application is accessed, collected, stored and applied to subsequent analysis and processing after the user or the relevant data owner agrees and authorizes the collection content, data use, processing method and other information of the data, and the user or the relevant data owner is provided with the access, correction, deletion of the data, and the method of revoking the authorization.

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0023] It should be noted that for the method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the present application.

[0024] In order to facilitate understanding, first of all, some related concepts and application scenarios involved in the embodiments of the present application will be introduced.

[0025] I. Related Concepts

[0026] Multi-target tracking refers to associating and tracking multiple targets observed at different times to obtain state information such as position / trajectory, speed, direction, etc. of each target.

[0027] Trajectory prediction refers to predicting the future driving trajectory of obstacles around a vehicle by analyzing various data during the vehicle's driving process.

[0028] Spatial-Temporal Attention Mechanism (STAM) refers to using attention mechanism in video understanding to dynamically control the attention degree for different time points, thereby improving the accuracy and efficiency of video understanding. The core idea is to adaptively adjust the attention degree of the model according to different time periods and spatial positions of the video. In video understanding, the application of spatial-temporal attention mechanism can be divided into two aspects: one is spatial attention mechanism, and the other is temporal attention mechanism.

[0029] Spatial attention mechanism refers to giving different attention degrees to different spatial positions of pixels in a video. In image processing, convolutional neural networks (CNN) are usually used to extract image features, but in video processing, due to the complexity of video data, simply using CNN cannot well extract important information in the video, so the importance of different pixels in space needs to be further considered. Spatial attention mechanism can adaptively adjust the weights in the CNN model according to the importance of the pixels, thereby improving the accuracy of video understanding.

[0030] Temporal attention mechanism refers to giving different attention degrees to frame images of different time periods in a video. In video processing, since each frame of image represents a different time point, we need to consider the correlation between different time points. Temporal attention mechanism can adaptively adjust the weights of the model according to the similarity of frame images, thereby improving the efficiency of video understanding.

[0031] Multilayer Perceptron (MLP) is a kind of feedforward artificial neural network model, which maps multiple input data sets to a single output data set.

[0032] Multi-Head Cross Attention (MHCA) is a kind of attention mechanism that cross-fuses multi-dimensional information.

[0033] II. Application Scenarios

[0034] Multi-target tracking and trajectory prediction are two crucial tasks in the field of intelligent driving, playing a role of connecting the past and the future. Currently, domestic and foreign manufacturers generally use a single-module training paradigm, i.e., tracking task and prediction task are performed separately. However, due to the separate training of the two tasks, the current paradigm has some problems: the data input to the separate trajectory prediction model is standard data, and imperfect tracking results will interfere with the results of the model, causing errors to accumulate between different tasks. A current strategy is to combine the tracking and prediction tasks together and use the same model to give tracking and prediction results at the same time.

[0035] In the related art, this strategy has three disadvantages: first, a single-frame model training method is used, which is quite different from the streaming scenario in actual deployment, leading to inconsistency between training and testing; second, the tracking task and the prediction task have inconsistent requirements for the coordinate system, the tracking model is represented in the vehicle coordinate system, while the prediction model is represented in the obstacle coordinate system; third, the current strategy only focuses on predicting the objects detected in the current frame, lacking the ability to handle occlusion and missed detection of upstream modules. It can be seen that in the related art, the tracking results and trajectory prediction results output by the model are inaccurate.

[0036] To solve the above technical problems, the embodiments of the present application provide a vehicle-mounted multi-target tracking and trajectory prediction method, an electronic device and a storage medium.

[0037] Next, a vehicle-mounted multi-target tracking and trajectory prediction method provided by the embodiments of the present application will be described in conjunction with the accompanying drawings.

[0038] It should be noted that the vehicle-mounted multi-target tracking and trajectory prediction method provided by the embodiments of the present application is applicable to an electronic device, which can be a vehicle-mounted device, a computer device, or a server, etc. in actual application, and the embodiments of the present application do not limit this.

[0039] FIG. 1 is a flowchart of a vehicle-mounted multi-target tracking and trajectory prediction method provided by the embodiments of the present application, as shown in FIG. 1, the method can include the following steps: step 101 and step 102;

[0040] In step 101, the first detection box of each obstacle in the video frame at the first time to be predicted and the first tracking trajectory of each obstacle in each video frame within the first time length before the first time are obtained.

[0041] In the embodiments of the present application, the obstacle can be any moving object in a driving scene, for example, a car, a bicycle, an electric scooter, a person, or an animal, etc.

[0042] In the embodiments of the present application, the first time usually refers to the current time.

[0043] In the embodiments of the present application, each obstacle in the video frame at the first time corresponds to a first detection box.

[0044] In the embodiments of the present application, the video frames within the first time length before the first time can be the video frames within the time period of [first time-first time length, first time-1 millisecond].

[0045] In the embodiments of the present application, the first tracking trajectory can be the tracking trajectory of each frame detection box of the obstacle within the time period of [first time-first time length, first time-1 millisecond] linked by the real matching relationship.

[0046] Exemplarily, the first time is t 10 The first time length includes three time points, so the first detection box of each obstacle in the video frame at t 10 The first detection box of each obstacle in the video frame at t

[0047] In the embodiments of the present application, the above-mentioned video frame can be an image collected by a vehicle-mounted camera, or a point cloud video frame collected by a laser radar, which is not limited in the embodiments of the present application.

[0048] In step 102, the first detection box and the first tracking trajectory are input into a tracking prediction model, the first detection box and the first tracking trajectory are processed by the tracking prediction model, and the matching relationship between the first detection box and the first tracking trajectory and the motion trajectory of the obstacle in the first detection box within a second time length after the first time are obtained.

[0049] In the embodiments of the present application, each first detection box and each first tracking trajectory are input into the tracking prediction model, each first detection box and each first tracking trajectory are processed by the tracking prediction model, and the matching relationship between each first detection box and the first tracking trajectory and the motion trajectory of the obstacle in each first detection box within a second time length after the first time are obtained.

[0050] In the embodiments of the present application, the tracking prediction model is trained based on the second detection box of each obstacle in each frame of the sample video, the second tracking trajectory of each obstacle in the video frame within the first time length before the time point of each frame, the real matching relationship between the second detection box and the second tracking trajectory, and the real motion trajectory of the obstacle in the second detection box within the second time length after the time point of each frame.

[0051] In the embodiments of the present application, considering that the samples need to have a time sequence relationship, therefore, the sample video used for training the tracking prediction model is usually a video with a long time length.

[0052] In the embodiments of the present application, the training phase of the tracking prediction model can include the following steps: constructing a backbone network; slicing the sample video through a sliding window, each frame in the slice is taken as a current frame in turn, thereby dividing a first time length of historical time and a second time length of future time, inputting the second detection box of each obstacle in a batch of current frames and the second tracking trajectory of each obstacle in the video frames of the first time length of historical time into the backbone network, the network outputs the prediction matching relationship between the second detection box and the second tracking trajectory and the predicted motion trajectory of the obstacle in the second detection box in the future time, calculating the loss value of the multi-target tracking dimension according to the prediction matching relationship between the second detection box and the second tracking trajectory and the real matching relationship, calculating the loss value of the trajectory prediction dimension according to the predicted motion trajectory of the obstacle in the second detection box in the future time and the real motion trajectory; the data of the next batch is obtained by cyclic promotion, and the loss values of the two dimensions at each time are calculated, when the entire sliding window is traversed, the loss values of the two dimensions at all times are accumulated, and the parameters of the network are adjusted through gradient back propagation. The network is input after the parameters are updated, and the above process is repeated until the model converges, and a streaming tracking prediction model is obtained.

[0053] In the embodiments of the present application, since the tracking prediction model is trained based on the second detection box of each obstacle in each frame of the sample video, the second tracking trajectory of each obstacle in the video frames within the first time length before the time at which each frame is located, the real matching relationship between the second detection box and the second tracking trajectory, and the real motion trajectory of the obstacle in the second detection box within the second time length after the time at which each frame is located, the shared characteristics of the real trajectory matching relationship between the current time and a period of time before the current time and the shared characteristics of the real future trajectory can be learned, and therefore the multi-target tracking and trajectory prediction can be accurately realized based on the tracking prediction model.

[0054] As can be seen from the above embodiments, in the embodiments, the first detection box of each obstacle in the video frame of the first time to be predicted and the first tracking trajectory of each obstacle in each video frame within the first time length before the first time are obtained; the first detection box and the first tracking trajectory are input into the tracking prediction model, the first detection box and the first tracking trajectory are processed through the tracking prediction model, the matching relationship between the first detection box and the first tracking trajectory, and the motion trajectory of the obstacle in the first detection box within the second time length after the first time are obtained; the tracking prediction model is trained based on the second detection box of each obstacle in each frame of the sample video, the second tracking trajectory of each obstacle in the video frames within the first time length before the time at which each frame is located, the real matching relationship between the second detection box and the second tracking trajectory, and the real motion trajectory of the obstacle in the second detection box within the second time length after the time at which each frame is located.

[0055] It can be seen that, in the embodiment of the application, considering the streaming scenario when the vehicle-mounted end is actually deployed, the training manner of the video stream can be used to combine the multi-target tracking and trajectory prediction two tasks, train the tracking prediction model, and perform multi-target tracking and trajectory prediction of the vehicle-mounted end through the tracking prediction model. Since the tracking prediction model is adapted to the streaming scenario when the vehicle-mounted end is actually deployed, and can correct the occlusion and missed detection results of the upstream module, the accuracy of the multi-target tracking result and the trajectory prediction result of the vehicle-mounted end can be improved.

[0056] In some embodiments provided in the application, as shown in FIG. 2, the step 102 can include the following steps: step 201, step 202, step 203, step 204 and step 205, and the steps 202-205 are realized through the tracking prediction model.

[0057] In step 201, a current detection box feature vector corresponding to the first detection box is generated.

[0058] In the embodiment of the application, considering that the first detection box and the first tracking trajectory are in different coordinate spaces, it is difficult to process in different coordinate spaces and the processing result may have errors, therefore, it is necessary to convert the first detection box and the first tracking trajectory to the same coordinate space. Specifically, a current detection box feature vector corresponding to the first detection box is generated, and a historical trajectory feature vector corresponding to the first tracking trajectory is generated, so that the current detection box feature vector and the historical trajectory feature vector are in the same coordinate space.

[0059] In the embodiment of the application, each first detection box can be directly feature-encoded to obtain a current detection box feature vector corresponding to each first detection box.

[0060] In step 202, a historical trajectory feature vector corresponding to the first tracking trajectory is generated.

[0061] In some embodiments, each first tracking trajectory can be directly feature-encoded to obtain a historical trajectory feature vector corresponding to each first tracking trajectory.

[0062] In step 203, the current detection box feature vector and the historical trajectory feature vector are processed by feature interaction to obtain a first context feature vector corresponding to the current detection box feature vector and a second context feature vector corresponding to the historical trajectory feature vector.

[0063] In the embodiment of the application, the first context feature vector contains the information of the current detection box and the information of the historical trajectory, but the information of different obstacles corresponds to the information of the current detection box one by one.

[0064] In the embodiments of the present application, the second context feature vector contains information of the current detection box and information of the historical trajectory, but the information of different obstacles and the information of the historical trajectory correspond one by one.

[0065] In step 204, a matching relationship between the first detection box and the first tracking trajectory is determined according to the first context feature vector and the second context feature vector.

[0066] In the embodiments of the present application, the matching relationship between the first detection box and the first tracking trajectory can be determined by calculating the similarity between each first context feature vector and each second context feature vector, and determining the matching relationship according to the calculated similarities; wherein the similarity is positively correlated with the matching degree.

[0067] In step 205, a motion trajectory of the obstacle in the first detection box within a second time duration after the first time is determined according to the first context feature vector, the second context feature vector and a trajectory template, wherein the trajectory template includes a plurality of candidate motion trajectories.

[0068] In the embodiments of the present application, the commonly used motion trajectories of human driving can be fitted in advance according to big data in a driving scene, and the fitted motion trajectories are determined as the trajectory template.

[0069] In the embodiments of the present application, the motion trajectory of the obstacle in each first detection box within the second time duration after the first time can be fitted according to the first context feature vector, the second context feature vector and the trajectory template.

[0070] It can be seen that, in the embodiments of the present application, since the above-mentioned tracking prediction model is adapted to the stream scene when the vehicle terminal is actually deployed, and can correct the occlusion and missed detection results of the upstream module, the accuracy of the multi-target tracking result and the trajectory prediction result of the vehicle terminal can be improved by using the tracking prediction model for multi-target tracking and trajectory prediction of the vehicle terminal.

[0071] In some embodiments provided by the present application, the above-mentioned step 203 can specifically include the following step 2031.

[0072] In step 2031, the feature interaction processing is performed on the current detection box feature vector and the historical trajectory feature vector based on a space-time attention mechanism, to obtain the first context feature vector corresponding to the current detection box feature vector and the second context feature vector corresponding to the historical trajectory feature vector; wherein the space-time attention mechanism is used to model the relative position relationship between different obstacles at different times through relative position coding.

[0073] In the embodiments of the present application, the representation of the context information is obtained through a space-time attention mechanism, and the space-time attention models the relative position relationship between different objects at different time through relative position encoding. Through the relative position encoding, the coordinate relationship of the tracking and prediction tasks remains consistent in different coordinate systems, thereby realizing unified feature expression in the vehicle body coordinate system and meeting the translational and rotational invariance required by the prediction task.

[0074] In some embodiments provided in the present application, the above step 202 can include the following steps: step 2021, step 2022, step 2023 and step 2024.

[0075] In step 2021, a third context feature vector associated with the first tracking trajectory is obtained from a memory module of the tracking and prediction model, wherein the memory module records historical context feature vectors of the vehicle terminal.

[0076] In the embodiments of the present application, the memory module is used to model long-time sequence data, so that the tracking and prediction model can store historical trajectory information of a certain time length to guide multi-target tracking and trajectory prediction.

[0077] In step 2022, the third context feature vector is converted to the coordinate system at the first time to obtain a feature conversion result.

[0078] In the embodiments of the present application, in order to facilitate processing, the information in the memory module is transferred to the vehicle body coordinate system at the current time through affine transformation; wherein the following formula can be used for affine transformation: wherein α, β are parameters of affine transformation, α, β = MLP(E t -E s ), E t -E s represents the difference between two frames in the vehicle body coordinate system.

[0079] In step 2023, the feature conversion result and the first tracking trajectory are respectively encoded.

[0080] In step 2024, a historical trajectory feature vector corresponding to the first tracking trajectory is determined according to the feature encoding result of the first tracking trajectory and the feature encoding result of the feature conversion result.

[0081] In the embodiments of the present application, the feature encoding result of the first tracking trajectory and the feature encoding result of the feature conversion result can be added to obtain the historical trajectory feature vector corresponding to the first tracking trajectory.

[0082] It can be seen that in the embodiments of the present application, the historical tracking trajectory information before each first tracking trajectory can be obtained, and the historical trajectory feature vector corresponding to the first tracking trajectory is generated by comprehensively combining the two trajectories, thereby ensuring the accuracy of the final tracking result and the prediction result.

[0083] In some embodiments provided in the present application, as shown in FIG. 3, the tracking prediction model 30 can include a first encoding module 31, a second encoding module 32, a third encoding module 33, a first cross attention module 34, a first self attention module 35, a second cross attention module 36, a second self attention module 37, a memory module 38, a tracker 39 and a predictor 310.

[0084] For ease of understanding, the vehicle-mounted multi-target tracking and trajectory prediction method is introduced in combination with the tracking prediction model shown in FIG. 3. The third context feature vector associated with each first tracking trajectory is obtained from the memory module 38 of the tracking prediction model 30, and each third context feature vector is converted to the coordinate system at the first time to obtain a feature conversion result. The historical context feature vector of the vehicle-mounted terminal is recorded in the memory module.

[0085] The feature encoding of each feature conversion result is performed by the first encoding module 31 of the tracking prediction model 30. The first encoding module 31 can include a multi-head cross attention network and a multi-layer perceptron, and the multi-layer perceptron is connected behind the multi-head cross attention network. The multi-head cross attention network is used to fuse the time information of the trajectory.

[0086] The feature encoding of each first tracking trajectory is performed by the second encoding module 32 of the tracking prediction model 30. The network structure of the second encoding module 32 is the same as that of the first encoding module 31. The encoding results output by the first encoding module 31 and the second encoding module 32 are added to obtain the historical trajectory feature vector.

[0087] The feature encoding of each first detection box is performed by the third encoding module 33 of the tracking prediction model 30 to obtain the current detection box feature vector corresponding to each first detection box. The third encoding module 33 can include a multi-layer perceptron. For example, there are M first detection boxes, and the feature encoding of the M detection boxes is performed.

[0088] The cross attention processing of the current detection box feature vector and the historical trajectory feature vector is performed by the first cross attention module 34 of the tracking prediction model 30, and the self attention processing of the feature vector output by the first cross attention module 34 is performed by the first self attention module 35 of the tracking prediction model 30. After three rounds of cross attention processing and self attention processing by the first cross attention module 34 and the first self attention module 35, the first context feature vector corresponding to the current detection box feature vector is obtained.

[0089] The second cross-attention module 36 of the tracking prediction model 30 is used to perform cross-attention processing on the historical trajectory feature vector and the current detection box feature vector, and the second self-attention module 37 of the tracking prediction model 30 is used to perform self-attention processing on the feature vector output by the second cross-attention module 36. After the three rounds of cross-attention processing and self-attention processing of the second cross-attention module 36 and the second self-attention module 37, a second context feature vector corresponding to the historical trajectory feature vector is obtained.

[0090] In the processing process of the first cross-attention module 34, the first self-attention module 35, the second cross-attention module 36 and the second self-attention module 37, the feature vector to be processed is decomposed into a coordinate-dependent intermediate feature vector and a coordinate-independent intermediate feature vector, the coordinate-dependent intermediate feature vector is subjected to relative position coding processing, and the relative position coding processing result and the coordinate-independent intermediate feature vector are added together and participate in attention operation. The network parameter values of the first cross-attention module 34 and the second cross-attention module 36 are different, and the network parameter values of the first self-attention module 35 and the second self-attention module 37 are different.

[0091] In the embodiments of the present application, the feature is decomposed into coordinate-dependent feature and coordinate-independent feature, and the absolute coordinate is replaced by the relative coordinate, so that the dependence of the coordinate on a specific coordinate system is eliminated.

[0092] The second context feature vector output by the second self-attention module 37 is saved in the memory module 38.

[0093] In the embodiments of the present application, the memory module is added in the tracking prediction model, so that the model can obtain long-time modeling capability.

[0094] The tracker 39 of the tracking prediction model 30 is used to calculate the similarity between each first context feature vector and each second context feature vector, and based on the calculated similarities, the matching relationship between the first detection box and the first tracking trajectory is determined. Preferably, the optimal transport processing is performed on the calculated similarities to obtain a similarity matrix, and based on the similarity matrix, the matching relationship between the first detection box and the first tracking trajectory is determined, wherein the element value in the similarity matrix is positively correlated with the matching degree.

[0095] Exemplarily, the similarity matrix is obtained through similarity measurement and optimal transport optimization, the maximum value of each row in the similarity matrix is set to 1 and the other values are set to 0, and a matching relationship matrix is obtained. The value "1" in the matching relationship matrix can directly represent the matching relationship between each first detection box and the first tracking trajectory.

[0096] The first upper context feature vector, the second upper context feature vector, and the track template are processed by a predictor 310 of the prediction model 30 to obtain a motion track of the obstacle in the first detection box within a second duration after the first time; the predictor includes a first decoder and a second decoder, the first decoder is configured to decode the second upper context feature vector to obtain a prediction result of a first tracking track, and the second decoder is configured to fuse the first upper context feature vector, the second upper context feature vector, the track template, and the prediction result of the first tracking track, and decode the fusion result in the time dimension and the space dimension to obtain the motion track of the obstacle in the first detection box within the second duration after the first time.

[0097] In the embodiment of the present application, under the streaming framework, the predictor can perform double-branch prediction on the historical track and the current detection box at the same time, and use the prediction results between adjacent frames which have a large amount of overlap as prior information of the prediction result of the current frame to further improve the accuracy and time consistency of the predicted track.

[0098] It can be seen that in the embodiment of the present application, a streaming and coordinate unified training framework is proposed, which can solve the inconsistency between training and testing, the inconsistency of coordinate systems, and the lack of ability to handle errors of upstream modules.

[0099] In some embodiments provided by the present application, the step 101 can include the following steps: a step 1011, a step 1012, a step 1013, and a step 1014.

[0100] In step 1011, a point cloud video of the environment around the vehicle is collected by a laser radar on the vehicle side.

[0101] In the embodiment of the present application, the number of laser radars on the vehicle side can be one or multiple, and when the number of laser radars is multiple, there is an overlap between the fields of view of two adjacent laser radars.

[0102] In step 1012, the point cloud video is input into a detector to obtain a detection result of each frame, wherein the detection result includes the position, size, orientation angle, category, and detection confidence of the obstacle.

[0103] In the embodiment of the present application, the detector is a pre-trained model, which can detect the position, size, orientation angle, category, and detection confidence of the obstacle in the video frame.

[0104] In step 1013, a first detection box of each obstacle in the video frame of the first time to be predicted is determined according to the detection result of the last frame of the point cloud video.

[0105] In the embodiments of the present application, the last frame of the point cloud video in the reasoning stage refers to the video frame at the first time, i.e., the current frame.

[0106] In step 1014, the first tracking trajectory of each obstacle in each video frame within the first time length before the first time is determined according to the detection results of the frames other than the last frame of the point cloud video.

[0107] In the embodiments of the present application, the frames other than the last frame of the point cloud video refer to the video frames within the first time length before the first time.

[0108] In the embodiments of the present application, the first tracking trajectory of each obstacle in each video frame within the first time length before the first time can be determined according to the detection results of the frames other than the last frame of the point cloud video, the bounding boxes of the same category are associated together, and the first tracking trajectory of each obstacle in each video frame within the first time length before the first time is obtained.

[0109] It can be seen that in the embodiments of the present application, the point cloud video stream collected by the laser radar on the vehicle side can be detected in real time according to the pre-trained detector to obtain the first detection box of each obstacle in the video frame at the first time to be predicted and the first tracking trajectory of each obstacle in each video frame within the first time length before the first time, so that more accurate and smoother multi-target tracking and trajectory prediction are realized.

[0110] It should be noted that for the video stream collected by the vehicle camera, the above steps 1011-1014 can also be used for processing, which will not be described here.

[0111] FIG. 4 is a structural schematic diagram of a vehicle-side multi-target tracking and trajectory prediction device provided by the embodiments of the present application. As shown in FIG. 4, the vehicle-side multi-target tracking and trajectory prediction device 400 can include an acquisition module 401 and a processing module 402.

[0112] The acquisition module 401 is configured to acquire the first detection box of each obstacle in the video frame at the first time to be predicted and the first tracking trajectory of each obstacle in each video frame within the first time length before the first time.

[0113] The processing module 402 is configured to input the first detection box and the first tracking trajectory into a tracking prediction model, process the first detection box and the first tracking trajectory through the tracking prediction model, obtain the matching relationship between the first detection box and the first tracking trajectory, and the motion trajectory of the obstacle in the first detection box within the second time length after the first time.

[0114] The tracking prediction model is trained based on a second detection box of each obstacle in each frame of a sample video, a second tracking trajectory of each obstacle in a video frame within a first time length before a time at which the each frame is located, a real matching relationship between the second detection box and the second tracking trajectory, and a real motion trajectory of the obstacle within the second detection box within a second time length after the time at which the each frame is located.

[0115] As can be seen from the above embodiment, in the embodiment, a first detection box of each obstacle in a video frame at a first time to be predicted and a first tracking trajectory of each obstacle in each video frame within a first time length before the first time are obtained; the first detection box and the first tracking trajectory are input into a tracking prediction model, the tracking prediction model is used to process the first detection box and the first tracking trajectory, a matching relationship between the first detection box and the first tracking trajectory and a motion trajectory of the obstacle within the first detection box within a second time length after the first time are obtained; and the tracking prediction model is trained based on a second detection box of each obstacle in each frame of a sample video, a second tracking trajectory of each obstacle in a video frame within a first time length before a time at which the each frame is located, a real matching relationship between the second detection box and the second tracking trajectory, and a real motion trajectory of the obstacle within the second detection box within a second time length after the time at which the each frame is located.

[0116] As can be seen, in the embodiment of the application, considering the streaming scenario when the vehicle-mounted end is actually deployed, the training manner of the video stream can be used to combine the two tasks of multi-target tracking and trajectory prediction, train the tracking prediction model, and perform multi-target tracking and trajectory prediction of the vehicle-mounted end through the tracking prediction model. Since the above tracking prediction model is adapted to the streaming scenario when the vehicle-mounted end is actually deployed, and can correct the occlusion and missed detection results of the upstream module, the accuracy of the multi-target tracking result and the trajectory prediction result of the vehicle-mounted end can be improved.

[0117] Any one of the steps in the embodiment of the vehicle-mounted end multi-target tracking and trajectory prediction method provided in the embodiment of the application and the specific operations in any one of the steps can be completed by the corresponding modules in the vehicle-mounted end multi-target tracking and trajectory prediction device. The processes of the corresponding operations completed by the modules in the vehicle-mounted end multi-target tracking and trajectory prediction device are referred to the processes of the corresponding operations described in the embodiment of the vehicle-mounted end multi-target tracking and trajectory prediction method.

[0118] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts are referred to the part of the method embodiment.

[0119] FIG. 5 is a structural block diagram of an electronic device according to an embodiment of the present application. The electronic device includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532, for storing instructions, such as application programs, executable by the processing component 522. The application programs stored in the memory 532 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute the instructions to perform the in-vehicle multi-target tracking and trajectory prediction method described above.

[0120] The electronic device can further include a power supply component 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input / output (I / O) interface 558. The electronic device can operate based on an operating system stored in the memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0121] According to still another embodiment of the present application, the present application further provides a computer readable storage medium having stored thereon computer programs / instructions, which, when executed by a processor, implement the steps of the in-vehicle multi-target tracking and trajectory prediction method according to any one of the above embodiments.

[0122] According to still another embodiment of the present application, the present application further provides a computer program product including computer programs / instructions, which, when executed by a processor, implement the steps of the in-vehicle multi-target tracking and trajectory prediction method according to any one of the above embodiments.

[0123] Each of the above-described embodiments of the present application is described in progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts among the embodiments can be mutually referred to.

[0124] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device, or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0125] The embodiments of the present application are described with reference to the flowchart illustrations and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing unit or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0126] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function specified in the flowchart illustrations and / or block diagrams block or blocks.

[0127] Although the preferred embodiments of the present application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments once they have been given the benefit of the basic inventive concepts presented with respect to the preferred embodiments. Accordingly, the appended claims are intended to encompass all modifications and variations of the preferred embodiments that fall within the scope of the present application.

[0128] Finally, it should be noted that, in the description above, relational terms such as first and second, and the like, can be used solely to distinguish one from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0129] The above provides a kind of vehicle-mounted end multi-target tracking and trajectory prediction method, electronic equipment and storage medium provided in the application, are introduced in detail, the principle and implementation mode of the present application are described in this paper, the above example is only used to help understand the method and its core idea of the present application;At the same time, for those skilled in the art, according to the idea of the present application, there will be changes in specific implementation mode and application range, as described above, the content of the specification should not be understood as limiting the present application.

Claims

1. A method for multi-target tracking and trajectory prediction on a vehicle-mounted terminal, wherein, The method includes: Obtain the first detection box of each obstacle in the video frame at the first moment to be predicted and the first tracking trajectory of each obstacle in each video frame within the first duration before the first moment; The first detection box and the first tracking trajectory are input into the tracking prediction model. The tracking prediction model processes the first detection box and the first tracking trajectory to obtain the matching relationship between the first detection box and the first tracking trajectory, as well as the movement trajectory of the obstacle in the first detection box within the second time period after the first moment. The tracking prediction model is trained based on the second detection box of each obstacle in each frame of the sample video, the second tracking trajectory of each obstacle in the video frames within a first duration before the time of each frame, the true matching relationship between the second detection box and the second tracking trajectory, and the true motion trajectory of the obstacle in the second detection box within a second duration after the time of each frame.

2. The method according to claim 1, wherein, The step of processing the first detection box and the first tracking trajectory using the tracking prediction model to obtain the matching relationship between the first detection box and the first tracking trajectory, and the motion trajectory of the obstacle within the first detection box during a second time period after the first moment, includes: The tracking prediction model generates the current detection box feature vector corresponding to the first detection box. Generate the historical trajectory feature vector corresponding to the first tracking trajectory; The current detection box feature vector and the historical trajectory feature vector are subjected to feature interaction processing to obtain a first context feature vector corresponding to the current detection box feature vector and a second context feature vector corresponding to the historical trajectory feature vector. Based on the first top and bottom feature vectors and the second context feature vector, the matching relationship between the first detection box and the first tracking trajectory is determined; Based on the first top and bottom feature vectors, the second context feature vector, and the trajectory template, the motion trajectory of the obstacle within the first detection box is determined within a second time period after the first moment, wherein the trajectory template includes multiple candidate motion trajectories.

3. The method according to claim 2, wherein, The step of performing feature interaction processing on the current detection box feature vector and the historical trajectory feature vector to obtain a first context feature vector corresponding to the current detection box feature vector and a second context feature vector corresponding to the historical trajectory feature vector includes: Based on the spatiotemporal attention mechanism, feature interaction processing is performed on the current detection box feature vector and the historical trajectory feature vector to obtain the first context feature vector corresponding to the current detection box feature vector and the second context feature vector corresponding to the historical trajectory feature vector. The spatiotemporal attention mechanism is used to model the relative positional relationships between different obstacles at different times through relative position encoding.

4. The method according to claim 3, wherein, The spatiotemporal attention mechanism involves performing feature interaction processing on the current detection box feature vector and the historical trajectory feature vector to obtain a first context feature vector corresponding to the current detection box feature vector and a second context feature vector corresponding to the historical trajectory feature vector, including: The tracking prediction model performs cross-attention processing on the current detection box feature vector and the historical trajectory feature vector through the first cross-attention module, and performs self-attention processing on the feature vector output by the first cross-attention module through the first self-attention module. After three rounds of cross-attention processing and self-attention processing by the first cross-attention module and the first self-attention module, the first context feature vector corresponding to the current detection box feature vector is obtained. The tracking prediction model performs cross-attention processing on the historical trajectory feature vector and the current detection box feature vector through the second cross-attention module, and performs self-attention processing on the feature vector output by the second cross-attention module through the second self-attention module. After three rounds of cross-attention processing and self-attention processing by the second cross-attention module and the second self-attention module, the second context feature vector corresponding to the historical trajectory feature vector is obtained. In this process, the first cross-attention module, the first self-attention module, the second cross-attention module, and the second self-attention module all decompose the feature vector to be processed into coordinate-dependent intermediate feature vectors and coordinate-independent intermediate feature vectors. The coordinate-dependent intermediate feature vectors are then subjected to relative position encoding. The result of the relative position encoding is added to the coordinate-independent intermediate feature vectors, and both are then included in the attention operation. The network parameter values ​​of the first cross-attention module and the second cross-attention module are different, as are the network parameter values ​​of the first self-attention module and the second self-attention module.

5. The method according to claim 2, wherein, The generation of the historical trajectory feature vector corresponding to the first tracking trajectory includes: The third context feature vector associated with the first tracking trajectory is obtained from the memory module of the tracking prediction model, wherein the memory module records the historical context feature vector of the vehicle terminal. The third context feature vector is transformed into the coordinate system of the first time step to obtain the feature transformation result; The feature conversion result and the first tracking trajectory are respectively feature encoded; Based on the feature encoding result of the first tracking trajectory and the feature encoding result of the feature transformation result, the historical trajectory feature vector corresponding to the first tracking trajectory is determined.

6. The method according to claim 5, wherein, After performing feature interaction processing on the current detection box feature vector and the historical trajectory feature vector to obtain the first context feature vector corresponding to the current detection box feature vector and the second context feature vector corresponding to the historical trajectory feature vector, the method further includes: The second context feature vector is saved to the memory module.

7. The method according to claim 5, wherein, The step of performing feature encoding on the feature transformation result and the first tracking trajectory respectively includes: The feature transformation result is feature-encoded by the first encoding module of the tracking prediction model, wherein the first encoding module includes a multi-head cross-attention network and a multilayer perceptron, and the multilayer perceptron is connected after the multi-head cross-attention network; The first tracking trajectory is feature-encoded using the second encoding module of the tracking prediction model; The second encoding module has the same network structure as the first encoding module.

8. The method according to claim 2, wherein, Generating the feature vector of the current detection box corresponding to the first detection box includes: The first detection box is feature-encoded by the third encoding module of the tracking prediction model to obtain the current detection box feature vector corresponding to the first detection box; wherein, the third encoding module includes a multilayer perceptron.

9. The method according to claim 2, wherein, Determining the matching relationship between the first detection box and the first tracking trajectory based on the first upper and lower feature vectors and the second context feature vector includes: The tracker of the tracking prediction model calculates the similarity between each of the first upper and lower feature vectors and each of the second context feature vectors; Based on the calculated similarities, the matching relationship between the first detection box and the first tracking trajectory is determined.

10. The method according to claim 9, wherein, Determining the matching relationship between the first detection box and the first tracking trajectory based on the calculated similarities includes: The calculated similarities are then subjected to optimal transmission processing to obtain a similarity matrix; Based on the similarity matrix, the matching relationship between the first detection box and the first tracking trajectory is determined, wherein the magnitude of the element value in the similarity matrix is ​​positively correlated with the matching degree.

11. The method according to claim 2, wherein, The step of determining the motion trajectory of the obstacle within the first detection frame during a second time period after the first moment, based on the first upper and lower feature vectors, the second context feature vector, and the trajectory template, includes: The predictor of the tracking prediction model processes the first up-down feature vector, the second context feature vector, and the trajectory template to obtain the motion trajectory of the obstacle in the first detection box within a second time period after the first moment. The predictor includes a first decoder and a second decoder. The first decoder is used to perform feature decoding on the second context feature vector to obtain the prediction result of the first tracking trajectory. The second decoder is used to fuse the first upper and lower feature vectors, the second context feature vector, the trajectory template and the prediction result of the first tracking trajectory, and to perform time and space dimension decoding on the fusion result to obtain the motion trajectory of the obstacle in the first detection box within a second time period after the first moment.

12. The method according to claim 1, wherein, The process of obtaining the first detection box of each obstacle in the video frame at the first moment to be predicted and the first tracking trajectory of each obstacle in each video frame within the first time period before the first moment includes: The vehicle-mounted LiDAR collects point cloud video of the environment around the vehicle. The point cloud video is input into the detector to obtain the detection results for each frame, wherein the detection results include: the position, size, orientation angle, category, and detection confidence of the obstacle; Based on the detection results of the last frame of the point cloud video, determine the first detection box of each obstacle in the video frame of the first moment to be predicted; Based on the detection results of frames other than the last frame of the point cloud video, determine the first tracking trajectory of each obstacle in each video frame within the first duration before the first moment.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein, The processor executes the computer program to implement the vehicle-mounted multi-target tracking and trajectory prediction method according to any one of claims 1-12.

14. A computer-readable storage medium having a computer program / instructions stored thereon, wherein, When the computer program / instruction is executed by the processor, it implements the vehicle-mounted multi-target tracking and trajectory prediction method according to any one of claims 1-12.

15. A computer program product comprising a computer program / instructions, wherein, When the computer program / instruction is executed by the processor, it implements the vehicle-mounted multi-target tracking and trajectory prediction method according to any one of claims 1-12.

Images