Target detection method, device and electronic equipment
Through the combination of deep learning models and space-time attention networks, the problem of poor target detection and tracking effect in complex scenarios is solved, better target recognition and correlation are achieved, and detection effect and generalization are improved.
Patent Information
- Application Number
- CN202411409221.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2044-10-10
AI Technical Summary
The existing millimeter-wave radar target detection and tracking methods are poor in complex scenarios and have poor generalization. Due to the low resolution and high noise of point cloud data, it is difficult to correlate the target of the front and back frame data, resulting in poor detection results.
The deep learning model is used to combine the spatiotemporal attention network and the orbital information inference network, and by acquiring and fusion of target query features, inter- and intra-frame attention enhancement is achieved, object detection and trajectory prediction are improved, and the effect of object detection and tracking is improved.
It improves the effect of object detection and tracking, enhances the recognition and association ability of the target in complex scenarios, improves generalization, and improves the feature stability of the front and back frame data.
Smart Images
Figure CN118918579B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target perception technology, and in particular to a target detection method, device and electronic equipment. Background Art
[0002] Millimeter-wave radar is a distance and speed measurement sensor that can generate point clouds at the edge of obstacles. At present, the use of millimeter-wave radar for trajectory generation generally adopts point cloud clustering + target tracking, but this method requires manual design of matching rules and debugging of related parameters. It performs poorly in the face of complex scenes and has poor generalization. Another integrated point cloud detection and tracking method based on deep learning, in which the model includes three outputs: the position of the target when it first appears in multiple point clouds; the motion offset of the target in multiple point clouds, which is used to update the target position; other attributes of the target, such as size and orientation. However, this method lacks feature interaction between target detection and target tracking. Moreover, due to the low resolution and high noise of millimeter-wave point cloud data, the point cloud features of the millimeter-wave point cloud data of the previous and next frames are prone to fluctuations, which also makes it difficult to associate the targets in the millimeter-wave point cloud data of the previous and next frames, and the target detection and tracking effects are not good. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a target detection method, device and electronic device to improve the effect of target detection and tracking. The specific technical solution is as follows:
[0004] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising:
[0005] Acquire a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and acquire the tth frame of millimeter wave point cloud data, wherein t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with a predicted position of a target in the tth frame of millimeter wave point cloud data;
[0006] Inputting the t-th frame of millimeter wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature;
[0007] Inputting the first encoded point cloud feature and the first target point cloud feature into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature;
[0008] Acquire a historical frame target query feature, and fuse the first decoded point cloud feature with the historical frame target query feature in time sequence to obtain a target query feature sequence, wherein the historical frame target query feature includes the target query feature of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data;
[0009] Inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature;
[0010] The first target query feature and the target query feature sequence are input into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction to obtain target detection results and target trajectory prediction results.
[0011] In a possible implementation, the step of inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain the first target query feature includes:
[0012] Inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model, and using the inter-frame attention mechanism module of the spatiotemporal attention network to perform inter-frame attention enhancement on the first decoded point cloud feature to obtain a first updated point cloud feature;
[0013] The intra-frame attention mechanism module of the spatiotemporal attention network is used to perform intra-frame attention enhancement on the first updated point cloud feature to obtain a first target query feature.
[0014] In a possible implementation, the inter-frame attention mechanism module of the spatiotemporal attention network is used to perform inter-frame attention enhancement on the first decoded point cloud feature to obtain a first updated point cloud feature, including:
[0015] The target query feature sequence is processed as follows using the inter-frame attention mechanism module of the spatiotemporal attention network: , get the first updated point cloud feature;
[0016] in, represents the inter-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. represents the feature of the i-th target in the first decoded point cloud feature, tk: k represents the k+1 frame of millimeter wave point cloud data from tk frame to t frame, represents the feature of the i-th target in the k+1 frame target query feature, represents the position embedding of k+1 frames of millimeter wave point cloud data, Represents the first updated point cloud feature.
[0017] In a possible implementation, the intra-frame attention mechanism module of the spatiotemporal attention network is used to perform intra-frame attention enhancement on the first updated point cloud feature to obtain the first target query feature, including:
[0018] The first updated point cloud feature is processed as follows using the intra-frame attention mechanism module of the spatiotemporal attention network:
[0019] , get the first target query feature;
[0020] in, represents the intra-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. From the first target to the target goals, Indicates the first updated point cloud feature The characteristics of the target, Indicates the first updated point cloud feature The location of the target, Indicates the first updated point cloud feature The position embedding of the target, Represents the first target query feature.
[0021] In a possible implementation, the step of inputting the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction to obtain a target detection result and a target trajectory prediction result includes:
[0022] Inputting the first target query feature into the track information inference network of the deep learning model, and using the target information prediction module of the track information inference network to predict the target update attribute of the first target query feature; using the predicted target update attribute information to update the position of the target in the t-th frame of millimeter wave point cloud data, and obtaining the target detection result of the t-th frame of millimeter wave point cloud data;
[0023] The motion trajectory prediction module of the track information inference network is used to predict the intermediate motion data of the target query feature sequence and the first target query feature, the motion trajectory prediction module of the track information inference network is used to decode the intermediate motion data to obtain decoded motion data, and the motion trajectory prediction module of the track information inference network is used to predict the trajectory of the target in the t+x-th frame millimeter wave point cloud data according to the decoded motion data and the target detection result to obtain the target trajectory prediction result of the t+x-th frame millimeter wave point cloud data.
[0024] In a possible implementation, the target information prediction module using the track information inference network predicts the target update attribute of the first target query feature; including:
[0025] The target information prediction module of the track information inference network is used to process the first target query feature as follows:
[0026] , predict the target update attributes;
[0027] in, is a multi-layer perceptron, represents the feature of the i-th target in the first target query feature, represents the displacement of the i-th target on the x-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the y-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the z-axis in the t-th frame of millimeter-wave point cloud data, represents the length of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the width of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the height of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the orientation angle of the i-th target in the t-th frame of millimeter wave point cloud data, represents the speed of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the confidence of the target update attribute;
[0028] The predicted target update attribute information is used to update the position of the target in the t-th frame of millimeter wave point cloud data, and the target detection result of the t-th frame of millimeter wave point cloud data is obtained, including:
[0029] The predicted target update attribute information is used to process the position of the target in the t-th frame of millimeter wave point cloud data as follows:
[0030] , get the target detection result of the t-th frame millimeter wave point cloud data;
[0031] in, represents the x-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the y-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the z-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the target detection result of the t-th frame of millimeter wave point cloud data.
[0032] In a possible implementation, the motion trajectory prediction module of the track information inference network is used to predict intermediate motion data of the target query feature sequence and the first target query feature, the motion trajectory prediction module of the track information inference network is used to decode the intermediate motion data to obtain decoded motion data, and the motion trajectory prediction module of the track information inference network is used to predict the trajectory of the target in the t+x-th frame millimeter wave point cloud data according to the decoded motion data and the target detection result to obtain the target trajectory prediction result of the t+x-th frame millimeter wave point cloud data, including:
[0033] The motion trajectory prediction module of the track information inference network is used to process the target query feature sequence and the first target query feature as follows:
[0034] , to predict the intermediate motion data;
[0035] in, represents the inter-frame attention mechanism, Q, K, V represent the three feature vectors of Query, Key, and Value respectively, PE represents the position embedding, and X represents 1, 2, 3, etc. Represents the time from frame t to frame Frame contains Frame millimeter wave point cloud data, Initialized to 0, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame. represents the features of the i-th target in the target query features of the previous k frames and the first target query features, Indicates the k+x+1 frames of millimeter wave point cloud data from tk frame to t+k frame. represents the position embedding of k+x+1 frames of millimeter wave point cloud data, Represents intermediate motion data;
[0036] The motion trajectory prediction module of the track information inference network is used to process the intermediate motion data as follows:
[0037] Get decoded motion data, is a multi-layer perceptron, Decode motion data;
[0038] The motion trajectory prediction module using the track information inference network performs the following processing based on the decoded motion data and the target detection result:
[0039] , predict the trajectory of the target in the t+x-th frame of millimeter wave point cloud data, and obtain the target trajectory prediction result of the t+x-th frame of millimeter wave point cloud data;
[0040] in, represents the target detection result of the millimeter wave point cloud data of the tth frame, Represents the target trajectory prediction result of the t+xth frame of millimeter wave point cloud data.
[0041] In a possible implementation, the method further includes:
[0042] The first target query feature is labeled according to the target trajectory prediction result to obtain the target point cloud feature of the t-th frame of millimeter wave point cloud data.
[0043] In a second aspect, an embodiment of the present application provides a target detection device, the device comprising:
[0044] An acquisition module, used to acquire a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and to acquire the tth frame of millimeter wave point cloud data, wherein t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with a predicted position of a target in the tth frame of millimeter wave point cloud data;
[0045] An encoding module, used for inputting the t-th frame of millimeter wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature;
[0046] A decoding module, used for inputting the first encoded point cloud feature and the first target point cloud feature into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature;
[0047] A fusion module is used to obtain a historical frame target query feature, and fuse the first decoded point cloud feature with the historical frame target query feature in time sequence to obtain a target query feature sequence, wherein the historical frame target query feature includes the target query feature of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data;
[0048] A feature enhancement module, used for inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature;
[0049] A prediction module is used to input the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction, so as to obtain target detection results and target trajectory prediction results.
[0050] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0051] Memory, used to store computer programs;
[0052] The processor is used to implement any of the above-mentioned target detection methods when executing the program stored in the memory.
[0053] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the target detection methods described above.
[0054] In an embodiment of the present application, a first target point cloud feature of the t-1th frame of millimeter wave point cloud data is obtained, and the tth frame of millimeter wave point cloud data is obtained, the first target point cloud feature being a target query feature marked with the predicted position of the target in the tth frame of millimeter wave point cloud data; the tth frame of millimeter wave point cloud data is input into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature; the first encoded point cloud feature and the first target point cloud feature are input into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature; the historical frame target query feature is obtained, and the first decoded point cloud feature is fused with the historical frame target query feature in time sequence to obtain a target query feature sequence; the target query feature sequence is input into the spatiotemporal attention network of the deep learning model for feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature; the first target query feature and the target query feature sequence are input into the track information inference network of the deep learning model for target prediction and motion trajectory prediction to obtain a target detection result and a target trajectory prediction result. There is a correlation between the target detection process and the target trajectory prediction process. The target trajectory prediction uses the first target query feature, and the target trajectory prediction is used as the input of the target detection result of the next frame of millimeter-wave point cloud data, which improves the effect of target detection and tracking.
[0055] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0057] Figure 1 A first flow chart of the target detection method provided in the embodiment of the present application;
[0058] Figure 2 A second flow chart of the target detection method provided in the embodiment of the present application;
[0059] Figure 3 A third flow chart of the target detection method provided in the embodiment of the present application;
[0060] Figure 4 A schematic diagram of the structure of a target detection device provided in an embodiment of the present application;
[0061] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0063] Millimeter-wave radar is a distance and speed measurement sensor that can generate point clouds at the edge of obstacles. At present, the use of millimeter-wave radar for trajectory generation generally adopts point cloud clustering + target tracking, but this method requires manual design of matching rules and debugging of related parameters. It performs poorly in the face of complex scenes and has poor generalization. Another integrated point cloud detection and tracking method based on deep learning, in which the model includes three outputs: the position of the target when it first appears in multiple point clouds; the motion offset of the target in multiple point clouds, which is used to update the target position; other attributes of the target, such as size and orientation. However, this method lacks feature interaction between target detection and target tracking. Moreover, due to the low resolution and high noise of millimeter-wave point cloud data, the point cloud features of the millimeter-wave point cloud data of the previous and next frames are prone to fluctuations, which also makes it difficult to associate the targets in the millimeter-wave point cloud data of the previous and next frames.
[0064] To this end, the first aspect of the present application provides a target detection method, see Figure 1 , the method comprising:
[0065] S101, obtaining a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and obtaining the tth frame of millimeter wave point cloud data.
[0066] Wherein, t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with the predicted position of the target in the t-th frame of millimeter wave point cloud data. The first target point cloud feature also includes the features of each target in the t-1th frame of millimeter wave point cloud data. The t-th frame of millimeter wave point cloud data is represented as , Indicates time. Different frames of millimeter-wave point cloud data correspond to different times. Therefore, multiple frames of millimeter-wave point cloud data can be distinguished by time t. represents the three-dimensional coordinates of the i-th target in the t-th frame of millimeter wave point cloud data, represents the Doppler velocity of the i-th target. The first target point cloud feature is expressed as , represents the characteristics of the i-th target in the t-1-th frame of millimeter wave point cloud data, represents the predicted position of the i-th target in the t-th frame of millimeter wave point cloud data. This feature design naturally solves the tracking task because the feature will carry the ID information of the target over time (i.e., the i mentioned above).
[0067] S102. Input the t-th frame of millimeter-wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature.
[0068] The encoder can be a point cloud feature extraction structure based on Pillar, Voxel, and Transformer. Pillar is a point cloud processing method used to generate 2D BEV features, which refer to bird's-eye view features; Voxel is a point cloud processing method used to generate 3D voxel features; Transformer is an attention mechanism network.
[0069] S103: Input the first encoded point cloud feature and the first target point cloud feature into the decoder of the deep learning model for decoding to obtain a first decoded point cloud feature.
[0070] The decoder can be an attention-based Transformer network. , for decoding. represents the first coded point cloud feature, represents the first target point cloud feature, represents the target box of each target in the t-th frame of millimeter wave point cloud data, Represents the first decoded point cloud feature, including the position and features of the target in the t-th frame of millimeter wave point cloud data. During the decoding process, the Decoder sets the target position in the first target point cloud feature as the position code, thereby focusing on the bird's-eye view area related to the target spatial position.
[0071] In addition to the first target point cloud features, the present application can also add a fixed number of target query features, including the location of the target and the features of the target, for discovering new targets. After initializing the fixed number of target query features as learnable embedded target query features, they are input into the decoder together with the first encoded point cloud features and the first target point cloud features. For example, if 10 targets have been detected in the t-1 frame, and the total number of targets that the model can detect is certain, such as 15, then the fixed number is 15-10=5. The first target point cloud features of the 10 detected targets and the 5 learnable embedded target query features are input into the decoder, and the output can be the first decoded point cloud features of the 12 detected targets, and the extra 2 detected targets are decoded by the 5 learnable embedded target query features.
[0072] S104: Acquire historical frame target query features, and fuse the first decoded point cloud features with the historical frame target query features in time sequence to obtain a target query feature sequence.
[0073] The target query features of the historical frames include the target query features of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data, including the position of the target and the features of the target. The fused target query feature sequence includes in time sequence: the target query features of the tk-th frame of millimeter wave point cloud data, the target query features of the t-k+1-th frame of millimeter wave point cloud data, the target query features of the t-k+2-th frame of millimeter wave point cloud data... the first decoded point cloud features.
[0074] S105. Input the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature.
[0075] Step 1: Input the target query feature sequence into the spatiotemporal attention network of the deep learning model, and use the inter-frame attention mechanism module of the spatiotemporal attention network to perform inter-frame attention enhancement on the first decoded point cloud feature to obtain a first updated point cloud feature.
[0076] The spatiotemporal attention network includes an inter-frame attention mechanism module and an intra-frame attention mechanism module. The inter-frame attention mechanism module starts from the inter-frame relationship and performs feature enhancement on the first decoded point cloud features; the intra-frame attention mechanism module starts from each target in the frame and performs feature enhancement on the first decoded point cloud features.
[0077] For example, if t is 5 and k is 2, the inter-frame attention feature enhancement is first performed on the 3rd frame millimeter-wave point cloud data, the 4th frame millimeter-wave point cloud data, and the 5th frame millimeter-wave point cloud data. The target query features of these three frames of millimeter-wave point cloud data can be weighted summed, and the result of the weighted summation is used as the first updated point cloud feature; or the target query features of the first two frames of millimeter-wave point cloud data can be weighted summed, and the similarity between this result and the first decoded point cloud feature is calculated, and the point cloud feature with a similarity greater than a threshold is taken to obtain the first updated point cloud feature.
[0078] Step 2: Use the intra-frame attention mechanism module of the spatiotemporal attention network to perform intra-frame attention enhancement on the first updated point cloud feature to obtain a first target query feature.
[0079] The intra-frame attention mechanism module focuses on the features of each target in the frame, which can enhance the features of each target in the frame and obtain the first target query feature.
[0080] S106. Input the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction to obtain a target detection result and a target trajectory prediction result.
[0081] The track information inference network can obtain target detection results and target trajectory prediction results based on the first target query feature and the target query feature sequence.
[0082] In an embodiment of the present application, a first target point cloud feature of the t-1th frame of millimeter wave point cloud data is obtained, and the tth frame of millimeter wave point cloud data is obtained, the first target point cloud feature being a target query feature marked with the predicted position of the target in the tth frame of millimeter wave point cloud data; the tth frame of millimeter wave point cloud data is input into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature; the first encoded point cloud feature and the first target point cloud feature are input into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature; the historical frame target query feature is obtained, and the first decoded point cloud feature is fused with the historical frame target query feature in time sequence to obtain a target query feature sequence; the target query feature sequence is input into the spatiotemporal attention network of the deep learning model for feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature; the first target query feature and the target query feature sequence are input into the track information inference network of the deep learning model for target prediction and motion trajectory prediction to obtain a target detection result and a target trajectory prediction result. There is a correlation between the target detection process and the target trajectory prediction process. The target trajectory prediction uses the first target query feature, and the target trajectory prediction is used as the input of the target detection result of the next frame of millimeter-wave point cloud data, which improves the effect of target detection and tracking.
[0083] In one example, the above step 1 includes:
[0084] The target query feature sequence is processed as follows using the inter-frame attention mechanism module of the spatiotemporal attention network: , get the first updated point cloud feature;
[0085] in, represents the inter-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. represents the feature of the i-th target in the first decoded point cloud feature, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame, represents the feature of the i-th target in the k+1 frame target query feature, represents the position embedding of k+1 frames of millimeter wave point cloud data, Represents the first updated point cloud feature.
[0086] In an embodiment of the present application, the first decoded point cloud features of the tth frame are enhanced by performing an inter-frame attention mechanism on multiple frames of millimeter-wave point cloud data.
[0087] In one example, step 2 above includes:
[0088] The first updated point cloud feature is processed as follows using the intra-frame attention mechanism module of the spatiotemporal attention network:
[0089] , get the first target query feature;
[0090] in, represents the intra-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. From the first target to the target goals, Indicates the first updated point cloud feature The characteristics of the target, Indicates the first updated point cloud feature The location of the target, Indicates the first updated point cloud feature The position embedding of the target, Represents the first target query feature.
[0091] In an example, the above step S106 may specifically include:
[0092] Step one: input the first target query feature into the track information inference network of the deep learning model, and use the target information prediction module of the track information inference network to predict the target update attribute of the first target query feature; use the predicted target update attribute information to update the position of the target in the t-th frame millimeter wave point cloud data to obtain the target detection result of the t-th frame millimeter wave point cloud data.
[0093] Specifically, the target information prediction module of the track information inference network may be used to process the first target query feature as follows:
[0094] , predict the target update attributes;
[0095] in, is a multi-layer perceptron, represents the feature of the i-th target in the first target query feature, represents the displacement of the i-th target on the x-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the y-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the z-axis in the t-th frame of millimeter-wave point cloud data, represents the length of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the width of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the height of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the orientation angle of the i-th target in the t-th frame of millimeter wave point cloud data, represents the speed of the i-th target in the t-th frame of millimeter-wave point cloud data, Indicates the confidence of the target update attribute.
[0096] Specifically, the predicted target update attribute information can be used to process the position of the target in the t-th frame of millimeter wave point cloud data as follows:
[0097] , get the target detection result of the t-th frame millimeter wave point cloud data;
[0098] in, represents the x-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the y-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the z-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the target detection result of the t-th frame of millimeter wave point cloud data.
[0099] Step 2: Use the motion trajectory prediction module of the track information inference network to predict the intermediate motion data of the target query feature sequence and the first target query feature, use the motion trajectory prediction module of the track information inference network to decode the intermediate motion data to obtain decoded motion data, and use the motion trajectory prediction module of the track information inference network to predict the trajectory of the target in the t+x-th frame millimeter wave point cloud data according to the decoded motion data and the target detection result to obtain the target trajectory prediction result of the t+x-th frame millimeter wave point cloud data.
[0100] Specifically, the motion trajectory prediction module of the track information inference network may be used to process the target query feature sequence and the first target query feature as follows:
[0101] , to predict the intermediate motion data;
[0102] in, represents the inter-frame attention mechanism, Q, K, V represent the three feature vectors of Query, Key, and Value respectively, PE represents the position embedding, and X represents 1, 2, 3, etc. Represents the time from frame t to frame Frame contains Frame millimeter wave point cloud data, Initialized to 0, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame. represents the features of the i-th target in the target query features of the previous k frames and the first target query features, Indicates the k+x+1 frames of millimeter wave point cloud data from tk frame to t+k frame. represents the position embedding of k+x+1 frames of millimeter wave point cloud data, Represents intermediate motion data;
[0103] Specifically, the motion trajectory prediction module of the track information inference network may be used to process the intermediate motion data as follows:
[0104] Get decoded motion data, is a multi-layer perceptron, Decode motion data; with the decoder in and The predicted speed v is compared to the decoded motion data With better accuracy, the position of the target in the next frame (t+1) can be predicted by adding a single-step reasoning of the trajectory, that is, .
[0105] Specifically, the motion trajectory prediction module of the track information inference network may be used to perform the following processing according to the decoded motion data and the target detection result:
[0106] , predict the trajectory of the target in the t+x-th frame of millimeter wave point cloud data, and obtain the target trajectory prediction result of the t+x-th frame of millimeter wave point cloud data;
[0107] in, represents the target detection result of the millimeter wave point cloud data of the tth frame, Represents the target trajectory prediction result of the t+xth frame of millimeter wave point cloud data.
[0108] The location information with motion prediction can then be And the first target query feature Propagate to the next frame (t+1), thus completing the entire tracking process.
[0109] The first target query feature is labeled according to the target trajectory prediction result to obtain the target point cloud feature of the t-th frame of millimeter wave point cloud data.
[0110] For the network input being the t-th frame of millimeter-wave point cloud data, the target trajectory prediction result includes the predicted position of the target in the t+1-th frame of millimeter-wave point cloud data and the first target query feature of the t-th frame of millimeter-wave point cloud data. These two data are then used as the target point cloud features of the t-th frame of millimeter-wave point cloud data and propagated to the next frame (t+1) for target detection and target trajectory prediction for the next frame (t+1).
[0111] For an example, see Figure 2 , which is a second flow chart of the target detection method provided in an embodiment of the present application.
[0112] First, obtain millimeter wave point cloud data, which is single-frame data, for example, the t-th frame of millimeter wave point cloud data, and encode the t-th frame of millimeter wave point cloud data through an encoder to obtain a first encoded point cloud feature. Obtain the first target point cloud feature of the previous frame of the current frame, the first target point cloud feature includes the target feature of the previous frame and the target predicted position of the current frame, and input the first target point cloud feature and the first encoded point cloud feature into a decoder for decoding to obtain a first decoded point cloud feature, which includes the target position and target feature of the current frame.
[0113] For an example, see Figure 3 , which is a third flow chart of the target detection method provided in an embodiment of the present application.
[0114] Decode the target position in a single frame from the single frame point cloud features (first encoded point cloud features) and target features After that, the spatiotemporal attention network and the track information reasoning network are input in turn to convert the current target position Optimized for , and the target feature Optimized for (R is the abbreviation for "refinement").
[0115] The main task of these two networks is to aggregate the previous frame information to generate target optimization features and target optimized position Components , which can be expressed as follows:
[0116]
[0117] in, is the target query feature sequence, which maintains the target query features of the past k frames.
[0118] In order to solve the uncertainty problem in millimeter wave point cloud data detection, this application proposes to input the target query feature sequence into the spatiotemporal attention network to enhance the features of inter-frame attention and intra-frame attention, and obtain the first target query feature, specifically including:
[0119] (1) Inter-frame attention
[0120] First, attention is applied to the inter-frame target query features on the temporal axis to explicitly update the first decoded point cloud features using historical information. Inter-frame attention enables the features in the history window of each target k frame to interact with each other. Inter-frame attention can be expressed as follows:
[0121]
[0122] in, represents the inter-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. represents the feature of the i-th target in the first decoded point cloud feature, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame, represents the feature of the i-th target in the k+1 frame target query feature, represents the position embedding of k+1 frames of millimeter wave point cloud data, Represents the first updated point cloud feature.
[0123] (2) Intra-frame attention
[0124] Then, attention is applied to the first updated point cloud features between objects in the frame on the object axis to integrate the contextual information between objects, making the feature representation of each object more discriminative. The intra-frame attention can further update the first updated point cloud features to obtain the first object query features in the following way, and then use the first object query features to update the first decoded point cloud features in the object feature query sequence:
[0125] , get the first target query feature;
[0126] in, represents the intra-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. From the first target to the target goals, Indicates the first updated point cloud feature The characteristics of the target, Indicates the first updated point cloud feature The location of the target, Indicates the first updated point cloud feature The position embedding of the target, Represents the first target query feature.
[0127] In this module, decoupling cross-frame and cross-object attention has two effects. First, separating attention between frames (cross-frame) and objects (cross-object) allows for the design of dedicated temporal and positional encodings for each frame and position, thereby modeling the object's spatiotemporal relationship more accurately; second, the computational complexity of global cross-attention is reduced from Reduce to , the computational complexity is significantly reduced.
[0128] The track information inference network first optimizes the s target information of the current frame through the feature-enhanced first target query feature, and then learns the motion trajectory prediction. In order to use the predicted future trajectory to maintain the trajectory of occluded or noisy targets, and at the same time improve the propagation of target query features between frames, a motion trajectory prediction module with cross-frame propagation is proposed. The following is a detailed description:
[0129] (1) Target information prediction module
[0130] This module receives the first target query feature, infers the target bounding box, and uses a multi-layer perceptron MLP to predict the updated properties of the target, including the center point displacement ( , , ), size (l, w, h), orientation angle (θ), velocity (v), and confidence score (s):
[0131] , predict the target update attributes;
[0132] in, is a multi-layer perceptron, represents the feature of the i-th target in the first target query feature, represents the displacement of the i-th target on the x-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the y-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the z-axis in the t-th frame of millimeter-wave point cloud data, represents the length of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the width of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the height of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the orientation angle of the i-th target in the t-th frame of millimeter wave point cloud data, represents the speed of the i-th target in the t-th frame of millimeter-wave point cloud data, Indicates the confidence of the target update attribute.
[0133] The predicted information will then be used to adjust the original target box as follows:
[0134]
[0135] in, represents the x-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the y-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the z-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the target detection result of the t-th frame of millimeter wave point cloud data.
[0136] Thus, the target detection result of the predicted millimeter-wave point cloud data of the tth frame is obtained. , is the final target information output at frame time t.
[0137] (2) Motion trajectory prediction module
[0138] Trajectory prediction can supervise the model's ability to capture object motion and further facilitate the propagation of target positions between timestamps. Similar to the feature enhancement part in the spatiotemporal attention network, the motion trajectory prediction module adopts an inter-frame attention architecture. This module uses the target detection results and target query feature sequences output by the target information prediction module to generate motion embeddings for future x-frame timestamps with cross-frame attention. :
[0139] , to predict the intermediate motion data;
[0140] in, represents the inter-frame attention mechanism, Q, K, V represent the three feature vectors of Query, Key, and Value respectively, PE represents the position embedding, and X represents 1, 2, 3, etc. Represents the time from frame t to frame Frame contains Frame millimeter wave point cloud data, Initialized to 0, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame. represents the features of the i-th target in the target query features of the previous k frames and the first target query features, Indicates the k+x+1 frames of millimeter wave point cloud data from tk frame to t+k frame. represents the position embedding of k+x+1 frames of millimeter wave point cloud data, Represents intermediate motion data;
[0141]
[0142] With Decoder and Compared with the predicted velocity v in With better accuracy. Therefore, the position of the target in the next frame (t+1) can be predicted by adding a single-step reasoning of the trajectory:
[0143]
[0144] The target detection result of the t-th frame of millimeter wave point cloud data output by the target information prediction module. The output of the network is to generate target detection with consistent ID across frames. Finally, the target trajectory prediction result with motion prediction is And the feature-enhanced first target query feature is propagated to the next frame (t+1), thus finally completing the entire tracking process.
[0145] In a second aspect, the present application provides a target detection device, see Figure 4 , the device comprises:
[0146] The acquisition module 101 is used to acquire a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and acquire the tth frame of millimeter wave point cloud data, wherein t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with a predicted position of a target in the tth frame of millimeter wave point cloud data;
[0147] The encoding module 102 is used to input the t-th frame of millimeter wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature;
[0148] A decoding module 103, configured to input the first encoded point cloud feature and the first target point cloud feature into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature;
[0149] A fusion module 104 is used to obtain a historical frame target query feature, and fuse the first decoded point cloud feature with the historical frame target query feature in time sequence to obtain a target query feature sequence, wherein the historical frame target query feature includes the target query feature of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data;
[0150] A feature enhancement module 105 is used to input the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature;
[0151] The prediction module 106 is used to input the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction to obtain target detection results and target trajectory prediction results.
[0152] In a third aspect, the present application also provides an electronic device, such as Figure 5 As shown, including:
[0153] Memory 201, used for storing computer programs;
[0154] The processor 202 is used to implement any of the above-mentioned target detection methods when executing the program stored in the memory 201.
[0155] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 202, the communication interface, and the memory 201 communicate with each other via the communication bus.
[0156] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0157] The communication interface is used for communication between the above electronic device and other devices.
[0158] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0159] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0160] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above target detection methods are implemented.
[0161] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any target detection method in the above embodiments.
[0162] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0164] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0165] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A target detection method, characterized in that: The method comprises: Acquire a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and acquire the tth frame of millimeter wave point cloud data, wherein t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with a predicted position of a target in the tth frame of millimeter wave point cloud data; Inputting the t-th frame of millimeter wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature; Inputting the first encoded point cloud feature and the first target point cloud feature into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature; Acquire a historical frame target query feature, and fuse the first decoded point cloud feature with the historical frame target query feature in time sequence to obtain a target query feature sequence, wherein the historical frame target query feature includes the target query feature of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data; Inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature; The first target query feature and the target query feature sequence are input into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction to obtain target detection results and target trajectory prediction results.
2. The method according to claim 1, characterized in that The step of inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature includes: Inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model, and using the inter-frame attention mechanism module of the spatiotemporal attention network to perform inter-frame attention enhancement on the first decoded point cloud feature to obtain a first updated point cloud feature; The intra-frame attention mechanism module of the spatiotemporal attention network is used to perform intra-frame attention enhancement on the first updated point cloud feature to obtain a first target query feature.
3. The method according to claim 2, characterized in that The inter-frame attention mechanism module of the spatiotemporal attention network is used to perform inter-frame attention enhancement on the first decoded point cloud feature to obtain a first updated point cloud feature, including: The target query feature sequence is processed as follows using the inter-frame attention mechanism module of the spatiotemporal attention network: , get the first updated point cloud feature; in, represents the inter-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. represents the feature of the i-th target in the first decoded point cloud feature, tk: k represents the k+1 frame of millimeter wave point cloud data from tk frame to t frame, represents the feature of the i-th target in the k+1 frame target query feature, represents the position embedding of k+1 frames of millimeter wave point cloud data, Represents the first updated point cloud feature.
4. The method according to claim 2, characterized in that: The intra-frame attention mechanism module of the spatiotemporal attention network is used to perform intra-frame attention enhancement on the first updated point cloud feature to obtain a first target query feature, including: The first updated point cloud feature is processed as follows using the intra-frame attention mechanism module of the spatiotemporal attention network: , get the first target query feature; in, represents the intra-frame attention mechanism, Q, K, and V represent the three feature vectors of Query, Key, and Value respectively, and PE represents position embedding. From the first target to the target goals, Indicates the first updated point cloud feature The characteristics of the target, Indicates the first updated point cloud feature The location of the target, Indicates the first updated point cloud feature The position embedding of the target, Represents the first target query feature.
5. The method according to claim 1, characterized in that The step of inputting the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction, and obtaining a target detection result and a target trajectory prediction result, includes: Inputting the first target query feature into the track information inference network of the deep learning model, and using the target information prediction module of the track information inference network to predict the target update attribute of the first target query feature; using the predicted target update attribute information to update the position of the target in the t-th frame of millimeter wave point cloud data, and obtaining the target detection result of the t-th frame of millimeter wave point cloud data; The motion trajectory prediction module of the track information inference network is used to predict the intermediate motion data of the target query feature sequence and the first target query feature, the motion trajectory prediction module of the track information inference network is used to decode the intermediate motion data to obtain decoded motion data, and the motion trajectory prediction module of the track information inference network is used to predict the trajectory of the target in the t+x-th frame millimeter wave point cloud data according to the decoded motion data and the target detection result to obtain the target trajectory prediction result of the t+x-th frame millimeter wave point cloud data.
6. The method according to claim 5, characterized in that The target information prediction module using the track information inference network predicts the target update attribute of the first target query feature; comprising: The target information prediction module of the track information inference network is used to process the first target query feature as follows: , predict the target update attributes; in, is a multi-layer perceptron, represents the feature of the i-th target in the first target query feature, represents the displacement of the i-th target on the x-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the y-axis in the t-th frame of millimeter-wave point cloud data, represents the displacement of the i-th target on the z-axis in the t-th frame of millimeter-wave point cloud data, represents the length of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the width of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the height of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the orientation angle of the i-th target in the t-th frame of millimeter wave point cloud data, represents the speed of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the confidence of the target update attribute; The predicted target update attribute information is used to update the position of the target in the t-th frame of millimeter wave point cloud data, and the target detection result of the t-th frame of millimeter wave point cloud data is obtained, including: The predicted target update attribute information is used to process the position of the target in the t-th frame of millimeter wave point cloud data as follows: , get the target detection result of the t-th frame millimeter wave point cloud data; in, represents the x-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the y-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, represents the z-axis value of the i-th target in the t-th frame of millimeter-wave point cloud data, Represents the target detection result of the t-th frame of millimeter wave point cloud data.
7. The method according to claim 5, characterized in that The method uses the motion trajectory prediction module of the track information inference network to predict intermediate motion data of the target query feature sequence and the first target query feature, uses the motion trajectory prediction module of the track information inference network to decode the intermediate motion data to obtain decoded motion data, and uses the motion trajectory prediction module of the track information inference network to predict the trajectory of the target in the t+x-th frame millimeter wave point cloud data according to the decoded motion data and the target detection result to obtain the target trajectory prediction result of the t+x-th frame millimeter wave point cloud data, including: The motion trajectory prediction module of the track information inference network is used to process the target query feature sequence and the first target query feature as follows: , to predict the intermediate motion data; in, represents the inter-frame attention mechanism, Q, K, V represent the three feature vectors of Query, Key, and Value respectively, PE represents the position embedding, and X represents 1, 2, 3, etc. Represents the time from frame t to frame Frame contains Frame millimeter wave point cloud data, Initialized to 0, tk:t represents the k+1 frame millimeter wave point cloud data from tk frame to t frame. represents the features of the i-th target in the target query features of the previous k frames and the first target query features, Indicates the k+x+1 frames of millimeter wave point cloud data from tk frame to t+k frame. represents the position embedding of k+x+1 frames of millimeter wave point cloud data, Represents intermediate motion data; The motion trajectory prediction module of the track information inference network is used to process the intermediate motion data as follows: Get decoded motion data, is a multi-layer perceptron, Decode motion data; The motion trajectory prediction module using the track information inference network performs the following processing based on the decoded motion data and the target detection result: , predict the trajectory of the target in the t+x-th frame of millimeter wave point cloud data, and obtain the target trajectory prediction result of the t+x-th frame of millimeter wave point cloud data; in, represents the target detection result of the millimeter wave point cloud data of the tth frame, Represents the target trajectory prediction result of the t+xth frame of millimeter wave point cloud data.
8. The method according to claim 1, characterized in that The method further comprises: The first target query feature is labeled according to the target trajectory prediction result to obtain the target point cloud feature of the t-th frame of millimeter wave point cloud data.
9. A target detection device, characterized in that: The device comprises: An acquisition module, used to acquire a first target point cloud feature of the t-1th frame of millimeter wave point cloud data, and to acquire the tth frame of millimeter wave point cloud data, wherein t is an integer greater than 1, and the first target point cloud feature is a target query feature marked with a predicted position of a target in the tth frame of millimeter wave point cloud data; An encoding module, used for inputting the t-th frame of millimeter wave point cloud data into an encoder of a pre-trained deep learning model for encoding to obtain a first encoded point cloud feature; A decoding module, used for inputting the first encoded point cloud feature and the first target point cloud feature into a decoder of the deep learning model for decoding to obtain a first decoded point cloud feature; A fusion module is used to obtain a historical frame target query feature, and fuse the first decoded point cloud feature with the historical frame target query feature in time sequence to obtain a target query feature sequence, wherein the historical frame target query feature includes the target query feature of the millimeter wave point cloud data collected before the t-th frame of millimeter wave point cloud data; A feature enhancement module, used for inputting the target query feature sequence into the spatiotemporal attention network of the deep learning model to perform feature enhancement of inter-frame attention and intra-frame attention to obtain a first target query feature; A prediction module is used to input the first target query feature and the target query feature sequence into the track information inference network of the deep learning model to perform target prediction and motion trajectory prediction, so as to obtain target detection results and target trajectory prediction results.
10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.