A target long-time trajectory prediction method, system, device and medium based on a dynamic graph structure cell long short-term memory network

Abstract

A target long-time trajectory prediction method, system, device and medium based on a dynamic graph structure cell long short-term memory network, the method uses the effectiveness of a target interaction structure graph in representing interaction information in a motion space, combines a graph convolutional neural network to transfer the implicit features of a subject target, and according to a feature screening mechanism of a graph attention mechanism, focuses on information features of different dimensions, increases a spatial information memory unit, enhances the ability of a long short-term memory network (LSTM) to process spatial structure information, and enables the improved LSTM network to simultaneously process time sequence information and spatial information of the motion space; the system, device and medium are used to implement the method; through the characteristics of the graph convolution algorithm, the graph attention mechanism and the LSTM network, the target long-time trajectory prediction method can effectively solve the trajectory endpoint prediction ambiguity problem in target long-time trajectory prediction, and better results can be achieved, and the prediction results have smaller errors from the true values.

Description

Technical Field

[0001] This invention relates to the field of computer data quantization technology, specifically to a method, system, device, and medium for predicting long-term trajectory of a target based on a dynamic graph structure cellular long short-term memory network. Background Technology

[0002] Long-term target trajectory prediction technology aims to predict the position of a target in three-dimensional space from video or radar sequences, and is widely used in fields such as autonomous driving, vehicle-to-everything (V2X) communication, and target tracking. With the development of deep learning technology, this field has made significant progress.

[0003] However, long-term trajectory prediction of targets still faces the following major challenges: 1. Information in the motion space cannot be fully applied to target trajectory prediction; 2. Current prediction methods based on graph convolutional networks use spatial graph structures to represent target motion information in the motion space, but lack temporal information. Therefore, they cannot take into account both spatial interaction information and temporal sequence information, which may lead to increased long-term trajectory prediction errors.

[0004] To address these challenges, researchers have utilized Graph Convolutional Networks (GCNs) for target trajectory prediction. Compared to traditional convolutional neural networks, GCNs extend convolutional operations from traditional image processing to graph data processing. The key idea is to create a mapping function that extracts interaction-aware features from node features and their neighboring nodes in the network, considering interaction-related factors by treating the moving target as a node in the network at each sampling time. Furthermore, GCNs offer a promising approach to trajectory prediction by explicitly modeling spatial dependencies and relationships between objects. They can effectively capture contextual information and handle irregular graph structures. However, some drawbacks remain when applying GCNs to trajectory prediction tasks. First, trajectory prediction tasks often involve dynamically changing graph structures, such as the position and velocity of a moving object changing over time. While GCNs perform well on static graph structures, additional mechanisms may be needed to capture the changes in time-series data when dealing with dynamic graphs. Second, although GCNs can capture local connection patterns between nodes, they may be insufficient to capture complex spatial relationships, especially in high-dimensional spaces. This may necessitate combining them with other techniques, such as attention mechanisms or transformers, to better understand spatial dependencies. Finally, while GCNs may perform well on training data, their generalization ability may be limited when faced with unseen data or environments. This is because GCNs typically rely on the local structure of the graph, while global contextual information is equally important for accurate predictions in trajectory prediction. Therefore, more complex models may need to be designed to improve the generalization ability of GCNs in trajectory prediction tasks.

[0005] The patent application with publication number CN118264670A discloses a service migration method based on priority and spatiotemporal trajectory prediction. It constructs a time-series user interaction relationship trajectory map, concatenates the feature representations output by the spatiotemporal trajectory map through the Transformer model to obtain a feature vector, and uses an MLP decoder to obtain the user's future movement trajectory. However, due to the complex structure of the Transformer model, many hyperparameters need to be adjusted, resulting in high computational cost and large processing latency.

[0006] The patent application with publication number CN118261051A discloses a method for constructing a pedestrian and vehicle trajectory prediction model based on heterogeneous graph networks. It uses a heterogeneous graph structure to represent the spatial relationships between heterogeneous traffic individuals and transforms them into physical knowledge hidden features through the heterogeneous graph edge convolution method. However, since this method mainly focuses on the movement trajectory of vehicles and does not fully consider the influence of external factors such as the interaction between vehicles, it is difficult to accurately capture social behavior patterns. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device and medium for predicting the long-term trajectory of a target based on a dynamic graph structure cellular long short-term memory network. By constructing a spatial information memory unit, the method combines the extraction of the target's temporal and spatial information in the motion space, and comprehensively analyzes the target's historical information and state. It can accurately predict the long-term future trajectory range of a target by combining its historical interaction information and temporal information with the target that has lost long-term information.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for predicting the long-term trajectory of a target based on a dynamic graph-structured cellular long short-term memory network includes the following steps:

[0010] Step 1: Use a 3D target detector and a 3D target tracker to detect the input video and obtain the 3D motion information features of all targets in the video;

[0011] Step 2: Perform uniform standardization processing on the 3D motion information features of all targets in the video detected in Step 1 to obtain a motion target information feature dataset;

[0012] Step 3: Extract each scene from the motion target information feature dataset obtained in Step 2, and represent each moment in each scene as a graph. In this graph, the target ID in each moment is used as a node, the predicted moving target in the target ID is used as the main node, and the other moving targets in the target ID are used as other nodes. The main node and other nodes are used to construct a graph adjacency matrix that reflects the motion space.

[0013] Step 4: Take the target information features in each scene at each time step of the standardized motion target information feature dataset from Step 2 as the initial input feature matrix of the graph convolutional network and graph attention mechanism. This provides 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in Step 3. The spatial graph feature extraction module (SGFE) is used to update the features of each node in the graph adjacency matrix and output implicit spatial motion structure information, including motion information between targets in the motion space.

[0014] Step 5: Input the implicit spatial motion structure information output in Step 4 into the Dynamic Graph Structural Cell Long Short-Term Memory Network (TSGCell-LSTM). The Dynamic Graph Structural Cell Long Short-Term Memory Network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

[0015] In step 1, the 3D motion information features include, but are not limited to, position coordinate information, velocity information, and acceleration information.

[0016] In step 2, the specific standardization steps are as follows:

[0017] Step 2.1, Coordinate Transformation: Transform the 3D motion information features of all targets in the video obtained by the 3D target detector and 3D target tracker in Step 1 from the camera coordinate system to the virtual coordinate system;

[0018] Step 2.2, Normalization: Normalize the 3D motion information features of the predicted moving targets in all targets after transformation in Step 2.1. Take the center point of the virtual coordinate system as the origin and center the moving targets. At the same time, perform the same normalization process on other moving targets in all targets to obtain different moving targets and their information features, and construct a moving target information feature dataset.

[0019] Step 2.3, Data Segmentation and Augmentation: The moving target information feature dataset obtained in Step 2.2 is segmented according to the task requirements. At the same time, the segmented moving target information feature dataset is downsampled. The downsampled moving target information feature dataset is then rotated, translated, and scaled to obtain the augmented moving target information feature dataset.

[0020] The construction steps in step 3 are as follows:

[0021] Step 3.1: Using the moving target information feature dataset obtained in Step 2.3 as the input to the dataset scene extraction module, extract each scene from the moving target information feature dataset;

[0022] Step 3.2: Represent each moment in each scene obtained in Step 3.1 as a graph, where the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph.

[0023] Step 3.3: Based on the master node and other nodes in Step 3.2, traverse the motion target information features of each scene in Step 3.1, extract the connection relationship between the predicted motion target and other motion targets, and construct the connection relationship between the master node and other nodes at each time step;

[0024] Step 3.4: Convert the connection relationships between the master node and other nodes at each time step obtained in Step 3.2 and the master node and other nodes at each time step constructed in Step 3.3 into a sparse matrix, where the rows and columns of the matrix correspond to the target IDs of the moving target, and the values ​​of the matrix indicate whether there is a connection between the corresponding nodes; based on the connection status of the nodes, convert the graph adjacency matrix into a symmetric matrix.

[0025] Step 3.5: Normalize the graph adjacency matrix after symmetry transformation in step 3.4 to obtain the graph adjacency matrix that reflects the motion space.

[0026] The specific process of step 4 is as follows:

[0027] In step 4, the Spatial Graph Feature Extraction (SGFE) module includes an input layer, an intermediate layer, and an output layer. The input layer consists of a graph convolutional layer and a GELU activation function. The intermediate layer consists of a graph convolutional layer and a graph attention mechanism layer connected in parallel. The graph convolutional layer is used for further extraction of spatial information, and the graph attention mechanism layer is used to filter key dimensional features of spatial information. The output layer includes a fully connected layer for generating implicit spatial structure information.

[0028] Step 4.1: Construct the target information features of each scene at each time step in the standardized motion target information feature dataset from Step 2 into an initial input feature matrix. Use the graph convolutional layer of the input layer of the Spatial Graph Feature Extraction (SGFE) module to perform convolution operation on the initial input feature matrix to extract motion features in the local neighborhood. At the same time, the GELU activation function is used to introduce non-linear characteristics into the convolution output, enhance the expressive power of the model, ensure effective feature learning, and finally obtain the initial hidden spatial information feature matrix.

[0029] Step 4.2: The initial hidden spatial information feature matrix obtained in Step 4.1 is used as the input of the intermediate layer. The graph convolutional layer further extracts spatial information from the initial hidden spatial information features, maintaining the information flow between nodes, and obtains the intermediate layer hidden spatial information feature matrix. The graph attention mechanism layer calculates the attention weights between nodes based on the intermediate layer hidden spatial information features and the graph adjacency matrix obtained in Step 3, so as to selectively emphasize important features. Through weighted feature fusion, the key dimensional features in the motion space are extracted.

[0030] Step 4.3: Integrate the hidden spatial information feature matrix of the intermediate layer and the key dimensional features in the motion space obtained in Step 4.2 to ensure that the output dimension matches the task requirements, and convert it into implicit spatial motion structure information through a fully connected layer.

[0031] The specific process of step 5 is as follows:

[0032] The Dynamic Graph Structured Cellular Long Short-Term Memory Network (TSGCell-LSTM) comprises a standard memory unit, a spatial information memory unit, and a forgetting unit. The standard memory unit stores state information updated over time steps, responsible for determining information updates and retention at each time step based on input data and the network's historical state. The spatial information memory unit enhances the spatial information memory capability of the standard LSTM network, jointly extracting the temporal and spatial information of the target in the motion space and updating the interaction information at each time step in the motion space. The forgetting unit helps the network clear irrelevant or outdated information, ensuring that the network retains only important feature information. The specific implementation steps of the Dynamic Graph Structured Cellular Long Short-Term Memory Network (TSGCell-LSTM) are as follows:

[0033] Step 5.1, Feature Input: Input the implicit spatial motion structure information obtained in Step 4 into the Dynamic Graph Structure Cell Long Short-Term Memory Network (TSGCell-LSTM);

[0034] Step 5.2, Spatial Information Filtering: A portion of the implicit interaction information in the motion space input in Step 5.1 is filtered and transmitted. The information filtering method is as follows:

[0035]

[0036] Gr(t) = sigmoid(net) G (t)) (5)

[0037] Among them, G t h represents the spatial interaction information for the current frame. t-1 Gr(t) represents the information transmitted from the previous frame, and Gr(t) represents the filtered spatial information.

[0038] Step 5.3, Information Update: The spatial information Gr(t) filtered in Step 5.2 enters the spatial information memory unit in the Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM). The spatial information memory unit reinforces the memory of the current spatial structure information through a memory gate-like operation. The information transmission method is represented as follows:

[0039] C t =C t1 +I(t)*R(t)*Gr(t) (6)

[0040] Among them, C t1 For memory information after passing through standard memory units and forgetting units, C t This refers to memory information updated using spatial information.

[0041] Step 5.4, Output update result: Add a bias term to the memory information updated in Step 5.3, and output the iterative update result;

[0042] Step 5.5: The update result from step 5.4 is output and fed into the last fully connected layer. Regression is performed to obtain the predicted future 3D motion position information of the moving target.

[0043] This invention also provides a target long-term trajectory prediction system based on a dynamic graph-structured cellular long short-term memory network, comprising:

[0044] The 3D motion information feature acquisition module is used to detect the input video using a 3D target detector and a 3D target tracker, and acquire the 3D motion information features of all targets in the video.

[0045] The moving target information feature dataset acquisition module is used to perform unified standardization processing on the 3D motion information features of all targets in the detected video to obtain the moving target information feature dataset.

[0046] The graph adjacency matrix construction module is used to extract each scene from the moving target information feature dataset and represent each moment in each scene as a graph. In this graph, the target ID at each moment is used as a node, the predicted moving target in the target ID is used as the master node, and the other moving targets in the target ID are used as other nodes. The master node and other nodes are used to construct a graph adjacency matrix that reflects the motion space.

[0047] The implicit spatial motion structure information acquisition module is used to take the target information features in each scene at each time step in the standardized motion target information feature dataset as the initial input feature matrix of the graph convolutional network and graph attention mechanism, and provide 3D motion information features for the graph adjacency matrix that reflects the motion space. The spatial graph feature extraction module (SGFE) is used to update the features of each node in the graph adjacency matrix and output implicit spatial motion structure information, including the motion information between each target in the motion space.

[0048] The long-term trajectory information output module for the moving target is used to input implicit spatial motion structure information into the Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM). The Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the predicted long-term trajectory information of the moving target, including the predicted future 3D motion position information of the moving target.

[0049] This invention also provides a target long-term trajectory prediction device based on a dynamic graph-structured cellular long short-term memory network, comprising:

[0050] Memory: A computer program that stores the above-mentioned method for predicting long-term trajectory of a target based on a dynamic graph structure cellular long short-term memory network, and is a computer-readable device;

[0051] Processor: Used to implement the target long-term trajectory prediction method based on dynamic graph structure cellular long short-term memory network when executing the computer program.

[0052] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the target long-term trajectory prediction method based on a dynamic graph structure cellular long short-term memory network.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0054] (1) This invention proposes a target long-term trajectory prediction method based on dynamic graph structure cell long short-term memory network. By jointly extracting the target temporal information and spatial information in the motion space, the interaction information at each moment in the motion space is updated and the trajectory is predicted, which can achieve a high degree of integration of spatial and temporal information in the prediction process.

[0055] (2) This invention proposes a novel spatial implicit interaction feature extraction method. By combining graph convolutional networks and graph attention mechanisms, the features of each node in the graph are updated. This method can more effectively extract and utilize spatial information, and provides an effective solution to the problem of graph convolutional networks struggling to mine spatial hidden information. Furthermore, this algorithm can consider the motion interactions of various targets in the motion space, thereby accurately extracting the interaction information between targets.

[0056] (3) This invention proposes a novel long-term target prediction method based on spatiotemporal information features. By enhancing the spatial information memory capability of the standard LSTM network, it can more effectively process and update the interaction information in the motion space. At the same time, it can use the spatial structure information and time series information in the motion space to accurately predict the long-term trajectory of moving targets, and can reduce the computational load compared with Transformer.

[0057] In summary, this invention effectively improves the accuracy and reliability of long-term trajectory prediction through innovative feature extraction and model design, providing a new solution for target motion prediction in complex dynamic environments. Attached Figure Description

[0058] Figure 1 This is a flowchart of a target long-term trajectory prediction method based on a dynamic graph structured cellular long short-term memory network according to the present invention.

[0059] Figure 2 This invention is based on the principle of Spatial Graph Feature Extraction Module (SGFE).

[0060] Figure 3 It is a dynamic graph structured cell long short-term memory network (TSGCell-LSTM) built by benfam based on the principle.

[0061] Figure 4 This is a graph showing the experimental results of a specific implementation of this method. Detailed Implementation

[0062] The specific usage of this invention will be described below with reference to the accompanying drawings to help users better understand the invention. It should be noted that detailed descriptions of existing functions and methods will be de-emphasized in the following description. This invention primarily focuses on methods for predicting long-term target trajectories.

[0063] Figure 1 This is a flowchart illustrating a target long-term trajectory prediction method based on a dynamic graph-structured cellular long short-term memory network, according to a specific embodiment of the present invention. A network model was built based on this method for target long-term trajectory prediction applications.

[0064] In this embodiment, as Figure 1 As shown, a method for predicting the long-term trajectory of a target based on a dynamic graph-structured cellular long short-term memory network includes the following steps:

[0065] Step 1: Use a 3D target detector and a 3D target tracker to detect the input video and obtain the 3D motion information features of all targets in the video;

[0066] The 3D motion information features include, but are not limited to, position coordinate information, velocity information, and acceleration information.

[0067] In step 1, the 3D target detector is a fully convolutional single-stage 3D target detection (FCOS-LiDAR) model; the video is used as the input of the 3D target detector to obtain the ID of the moving target in the video and its 3D motion coordinates, which are then used in the subsequent long-term trajectory prediction step.

[0068] In step 1, the 3D target tracker is the EagerMOT model; the video is used as the input to the 3D target tracker to obtain the 3D motion information of the moving target in the video, which is then used in the subsequent long-term trajectory prediction step.

[0069] Step 2: Perform uniform standardization processing on the 3D motion information features of all targets in the video detected in Step 1 to obtain a motion target information feature dataset;

[0070] In step 2, the specific standardization steps are as follows:

[0071] Step 2.1, Coordinate Transformation: Transform the 3D motion information features of all targets in the video obtained by the 3D target detector and 3D target tracker in Step 1 from the camera coordinate system to the virtual coordinate system;

[0072] Step 2.2, Normalization: Normalize the 3D motion information features of the predicted moving targets in all targets after transformation in Step 2.1. Take the center point of the virtual coordinate system as the origin and center the moving targets. At the same time, perform the same normalization process on other moving targets in all targets to obtain different moving targets and their information features, and construct a moving target information feature dataset.

[0073] Step 2.3, Data Segmentation and Augmentation: The moving target information feature dataset obtained in Step 2.2 is segmented according to the task requirements. At the same time, the segmented moving target information feature dataset is downsampled. The downsampled moving target information feature dataset is then rotated, translated, and scaled to obtain the augmented moving target information feature dataset.

[0074] Data augmentation aims to increase the diversity of training data used in neural networks, thereby enhancing the model's robustness under different conditions.

[0075] Step 3: Extract each scene from the motion target information feature dataset obtained in Step 2, and represent each moment in each scene as a graph. In this graph, the target ID in each moment is used as a node, the predicted moving target in the target ID is used as the main node, and the other moving targets in the target ID are used as other nodes. The main node and other nodes are used to construct a graph adjacency matrix that reflects the motion space.

[0076] The construction steps in step 3 are as follows:

[0077] Step 3.1: Using the moving target information feature dataset obtained in Step 2.3 as the input to the dataset scene extraction module, extract each scene from the moving target information feature dataset;

[0078] Step 3.2: Represent each moment in each scene obtained in Step 3.1 as a graph, where the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph.

[0079] Step 3.3: Based on the master node and other nodes in Step 3.2, traverse the motion target information features of each scene in Step 3.1, extract the connection relationship between the predicted motion target and other motion targets, and construct the connection relationship between the master node and other nodes at each time step;

[0080] Step 3.4: Convert the connection relationships between the master node and other nodes at each time step obtained in Step 3.2 and the master node and other nodes at each time step constructed in Step 3.3 into a sparse matrix, where the rows and columns of the matrix correspond to the target IDs of the moving targets, and the values ​​of the matrix indicate whether there is a connection between the corresponding nodes; based on the connection status of the nodes, convert the graph adjacency matrix into a symmetric matrix to reflect the bidirectional connections between the predicted moving targets and other moving targets.

[0081] Step 3.5: Normalize the graph adjacency matrix after symmetry transformation in step 3.4 to obtain a graph adjacency matrix that reflects the motion space; in order to ensure the stability and numerical range of the graph adjacency matrix.

[0082] Step 4: The target information features in each scene at each time step of the standardized moving target information feature dataset from Step 2 are used as the initial input feature matrix of the graph convolutional network and graph attention mechanism. This provides 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in Step 3. The Spatial Graph Feature Extraction (SGFE) module is used to update the features of each node (i.e., each moving target) in the graph adjacency matrix (i.e., the motion information features of each moving target), and outputs implicit spatial motion structure information, including the motion information between targets in the motion space, thereby achieving accurate extraction of the interaction information of moving targets in the input video.

[0083] The specific process of step 4 is as follows:

[0084] like Figure 2 As shown, in step 4, the Spatial Graph Feature Extraction (SGFE) module includes an input layer, an intermediate layer, and an output layer; the input layer consists of a graph convolutional layer and a GELU activation function; the intermediate layer consists of a graph convolutional layer and a graph attention mechanism layer connected in parallel, wherein the graph convolutional layer is used for further extraction of spatial information, and the graph attention mechanism layer is used to filter key dimensional features of spatial information; the output layer includes a fully connected layer for generating implicit spatial structure information;

[0085] Step 4.1: Construct the target information features of each scene at each time step in the standardized motion target information feature dataset from Step 2 into an initial input feature matrix. Use the graph convolutional layer of the input layer of the Spatial Graph Feature Extraction (SGFE) module to perform convolution operation on the initial input feature matrix to extract motion features in the local neighborhood. At the same time, the GELU activation function is used to introduce non-linear characteristics into the convolution output, enhance the expressive power of the model, ensure effective feature learning, and finally obtain the initial hidden spatial information feature matrix.

[0086] Step 4.2: The initial hidden spatial information feature matrix obtained in Step 4.1 is used as the input to the intermediate layer. The graph convolutional layer further extracts spatial information from the initial hidden spatial information features, maintaining the information flow between nodes, to obtain the intermediate layer hidden spatial information feature matrix. (Increasing the number of convolutional layers can further improve the depth of feature extraction.) The graph attention mechanism layer calculates the attention weights between nodes based on the intermediate layer hidden spatial information features and the graph adjacency matrix obtained in Step 3, selectively emphasizing important features. Through weighted feature fusion, key dimensional features in the motion space are extracted.

[0087] Step 4.3: Integrate the hidden spatial information feature matrix of the intermediate layer and the key dimensional features in the motion space obtained in Step 4.2 to ensure that the output dimension matches the task requirements, and convert it into implicit spatial motion structure information through a fully connected layer.

[0088] Step 5: Input the implicit spatial motion structure information output in Step 4 into the Dynamic Graph Structural Cell Long Short-Term Memory Network (TSGCell-LSTM). The Dynamic Graph Structural Cell Long Short-Term Memory Network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target, thereby realizing the accurate prediction of the trajectory of the main moving target in the input video.

[0089] The specific process of step 5 is as follows:

[0090] like Figure 3 As shown, the Dynamic Graph Structured Cellular Long Short-Term Memory Network (TSGCell-LSTM) includes standard memory units, spatial information memory units, and forgetting units. The standard memory units store state information updated over time steps, responsible for determining information updates and retention at each time step based on input data and the network's historical state. The spatial information memory units enhance the spatial information memory capability of the standard LSTM network, jointly extracting the temporal and spatial information of the target in the motion space and updating the interaction information at each time step in the motion space. The forgetting units help the network clear irrelevant or outdated information, ensuring the network retains only important feature information. The specific implementation steps of the Dynamic Graph Structured Cellular Long Short-Term Memory Network (TSGCell-LSTM) are as follows:

[0091] Step 5.1, Feature Input: Input the implicit spatial motion structure information obtained in Step 4 into the Dynamic Graph Structure Cell Long Short-Term Memory Network (TSGCell-LSTM);

[0092] Step 5.2, Spatial Information Filtering: A portion of the implicit interaction information in the motion space input in Step 5.1 is filtered and transmitted. The information filtering method is as follows:

[0093]

[0094] Gr(t) = sigmoid(net) G (t)) (5)

[0095] Among them, G t h represents the spatial interaction information for the current frame. t-1 Gr(t) represents the information transmitted from the previous frame, and Gr(t) represents the filtered spatial information.

[0096] Step 5.3, Information Update: The spatial information Gr(t) filtered in Step 5.2 enters the spatial information memory unit in the Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM). The spatial information memory unit reinforces the memory of the current spatial structure information through a memory gate-like operation. The information transmission method is represented as follows:

[0097] C t =C t1 +I(t)*R(t)*Gr(t) (6)

[0098] Among them, C t1 For memory information after passing through standard memory units and forgetting units, C t This refers to memory information updated using spatial information.

[0099] Step 5.4, Output update result: Add a bias term to the memory information updated in Step 5.3, and output the iterative update result;

[0100] Step 5.5: The update result from step 5.4 is output and fed into the last fully connected layer. Regression is performed to obtain the predicted future 3D motion position information of the moving target.

[0101] The above steps implement the updating of the motion features of the main target in the image using the TSGCell-LSTM network. The TSGCell-LSTM network performs multiple iterative updates, continuously seeking the optimal value through deep learning training.

[0102] Experimental Test

[0103] Experimental conditions

[0104] This invention employs experimental simulation based on a real dataset, specifically the "DAIR-V2X series dataset," which is a real-world temporal perception dataset for vehicle-infrastructure collaboration. Mean squared error (MSE), average displacement error (ADE), and final displacement error (FDE) are used as evaluation metrics to measure the average position distance and destination distance between the predicted trajectory and the ground truth (GT) trajectory, respectively. In the experiment, the past 40 frames of data are used as the observed trajectory, and the trajectory data for the next 40 frames is predicted.

[0105] Experimental data

[0106] This experiment is based on the "DAIR-V2X Vehicle-to-Infrastructure Cooperative Dataset," a real-world temporal perception dataset for vehicle-infrastructure cooperation. It contains approximately 100 image and point cloud sequences from both vehicle and roadside perspectives, along with corresponding 3D bounding boxes and tracking ID annotations. Data was collected from complex traffic intersections in the Beijing High-Level Autonomous Driving Demonstration Zone, utilizing equipment such as a 300-line LiDAR and a 1-inch global exposure CMOS camera, achieving GPS timing synchronization and intrinsic / extrinsic parameter calibration. The dataset also includes detailed calibration and coordinate system transformation information, as well as 3D annotations for vehicle-infrastructure cooperation based on spatiotemporal synchronization and trajectory matching algorithms, providing rich data support for this experimental research.

[0107] Experimental results

[0108] Experiments were conducted to compare this invention with most target trajectory prediction methods, such as SGAN, MALS-Net, Social-LSTM, and Bi-LSTM, calculating the squared prediction error (MSE), average distance error (ADE), and final prediction error (FDE) between the test and true values. In long-term prediction, this invention demonstrates significant advantages. Long-term prediction typically faces increased uncertainty in target trajectories and the impact of accumulated errors; however, this invention still outperforms existing technologies in prediction accuracy and stability, effectively tracking the trend changes of target motion in long-term series and mitigating the growth of accumulated errors. Figure 4 This is particularly evident in the multi-frame prediction results: when the number of prediction frames reaches 20 frames or longer, the trajectory prediction of this invention still closely approximates the actual path. Therefore, this invention can effectively track the trend changes of target motion in long-term series and mitigate the growth of accumulated errors. Experimental results are as follows... Figure 4 As shown, the blue line represents the historical x-coordinate of the target to be predicted, the orange line represents the historical y-coordinate of the target to be predicted, the purple line represents the true x-coordinate of the target to be predicted, the brown line represents the true y-coordinate of the target to be predicted, the green line represents the predicted x-coordinate of the target to be predicted, and the red line represents the predicted y-coordinate of the target to be predicted.

[0109] Furthermore, as shown in Table 1, each row of data represents the MSE (Mean Squared Prediction Error) of the predicted trajectory for 50 frames based on 50 frames of historical motion data across 1000 motion scenarios using different methods; a smaller value indicates better performance. After validation using the test set of the dataset, tests were conducted on different motion scenarios within the dataset, and the squared prediction error (MSE), average distance error (ADE), and final prediction error (FDE) between the test and true values ​​were calculated. Comparison with classic methods confirmed the effectiveness of our proposed method.

[0110] The comparative experimental results in Table 1 show that each row represents the mean squared error (MSE), average distance error (ADE), and final prediction error (FDE) of different methods over 50 frames of prediction. Lower values ​​for these three metrics indicate better performance. This invention (TSGCell-LSTM) demonstrates significant advantages in long-term prediction, with MSE, ADE, and FDE all significantly lower than existing methods such as SGAN, MALS-Net, and Social-LSTM, indicating higher accuracy in long-term trajectory prediction. Therefore, this invention effectively controls accumulated error and adapts to multi-target interference and nonlinear motion in complex scenes, making it closer to the real trajectory over long time spans, showcasing good application potential and creativity.

[0111] Table 1. Experimental Results

[0112]

[0113] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

[0114] This invention also provides a target long-term trajectory prediction system based on a dynamic graph-structured cellular long short-term memory network, comprising:

[0115] The 3D motion information feature acquisition module is used to detect the input video using a 3D target detector and a 3D target tracker in step 1, and acquire the 3D motion information features of all targets in the video.

[0116] The moving target information feature dataset acquisition module is used to perform unified standardization processing on the 3D motion information features of all targets detected in the video in step 1 in step 2, so as to obtain the moving target information feature dataset.

[0117] The graph adjacency matrix construction module is used to extract each scene from the motion target information feature dataset obtained in step 2 in step 3, and represent each moment in each scene as a graph. In this graph, the target ID in each moment is used as a node, the predicted motion target in the target ID is used as the master node, and the other motion targets in the target ID are used as other nodes. The master node and other nodes are used to construct a graph adjacency matrix that reflects the motion space.

[0118] The implicit spatial motion structure information acquisition module is used to implement the following in step 4: the target information features in each scene at each time step in the motion target information feature dataset after step 2 are used as the initial input feature matrix of the graph convolutional network and graph attention mechanism. It provides 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in step 3. The spatial graph feature extraction module (SGFE) is used to update the features of each node in the graph adjacency matrix and output implicit spatial motion structure information, including the motion information between each target in the motion space.

[0119] The long-term trajectory information output module for the moving target is used to input the implicit spatial motion structure information output in step 4 into the Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM) in step 5. The Dynamic Graph Structured Cell Long Short-Term Memory Network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the predicted long-term trajectory information of the moving target, including the predicted future 3D motion position information of the moving target.

[0120] This invention also provides a target long-term trajectory prediction device based on a dynamic graph-structured cellular long short-term memory network, comprising:

[0121] Memory: A computer program that stores the above-mentioned method for predicting long-term trajectory of a target based on a dynamic graph structure cellular long short-term memory network, and is a computer-readable device;

[0122] Processor: Used to implement the target long-term trajectory prediction method based on dynamic graph structure cellular long short-term memory network when executing the computer program.

[0123] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the target long-term trajectory prediction method based on a dynamic graph structure cellular long short-term memory network.

Claims

1. A target long-time trajectory prediction method based on a dynamic graph structure cell long short-term memory network, characterized in that, The method comprises the following steps: Step 1, detecting the input video using a 3D target detector and a 3D target tracker to obtain 3D motion information features of all targets in the video; Step 2, performing uniform standardization processing on the 3D motion information features of all targets in the video detected in step 1 to obtain a motion target information feature dataset; Step 3, extracting each scene from the motion target information feature dataset obtained in step 2, representing each time point in each scene as a graph, wherein the target ID in each time point is taken as a node in the graph, the predicted motion target in the target ID is taken as a main node in the graph, and other motion targets in the target ID are taken as other nodes in the graph, and the main node and the other nodes are constructed into a graph adjacency matrix reflecting the motion space; Step 4, taking the target information features in each time point in each scene in the motion target information feature dataset standardized in step 2 as the initial input feature matrix of the graph convolution network and the graph attention mechanism, providing the 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in step 3, updating each node feature in the graph adjacency matrix using a spatial graph feature extraction module (SGFE), and outputting implicit spatial motion structure information including the motion information between targets in the motion space; Step 5, inputting the implicit spatial motion structure information output in step 4 into a dynamic graph structure cell long short-term memory network (TSGCell-LSTM), processing the time sequence information and the spatial interaction information in the motion space simultaneously through the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and finally outputting long-time trajectory information of the predicted motion target, including future 3D motion position information of the predicted motion target; The specific process of step 5 is as follows: The dynamic graph structure cell long short-term memory network (TSGCell-LSTM) comprises a standard memory unit, a spatial information memory unit and a forgetting unit; wherein the standard memory unit is used to store the state information updated with time steps, and is responsible for deciding the update and retention of information according to the input data and the historical state of the network at each time point; the spatial information memory unit enhances the spatial information memory capability of the standard LSTM network, jointly extracts the time sequence information and spatial information of the targets in the motion space, and updates the interaction information at each time point in the motion space; the forgetting unit is used to help the network to clear irrelevant or outdated information, so as to retain only important feature information; the specific implementation steps of the dynamic graph structure cell long short-term memory network (TSGCell-LSTM) are as follows: Step 5.1, feature input: inputting the implicit spatial motion structure information obtained in step 4 into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM); Step 5.2, spatial information screening: performing a part of screening and transmission on the implicit interaction information in the motion space input in step 5.1, and the information screening method is as follows: Gr(t) = sigmoid(net G (t)) (5) wherein G t is the spatial interaction information of the current frame, h t-1 is the information passed from the previous frame, and Gr(t) is the filtered spatial information; Step 5.3, information update: the spatial information Gr(t) screened in step 5.2 enters the spatial information memory unit in the dynamic graph structure cell long short-term memory network (TSG Cell-LSTM), which strengthens the memory of the current spatial structure information through a memory door-like operation, and the information transmission mode is represented as: C t = C t1 + I(t) * R(t) * Gr(t) (6) wherein C t1 is the memory information after the standard memory unit and the forgetting unit, C t is the memory information updated by the spatial information; Step 5.4, output update result: add a bias term to the updated memory information in step 5.3, and output the iterative update result; Step 5.5, the output update result in step 5.4 enters the last fully connected layer, and regression is performed to obtain the future 3D motion position information of the predicted moving target.

2. The target long-time trajectory prediction method based on a dynamic graph structure cell long short-term memory network according to claim 1, characterized in that, In step 1, the 3D motion information features include but are not limited to position coordinate information, velocity information, and acceleration information. 3.The target long-time trajectory prediction method based on dynamic graph structure cell-LSTM network according to claim 1, wherein, In step 2, the specific standardization processing steps are as follows: Step 2.1, coordinate conversion: convert the 3D motion information features of all targets in the video obtained by the 3D target detector and the 3D target tracker in step 1 from the camera coordinate system to the virtual coordinate system; Step 2.2, normalization: normalize the 3D motion information features of the predicted moving target in all targets converted in step 2.1, take the center point of the virtual coordinate system as the origin, and center the moving target, and at the same time, normalize other moving targets in all targets to obtain different moving target information feature datasets and construct a moving target information feature dataset; Step 2.3, data segmentation and enhancement: according to the task requirements, the moving target information feature dataset obtained in step 2.2 is segmented, and the segmented moving target information feature dataset is down-sampled, rotated, translated, and scaled to obtain an enhanced moving target information feature dataset.

4. The target long-time trajectory prediction method based on a dynamic graph structure cell long short-term memory network according to claim 1, characterized in that, The construction steps of step 3 are as follows: Step 3.1, taking the moving target information feature dataset obtained in step 2.3 as the input of the dataset scene extraction module, extract each scene from the moving target information feature dataset; Step 3.2, each time in each scene obtained in step 3.1 is represented as a graph, wherein the target ID in each time is taken as a node in the graph, and the predicted moving target in the target ID is taken as the main node in the graph, and the other moving targets in the target ID are taken as other nodes in the graph; Step 3.3, according to the main node and other nodes in step 3.2, traverse the motion target information features of each scene in step 3.1 to extract the connection relationship between the predicted moving target and other moving targets, and construct the connection relationship between the main node and other nodes at each time; Step 3.4, convert the main node and other nodes in each time obtained in step 3.2 and the connection relationship between the main node and other nodes at each time constructed in step 3.3 into a sparse matrix, wherein the rows and columns of the matrix correspond to the target IDs of the moving targets, and the values of the matrix represent whether there is a connection between the corresponding nodes; according to the connection of the nodes, the graph adjacency matrix is converted into a symmetric matrix; Step 3.5, normalize the graph adjacency matrix after the symmetry in step 3.4, to obtain a graph adjacency matrix reflecting the motion space.

5. The target long-time trajectory prediction method based on dynamic graph structure cell long short-term memory network according to claim 1, characterized in that, The specific process of step 4 is as follows: In step 4, the spatial graph feature extraction module (SGFE) includes an input layer, an intermediate layer and an output layer; the input layer is composed of a graph convolution layer and a GELU activation function; the intermediate layer is composed of a graph convolution layer and a graph attention mechanism layer in parallel, wherein the graph convolution layer is used for further extraction of spatial information, and the graph attention mechanism layer is used for screening key dimensional features of spatial information; the output layer includes a fully connected layer for generating implicit spatial structure information; Step 4.1, the target information features in each time in each scene in the motion target information feature dataset after standardization in step 2 are constructed into an initial input feature matrix, the graph convolution layer of the input layer of the spatial graph feature extraction module (SGFE) is used to perform convolution operation on the initial input feature matrix, to extract motion features in the local neighborhood, and the GELU activation function is used to introduce non-linear characteristics for convolution output, to enhance the expression ability of the model, ensure effective learning of the features, and finally obtain an initial hidden space information feature matrix; Step 4.2, the initial hidden space information feature matrix obtained in step 4.1 is taken as the input of the intermediate layer, wherein the graph convolution layer continues to further extract spatial information of the initial hidden space information feature, to maintain information flow between nodes, to obtain an intermediate layer hidden space information feature matrix; the graph attention mechanism layer calculates the attention weight between nodes according to the intermediate layer hidden space information feature and the graph adjacency matrix obtained in step 3, to selectively emphasize important features, and extracts key dimensional features in the motion space through weighted feature fusion; Step 4.3, the intermediate layer hidden space information feature matrix and the key dimensional features in the motion space obtained in step 4.2 are integrated to ensure that the output dimension matches the task requirements, and the implicit spatial motion structure information is converted through a fully connected layer.

6. A dynamic graph structure based cell long short-term memory network based target long-term trajectory prediction system based on the method of any one of claims 1 to 5, characterized in that, Comprise: The 3D motion information feature acquisition module is used for detecting the input video using a 3D target detector and a 3D target tracker, and acquiring 3D motion information features of all targets in the video; The motion target information feature dataset acquisition module is used for performing uniform standardization processing on the detected 3D motion information features of all targets in the video to obtain a motion target information feature dataset; The graph adjacency matrix construction module is used for extracting each scene from the motion target information feature dataset, representing each time in each scene as a graph, wherein the target ID in each time is taken as a node in the graph, the predicted motion target in the target ID is taken as a main node in the graph, and other motion targets in the target ID are taken as other nodes in the graph, and the main node and the other nodes are constructed into a graph adjacency matrix reflecting the motion space; The implicit spatial motion structure information acquisition module is configured to take the target information features in each time instant in each scene in the standardized motion target information feature dataset as initial input feature matrices of a graph convolution network and a graph attention mechanism, provide 3D motion information features for a graph adjacency matrix reflecting a motion space, update each node feature in the graph adjacency matrix by using a spatial graph feature extraction module (SGFE), and output implicit spatial motion structure information including motion information between targets in the motion space; The long-time trajectory information output module is configured to input the implicit spatial motion structure information into a dynamic graph structure cell long short-term memory network (TSGCell-LSTM), process time sequence information and spatial interaction information in the motion space simultaneously by using the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and finally output long-time trajectory information of a predicted motion target, including future 3D motion position information of the predicted motion target.

7. A target long-time trajectory prediction device based on a dynamic graph structure cell long short-term memory network, characterized by, The long-time trajectory prediction method based on the dynamic graph structure cell long short-term memory network comprises the following steps: a memory configured to store a computer program of the long-time trajectory prediction method based on the dynamic graph structure cell long short-term memory network according to any one of claims 1-5, and the computer program is readable by the device; a processor configured to implement the long-time trajectory prediction method based on the dynamic graph structure cell long short-term memory network according to any one of claims 1-5 when the computer program is executed.

8. A computer-readable storage medium, characterized in that, A computer readable storage medium stores a computer program, and the computer program can implement the long-time trajectory prediction method based on the dynamic graph structure cell long short-term memory network according to any one of claims 1-5 when the computer program is executed by a processor.

Images