Target long-time trajectory prediction method, system and device based on dynamic graph structure cell long-short-term memory network and medium

Abstract

The invention relates to a target long-time trajectory prediction method, system and device based on a dynamic graph structure cell long-short-term memory network and a medium. According to the method, effectiveness of interaction information of a target interaction structure graph in a motion space is represented, and implicit features of a graph convolutional neural network transmission main target are combined; according to a feature screening mechanism of a graph attention mechanism, paying attention to information features of different dimensions, and increasing a spatial information memory unit to enhance the spatial structure information processing capability of a long short-term memory network (LSTM), so that the improved LSTM network can process time sequence information and spatial information of a motion space at the same time; the system, the equipment and the medium are used for implementing the method. According to the method, through the characteristics of the graph convolution algorithm, the graph attention mechanism and the LSTM network, the problem of track end point prediction fuzziness in target long-time track prediction can be effectively solved, a better effect can be achieved, and a smaller error exists between a prediction result and a true value.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data quantization, and in particular to a method, system, device and medium for predicting target long-term trajectory based on dynamic graph structured cell long short-term memory network. Background Art

[0002] The target long-term trajectory prediction technology aims to predict the position of the target in three-dimensional space from video or radar sequences, and is widely used in the fields of autonomous driving, vehicle-road collaboration, target tracking, etc. With the development of deep learning technology, significant progress has been made in this field.

[0003] However, long-term target trajectory prediction still faces the following major challenges: 1. The information in the motion space cannot be fully applied to target trajectory prediction; 2. In the current prediction method based on graph convolutional networks, the target motion information in the motion space is represented by a spatial graph structure, but the information on the time series is relatively lacking, so it cannot take into account both spatial interaction information and time series information, which may lead to an increase in long-term trajectory prediction errors.

[0004] To solve the above problems, researchers use graph convolutional networks (GCN) to achieve target trajectory prediction. Compared with traditional convolutional neural networks, GCN extends the convolution operation from traditional image processing to graph data processing. The key idea is to create a mapping function that can extract interaction-aware features from node features in the network and their neighboring nodes, and consider interaction-related factors by considering the moving target as a node in the network at each sampling time. In addition, GCN provides a promising approach for trajectory prediction by explicitly modeling spatial dependencies and relationships between objects. They can effectively capture contextual information and handle irregular graph structures. However, there are still some shortcomings when applying GCN to trajectory prediction tasks. First, trajectory prediction tasks usually involve dynamically changing graph structures, such as the position and speed of moving objects changing over time. GCN performs well in processing static graph structures, but when processing dynamic graphs, additional mechanisms may be needed to capture changes in time series data. Second, although GCN can capture local connection patterns between nodes, it may not be enough to capture complex spatial relationships, especially in high-dimensional spaces. This may require the combination of other techniques, such as attention mechanisms or transformers, to better understand spatial dependencies. Finally, GCNs may perform well on training data, but their generalization ability may be limited when faced with unseen data or environments. This is because GCNs usually rely on the local structure of the graph, while in trajectory prediction, global contextual information is equally important for making accurate predictions. Therefore, it may be necessary to design more complex models to improve the generalization ability of GCNs in trajectory prediction tasks.

[0005] The patent application document with publication number CN118264670A discloses a service migration method based on priority and spatiotemporal trajectory prediction. By constructing a temporal user interaction relationship trajectory graph, the spatiotemporal trajectory graph is processed by a Transformer model and the feature representation output is concatenated to obtain a feature vector, and an MLP decoder is used to obtain the user's future movement trajectory. However, due to the complex structure of the Transformer model, many hyperparameters need to be adjusted, resulting in high computational cost and large processing delay.

[0006] The patent application document with publication number CN118261051A discloses a method for constructing a pedestrian and vehicle trajectory prediction model at an intersection based on a heterogeneous graph network. The method adopts a heterogeneous graph structure to characterize the spatial relationship between heterogeneous traffic individuals, and converts it into physical knowledge hidden features through a heterogeneous graph edge convolution method. However, since this method mainly focuses on the movement trajectory of vehicles and does not fully consider the influence of external factors such as interactions between vehicles, it makes it difficult to accurately capture social behavior patterns. Summary of the invention

[0007] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a target long-term trajectory prediction method, system, device and medium based on dynamic graph structured cell long short-term memory network, by constructing a spatial information memory unit, jointly extracting the target timing information and spatial information in the motion space, and comprehensively analyzing the target historical information and status. It can combine the historical interaction information and timing information of the target with long-term information loss, and has the function of accurately predicting the long-term future trajectory range of the target.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A method for predicting target long-term trajectory based on dynamic graph structured cell long short-term memory network, comprising the following steps:

[0010] Step 1: Use a 3D object detector and a 3D object tracker to detect the input video and obtain the 3D motion information features of all objects in the video;

[0011] Step 2, performing unified standardization processing on the 3D motion information features of all targets in the video detected in step 1 to obtain a moving target information feature data set;

[0012] Step 3, extract each scene from the moving target information feature data set obtained in step 2, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space;

[0013] Step 4: Use the target information features at each moment in each scene in the motion target information feature dataset standardized in step 2 as the initial input feature matrix of the graph convolution network and the graph attention mechanism, provide 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in step 3, and use the spatial graph feature extraction module (SGFE) to update the features of each node in the graph adjacency matrix, and output implicit spatial motion structure information, including the motion information between the targets in the motion space;

[0014] In step 5, the implicit spatial motion structure information outputted in step 4 is inputted into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The dynamic graph structure cell long short-term memory network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

[0015] In step 1, the 3D motion information features include but are not limited to position coordinate information, speed information, and acceleration information.

[0016] In step 2, the specific standardization processing steps are as follows:

[0017] Step 2.1, coordinate conversion: convert the 3D motion information features of all targets in the video obtained by the 3D target detector and 3D target tracker in step 1 from the camera coordinate system to the virtual coordinate system;

[0018] Step 2.2, normalization: normalize the 3D motion information features of the predicted moving targets among all the targets converted in step 2.1, take the center point of the virtual coordinate system as the origin, and perform central processing on the moving targets. At the same time, perform the same normalization processing on other moving targets among all the targets to obtain different moving targets and their information features, and construct a moving target information feature data set;

[0019] Step 2.3, data segmentation and enhancement: Segment the motion target information feature data set obtained in step 2.2 according to the task requirements, downsample the segmented motion target information feature data set, rotate, translate and scale the downsampled motion target information feature data set to obtain an enhanced motion target information feature data set.

[0020] The construction steps of step 3 are as follows:

[0021] Step 3.1, using the moving target information feature data set obtained in step 2.3 as the input of the data set scene extraction module, extracting each scene from the moving target information feature data set;

[0022] Step 3.2, each moment in each scene obtained in step 3.1 is represented as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph;

[0023] Step 3.3, according to the main node and other nodes in step 3.2, traverse the moving target information features of each scene in step 3.1, extract the connection relationship between the predicted moving target and other moving targets, and construct the connection relationship between the main node and other nodes at each moment;

[0024] Step 3.4, convert the connection relationship between the main node and other nodes at each moment obtained in step 3.2 and the main node and other nodes at each moment constructed in step 3.3 into a sparse matrix, where the rows and columns of the matrix correspond to the target ID of the moving target, and the value of the matrix indicates whether there is a connection between the corresponding nodes; according to the connection status of the nodes, the graph adjacency matrix is ​​converted into a symmetric matrix;

[0025] Step 3.5, normalize the graph adjacency matrix after symmetry in step 3.4 to obtain a graph adjacency matrix reflecting the motion space.

[0026] The specific process of step 4 is as follows:

[0027] In step 4, the spatial graph feature extraction module (SGFE) includes an input layer, an intermediate layer, and an output layer; the input layer consists of a graph convolution layer and a GELU activation function; the intermediate layer consists of a graph convolution layer and a graph attention mechanism layer in parallel, wherein the graph convolution layer is used to further extract spatial information, and the graph attention mechanism layer is used to filter the key dimensional features of spatial information; the output layer includes a fully connected layer for generating implicit spatial structure information;

[0028] Step 4.1, construct the target information features in each moment of each scene in the moving target information feature data set standardized in step 2 as the initial input feature matrix, use the graph convolution layer of the input layer of the spatial graph feature extraction module (SGFE) to perform convolution operation on the initial input feature matrix, extract the motion features in the local neighborhood, and use the GELU activation function to introduce nonlinear characteristics to the convolution output, enhance the expression ability of the model, ensure the effective learning of the features, and finally obtain the initial hidden spatial information feature matrix;

[0029] Step 4.2: The initial hidden spatial information feature matrix obtained in step 4.1 is used as the input of the middle layer, where the graph convolution layer continues to further extract spatial information from the initial hidden spatial information features, maintains the information flow between nodes, and obtains the middle layer hidden spatial information feature matrix; the graph attention mechanism layer calculates the attention weights between nodes based on the middle layer hidden spatial information features and the graph adjacency matrix obtained in step 3, so as to selectively emphasize important features, and extract the key dimensional features in the motion space through weighted feature fusion;

[0030] In step 4.3, the feature matrix of the hidden spatial information obtained in step 4.2 and the key dimensional features in the motion space are integrated to ensure that the output dimension matches the task requirements, and converted into implicit spatial motion structure information through the fully connected layer.

[0031] The specific process of step 5 is as follows:

[0032] The dynamic graph structured cell long short-term memory network (TSGCell-LSTM) includes a standard memory unit, a spatial information memory unit and a forgetting unit; among them, the standard memory unit is used to store state information updated with time steps, and is responsible for determining the update and retention of information according to the input data and the historical state of the network at each moment; the spatial information memory unit enhances the spatial information memory ability of the standard LSTM network, jointly extracts the target time sequence information and spatial information in the motion space, and updates the interactive information at each moment in the motion space; the forgetting unit is used to help the network clear irrelevant or outdated information so that the network only retains important feature information; the specific implementation steps of the dynamic graph structured cell long short-term memory network (TSGCell-LSTM) are:

[0033] Step 5.1, feature input: input the implicit spatial motion structure information obtained in step 4 into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM);

[0034] Step 5.2, spatial information screening: Part of the implicit interaction information in the motion space input in step 5.1 is screened and transmitted. The information screening method is as follows:

[0035]

[0036] Gr(t)=sigmoid(net G (t)) (5)

[0037] Among them, G t is the spatial interaction information of the current frame, h t-1 is the information transmitted in the previous frame, and Gr(t) is the filtered spatial information;

[0038] Step 5.3, information update: The spatial information Gr(t) filtered in step 5.2 enters the spatial information memory unit in the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The spatial information memory unit strengthens the memory of the current spatial structure information through an operation similar to a memory gate. The information transmission method is expressed as:

[0039] C t =C t1 +I(t)*R(t)*Gr(t) (6)

[0040] Among them, C t1 is the memory information after the standard memory unit and the forgetting unit, C t It is the memory information updated by spatial information;

[0041] Step 5.4, output the updated result: add a bias term to the memory information updated in step 5.3, and output the iterative update result;

[0042] Step 5.5: The updated result output in step 5.4 is input into the last fully connected layer, and the future 3D motion position information of the predicted moving target is obtained through regression.

[0043] The present invention also provides a target long-term trajectory prediction system based on a dynamic graph structured cell long short-term memory network, comprising:

[0044] The 3D motion information feature acquisition module is used to detect the input video using a 3D object detector and a 3D object tracker to obtain the 3D motion information features of all objects in the video;

[0045] A moving target information feature data set acquisition module is used to perform unified standardization processing on the 3D motion information features of all targets in the detected video to obtain a moving target information feature data set;

[0046] A graph adjacency matrix construction module is used to extract each scene from the moving target information feature data set, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space;

[0047] The implicit spatial motion structure information acquisition module is used to use the target information features in each scene at each moment in the standardized motion target information feature dataset as the initial input feature matrix of the graph convolution network and the graph attention mechanism, provide 3D motion information features for the graph adjacency matrix reflecting the motion space, and use the spatial graph feature extraction module (SGFE) to update the features of each node in the graph adjacency matrix, and output the implicit spatial motion structure information, including the motion information between the targets in the motion space;

[0048] The long-term trajectory information output module of the moving target is used to input the implicit spatial motion structure information into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and simultaneously process the temporal information and spatial interaction information in the motion space through the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and finally output the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

[0049] The present invention also provides a target long-term trajectory prediction device based on a dynamic graph structure cell long short-term memory network, comprising:

[0050] Memory: a computer program for storing the above-mentioned method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network, which is a computer-readable device;

[0051] Processor: used to implement the target long-term trajectory prediction method based on dynamic graph structure cell long short-term memory network when executing the computer program.

[0052] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the target long-term trajectory prediction method based on a dynamic graph structured cell long short-term memory network.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) The present invention proposes a target long-term trajectory prediction method based on a dynamic graph structured cell long short-term memory network. By jointly extracting the target temporal information and spatial information in the motion space, the interactive information at each moment in the motion space is updated and the trajectory is predicted, which can achieve a high degree of fusion of spatial information and temporal information in the prediction process.

[0055] (2) The present invention proposes a novel spatial implicit interaction feature extraction method. By combining a graph convolutional network with a graph attention mechanism, the features of each node in the graph are updated. This method can more effectively extract and utilize spatial information, and at the same time, it proposes an effective solution to the difficulty of graph convolutional networks in mining spatial hidden layer information. In addition, this algorithm can consider the motion interaction of each target in the motion space, thereby accurately extracting the interaction information between targets.

[0056] (3) The present invention proposes a novel long-term target prediction method based on spatiotemporal information features. By enhancing the spatial information memory capability of the standard LSTM network, the interactive information in the motion space can be processed and updated more effectively. At the same time, the spatial structure information and time series information in the motion space can be used to accurately predict the long-term trajectory of the moving target, and compared with the Transformer, the amount of calculation can be reduced.

[0057] In summary, the present invention effectively improves the accuracy and reliability of long-term trajectory prediction through innovative feature extraction and model design, and provides a new solution for target motion prediction in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a method for predicting target long-term trajectory based on a dynamic graph structured cell long short-term memory network of the present invention.

[0059] Figure 2 It is a spatial graph feature extraction module (SGFE) built based on the principle of the present invention.

[0060] Figure 3 It is a dynamic graph structured cell long short-term memory network (TSGCell-LSTM) built by benfam based on the principle.

[0061] Figure 4 It is a diagram of experimental results using a specific implementation method of the present method. DETAILED DESCRIPTION

[0062] The specific use method of the present invention is described below in conjunction with the accompanying drawings so that users can better understand the present invention. It should be noted that in the following description, the detailed description of existing functions and methods will be downplayed. The present invention mainly focuses on the target long-term trajectory prediction method.

[0063] Figure 1 This is a flow chart of a method for predicting a target's long-term trajectory based on a dynamic graph structured cell long short-term memory network in a specific implementation of the present invention. According to this method, a network model is built for the application of long-term trajectory prediction of a target.

[0064] In this embodiment, if Figure 1 As shown, a method for predicting target long-term trajectory based on dynamic graph structured cell long short-term memory network includes the following steps:

[0065] Step 1: Use a 3D object detector and a 3D object tracker to detect the input video and obtain the 3D motion information features of all objects in the video;

[0066] The 3D motion information features include but are not limited to position coordinate information, speed information, and acceleration information.

[0067] In step 1, the 3D target detector is a fully convolutional single-stage 3D target detection (FCOS-LiDAR) model; the video is used as the input of the 3D target detector, and the moving target ID and its 3D motion coordinates in the video are obtained, which are used in the subsequent long-term trajectory prediction step.

[0068] In the step 1, the 3D target tracker is an EagerMOT model; the video is used as the input of the 3D target tracker to obtain the 3D motion information of the moving target in the video, which is used in the subsequent long-term trajectory prediction step.

[0069] Step 2, performing unified standardization processing on the 3D motion information features of all targets in the video detected in step 1 to obtain a moving target information feature data set;

[0070] In step 2, the specific standardization processing steps are as follows:

[0071] Step 2.1, coordinate conversion: convert the 3D motion information features of all targets in the video obtained by the 3D target detector and 3D target tracker in step 1 from the camera coordinate system to the virtual coordinate system;

[0072] Step 2.2, normalization: normalize the 3D motion information features of the predicted moving targets among all the targets converted in step 2.1, take the center point of the virtual coordinate system as the origin, and perform central processing on the moving targets. At the same time, perform the same normalization processing on other moving targets among all the targets to obtain different moving targets and their information features, and construct a moving target information feature data set;

[0073] Step 2.3, data segmentation and enhancement: Segment the motion target information feature data set obtained in step 2.2 according to the task requirements, downsample the segmented motion target information feature data set, rotate, translate and scale the downsampled motion target information feature data set to obtain an enhanced motion target information feature data set.

[0074] Data augmentation is to increase the diversity of training data used for neural networks, thereby enhancing the robustness of the model under different conditions.

[0075] Step 3, extract each scene from the moving target information feature data set obtained in step 2, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space;

[0076] The construction steps of step 3 are as follows:

[0077] Step 3.1, using the moving target information feature data set obtained in step 2.3 as the input of the data set scene extraction module, extracting each scene from the moving target information feature data set;

[0078] Step 3.2, each moment in each scene obtained in step 3.1 is represented as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph;

[0079] Step 3.3, according to the main node and other nodes in step 3.2, traverse the moving target information features of each scene in step 3.1, extract the connection relationship between the predicted moving target and other moving targets, and construct the connection relationship between the main node and other nodes at each moment;

[0080] Step 3.4, convert the connection relationship between the main node and other nodes at each moment obtained in step 3.2 and the main node and other nodes at each moment constructed in step 3.3 into a sparse matrix, where the rows and columns of the matrix correspond to the target ID of the moving target, and the value of the matrix indicates whether there is a connection between the corresponding nodes; according to the connection status of the nodes, the graph adjacency matrix is ​​converted into a symmetric matrix to reflect the bidirectional connection between the predicted moving target and other moving targets;

[0081] Step 3.5, normalize the graph adjacency matrix after symmetry in step 3.4 to obtain a graph adjacency matrix reflecting the motion space; to ensure the stability and value range of the graph adjacency matrix.

[0082] In step 4, the target information features at each moment in each scene in the motion target information feature dataset standardized in step 2 are used as the initial input feature matrix of the graph convolutional network and the graph attention mechanism, and 3D motion information features are provided for the graph adjacency matrix reflecting the motion space obtained in step 3. The spatial graph feature extraction module (SGFE) is used to update the features of each node (i.e., each motion target) in the graph adjacency matrix (i.e., the motion information features of each motion target), and output implicit spatial motion structure information, including the motion information between the targets in the motion space, so as to achieve accurate extraction of the interaction information of the motion targets in the input video.

[0083] The specific process of step 4 is as follows:

[0084] like Figure 2 As shown, in step 4, the spatial graph feature extraction module (SGFE) includes an input layer, an intermediate layer and an output layer; the input layer is composed of a graph convolution layer and a GELU activation function; the intermediate layer is composed of a graph convolution layer and a graph attention mechanism layer in parallel, wherein the graph convolution layer is used to further extract spatial information, and the graph attention mechanism layer is used to filter the key dimensional features of spatial information; the output layer includes a fully connected layer for generating implicit spatial structure information;

[0085] Step 4.1, construct the target information features in each moment of each scene in the moving target information feature data set standardized in step 2 as the initial input feature matrix, use the graph convolution layer of the input layer of the spatial graph feature extraction module (SGFE) to perform convolution operation on the initial input feature matrix, extract the motion features in the local neighborhood, and use the GELU activation function to introduce nonlinear characteristics to the convolution output, enhance the expression ability of the model, ensure the effective learning of the features, and finally obtain the initial hidden spatial information feature matrix;

[0086] Step 4.2: The initial hidden spatial information feature matrix obtained in step 4.1 is used as the input of the middle layer, where the graph convolution layer continues to further extract spatial information from the initial hidden spatial information features, maintains the information flow between nodes, and obtains the middle layer hidden spatial information feature matrix; (Increasing the number of convolution layers can further improve the depth of feature extraction.) The graph attention mechanism layer calculates the attention weights between nodes based on the middle layer hidden spatial information features and the graph adjacency matrix obtained in step 3 to selectively emphasize important features, and extracts the key dimensional features in the motion space through weighted feature fusion;

[0087] In step 4.3, the feature matrix of the hidden spatial information obtained in step 4.2 and the key dimensional features in the motion space are integrated to ensure that the output dimension matches the task requirements, and converted into implicit spatial motion structure information through the fully connected layer.

[0088] In step 5, the implicit spatial motion structure information outputted from step 4 is inputted into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The dynamic graph structure cell long short-term memory network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target, thereby realizing accurate prediction of the trajectory of the main moving target in the input video.

[0089] The specific process of step 5 is as follows:

[0090] like Figure 3 As shown in the figure, the dynamic graph structured cell long short-term memory network (TSGCell-LSTM) includes a standard memory unit, a spatial information memory unit and a forgetting unit; wherein the standard memory unit is used to store state information updated with time steps, and is responsible for determining the update and retention of information according to the input data and the historical state of the network at each moment; the spatial information memory unit enhances the spatial information memory ability of the standard LSTM network, jointly extracts the target time sequence information and spatial information in the motion space, and updates the interactive information at each moment in the motion space; the forgetting unit is used to help the network clear irrelevant or outdated information so that the network only retains important feature information; the specific implementation steps of the dynamic graph structured cell long short-term memory network (TSGCell-LSTM) are:

[0091] Step 5.1, feature input: input the implicit spatial motion structure information obtained in step 4 into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM);

[0092] Step 5.2, spatial information screening: Part of the implicit interaction information in the motion space input in step 5.1 is screened and transmitted. The information screening method is as follows:

[0093]

[0094] Gr(t)=sigmoid(net G (t)) (5)

[0095] Among them, G t is the spatial interaction information of the current frame, h t-1 is the information transmitted in the previous frame, and Gr(t) is the filtered spatial information;

[0096] Step 5.3, information update: The spatial information Gr(t) filtered in step 5.2 enters the spatial information memory unit in the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The spatial information memory unit strengthens the memory of the current spatial structure information through an operation similar to a memory gate. The information transmission method is expressed as:

[0097] C t =C t1 +I(t)*R(t)*Gr(t) (6)

[0098] Among them, C t1 is the memory information after the standard memory unit and the forgetting unit, C t It is the memory information updated by spatial information;

[0099] Step 5.4, output the updated result: add a bias term to the memory information updated in step 5.3, and output the iterative update result;

[0100] Step 5.5: The updated result output in step 5.4 is input into the last fully connected layer, and the future 3D motion position information of the predicted moving target is obtained through regression.

[0101] The above steps use the TSGCell-LSTM network to update the motion features of the main object in the image. In the TSGCell-LSTM network, multiple cycles of iterative updates are implemented, and the optimal value is continuously sought through deep learning training.

[0102] Experimental testing

[0103] Experimental conditions

[0104] The present invention uses experimental simulation based on real data sets, and adopts the "DAIR-V2X series data set", which is a vehicle and infrastructure collaborative time series perception data set in real scenarios. At the same time, mean square error (MSE), average displacement error (ADE) and final displacement error (FDE) are used as evaluation indicators to measure the average position distance and destination distance between the predicted trajectory and the ground truth (GT) trajectory respectively. In the experiment, the experiment is set to use the past 40 frames of data as the observation trajectory and predict the trajectory data of the next 40 frames.

[0105] Experimental data

[0106] This experiment is based on the "DAIR-V2X vehicle-road collaboration dataset", which is a real-world vehicle-infrastructure collaborative temporal perception dataset, containing about 100 segments of images and point cloud sequences from the vehicle-side and road-side perspectives, as well as corresponding 3D target boxes and tracking ID annotations. The data was collected from the complex traffic intersections in Beijing's high-level autonomous driving demonstration area, involving equipment such as 300-line LiDAR and 1-inch global exposure CMOS cameras, achieving GPS timing synchronization and internal and external parameter calibration. The dataset also contains detailed calibration and coordinate system conversion information, as well as vehicle-road collaborative 3D annotations based on spatiotemporal synchronization algorithms and trajectory matching algorithms, providing rich data support for this experimental study.

[0107] Experimental Results

[0108] After experiments, compared with most target trajectory prediction methods, such as SGAN, MALS-Net, Social-LSTM, Bi-LSTM and other methods, the squared prediction error (MSE), average distance error (ADE) and final prediction error (FDE) between the test value and the true value are calculated. In terms of long-term prediction, the present invention shows significant advantages. Long-term predictions are usually faced with increased uncertainty in the target trajectory and the impact of cumulative errors, but the present invention is still superior to the existing technology in terms of prediction accuracy and stability, and can effectively track the trend changes of target motion in long-term series and slow down the growth of cumulative errors. Figure 4 This is particularly evident in the multi-frame prediction results shown in FIG. 1 : when the number of predicted frames reaches 20 frames or longer, the trajectory prediction of the present invention is still close to the actual path. Therefore, the present invention can effectively track the trend change of the target motion in a long time series and slow down the growth of the cumulative error. The experimental results are shown in FIG. Figure 4 As shown, the blue line is the historical x-coordinate of the target to be predicted, the orange line is the historical y-coordinate of the target to be predicted, the purple line is the actual x-coordinate of the target to be predicted, the brown line is the actual y-coordinate of the target to be predicted, the green line is the predicted x-coordinate of the target to be predicted, and the red line is the predicted y-coordinate of the target to be predicted.

[0109] In addition, as shown in Table 1, each row of data is the MSE of 50 frames of predicted trajectories output by different methods based on 50 frames of historical motion data in 1000 motion scenes. The smaller the value, the better the effect. After using the test set of the dataset for verification, different motion scenes in the dataset are tested, and the squared prediction error (MSE), average distance error (ADE) and final prediction error (FDE) between the test value and the true value are calculated. After comparison with the classic method, the effectiveness of this method is confirmed.

[0110] Through the comparative experimental results in Table 1, each row of data is the mean square error (MSE), average distance error (ADE) and final prediction error (FDE) of different methods under 50 frames of prediction results. The smaller the result value of the three indicators, the better the effect. The present invention (TSGCell-LSTM) has significant advantages in long-term prediction. The three indicators of MSE, ADE and FDE are significantly lower than those of existing methods such as SGAN, MALS-Net, Social-LSTM, etc., indicating that it has higher accuracy for trajectory in long-term prediction. Therefore, the present invention effectively controls the cumulative error and adapts to multi-target interference and nonlinear motion in complex scenes, so that it is closer to the real trajectory over a long time span, showing good application potential and creativity.

[0111] Table 1. Experimental results

[0112]

[0113] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

[0114] The present invention also provides a target long-term trajectory prediction system based on a dynamic graph structured cell long short-term memory network, comprising:

[0115] A 3D motion information feature acquisition module is used to implement the detection of the input video using a 3D object detector and a 3D object tracker in step 1 to obtain the 3D motion information features of all objects in the video;

[0116] A moving target information feature data set acquisition module is used to implement the unified standardization processing of the 3D motion information features of all targets in the video detected in step 1 in step 2 to obtain a moving target information feature data set;

[0117] A graph adjacency matrix construction module is used to extract each scene from the moving target information feature data set obtained in step 2 in step 3, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space;

[0118] The implicit spatial motion structure information acquisition module is used to implement the target information features in each scene at each moment in the motion target information feature data set standardized in step 2 as the initial input feature matrix of the graph convolution network and the graph attention mechanism in step 4, provide 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in step 3, use the spatial graph feature extraction module (SGFE) to update the features of each node in the graph adjacency matrix, and output implicit spatial motion structure information, including the motion information between each target in the motion space;

[0119] The long-term trajectory information output module of the moving target is used to implement the input of the implicit spatial motion structure information output in step 4 into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM) in step 5, and the dynamic graph structure cell long short-term memory network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

[0120] The present invention also provides a target long-term trajectory prediction device based on a dynamic graph structure cell long short-term memory network, comprising:

[0121] Memory: a computer program for storing the above-mentioned method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network, which is a computer-readable device;

[0122] Processor: used to implement the target long-term trajectory prediction method based on dynamic graph structure cell long short-term memory network when executing the computer program.

[0123] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the target long-term trajectory prediction method based on a dynamic graph structured cell long short-term memory network.

Claims

1. A target long-term trajectory prediction method based on dynamic graph structured cell long short-term memory network, characterized in that: The following steps are involved: Step 1: Use a 3D object detector and a 3D object tracker to detect the input video and obtain the 3D motion information features of all objects in the video; Step 2, performing unified standardization processing on the 3D motion information features of all targets in the video detected in step 1 to obtain a moving target information feature data set; Step 3, extract each scene from the moving target information feature data set obtained in step 2, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space; Step 4: Use the target information features at each moment in each scene in the motion target information feature dataset standardized in step 2 as the initial input feature matrix of the graph convolution network and the graph attention mechanism, provide 3D motion information features for the graph adjacency matrix reflecting the motion space obtained in step 3, and use the spatial graph feature extraction module (SGFE) to update the features of each node in the graph adjacency matrix, and output implicit spatial motion structure information, including the motion information between the targets in the motion space; In step 5, the implicit spatial motion structure information outputted in step 4 is inputted into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The dynamic graph structure cell long short-term memory network (TSGCell-LSTM) simultaneously processes the temporal information and spatial interaction information in the motion space, and finally outputs the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

2. According to claim 1, a method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network is characterized in that: In step 1, the 3D motion information features include but are not limited to position coordinate information, speed information, and acceleration information.

3. According to claim 1, a method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network is characterized in that: In step 2, the specific standardization processing steps are as follows: Step 2.1, coordinate conversion: convert the 3D motion information features of all targets in the video obtained by the 3D target detector and 3D target tracker in step 1 from the camera coordinate system to the virtual coordinate system; Step 2.2, normalization: normalize the 3D motion information features of the predicted moving targets among all the targets converted in step 2.1, take the center point of the virtual coordinate system as the origin, and perform central processing on the moving targets. At the same time, perform the same normalization processing on other moving targets among all the targets to obtain different moving targets and their information features, and construct a moving target information feature data set; Step 2.3, data segmentation and enhancement: Segment the motion target information feature data set obtained in step 2.2 according to the task requirements, downsample the segmented motion target information feature data set, rotate, translate and scale the downsampled motion target information feature data set to obtain an enhanced motion target information feature data set.

4. According to claim 1, a method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network is characterized in that: The construction steps of step 3 are as follows: Step 3.1, using the moving target information feature data set obtained in step 2.3 as the input of the data set scene extraction module, extracting each scene from the moving target information feature data set; Step 3.2, each moment in each scene obtained in step 3.1 is represented as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph; Step 3.3, according to the main node and other nodes in step 3.2, traverse the moving target information features of each scene in step 3.1, extract the connection relationship between the predicted moving target and other moving targets, and construct the connection relationship between the main node and other nodes at each moment; Step 3.4, convert the connection relationship between the main node and other nodes at each moment obtained in step 3.2 and the main node and other nodes at each moment constructed in step 3.3 into a sparse matrix, where the rows and columns of the matrix correspond to the target ID of the moving target, and the value of the matrix indicates whether there is a connection between the corresponding nodes; according to the connection status of the nodes, the graph adjacency matrix is ​​converted into a symmetric matrix; Step 3.5, normalize the graph adjacency matrix after symmetry in step 3.4 to obtain a graph adjacency matrix reflecting the motion space.

5. According to claim 1, a method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network is characterized in that: The specific process of step 4 is as follows: In step 4, the spatial graph feature extraction module (SGFE) includes an input layer, an intermediate layer, and an output layer; the input layer consists of a graph convolution layer and a GELU activation function; the intermediate layer consists of a graph convolution layer and a graph attention mechanism layer in parallel, wherein the graph convolution layer is used to further extract spatial information, and the graph attention mechanism layer is used to filter the key dimensional features of spatial information; the output layer includes a fully connected layer for generating implicit spatial structure information; Step 4.1, construct the target information features in each moment of each scene in the moving target information feature data set standardized in step 2 as the initial input feature matrix, use the graph convolution layer of the input layer of the spatial graph feature extraction module (SGFE) to perform convolution operation on the initial input feature matrix, extract the motion features in the local neighborhood, and use the GELU activation function to introduce nonlinear characteristics to the convolution output, enhance the expression ability of the model, ensure the effective learning of the features, and finally obtain the initial hidden spatial information feature matrix; Step 4.2: The initial hidden spatial information feature matrix obtained in step 4.1 is used as the input of the middle layer, where the graph convolution layer continues to further extract spatial information from the initial hidden spatial information features, maintains the information flow between nodes, and obtains the middle layer hidden spatial information feature matrix; the graph attention mechanism layer calculates the attention weights between nodes based on the middle layer hidden spatial information features and the graph adjacency matrix obtained in step 3, so as to selectively emphasize important features, and extract the key dimensional features in the motion space through weighted feature fusion; In step 4.3, the feature matrix of the hidden spatial information obtained in step 4.2 and the key dimensional features in the motion space are integrated to ensure that the output dimension matches the task requirements, and converted into implicit spatial motion structure information through the fully connected layer.

6. According to claim 1, a method for predicting target long-term trajectory based on dynamic graph structure cell long short-term memory network is characterized in that: The specific process of step 5 is as follows: The dynamic graph structured cell long short-term memory network (TSGCell-LSTM) includes a standard memory unit, a spatial information memory unit and a forgetting unit; among them, the standard memory unit is used to store state information updated with time steps, and is responsible for determining the update and retention of information according to the input data and the historical state of the network at each moment; the spatial information memory unit enhances the spatial information memory ability of the standard LSTM network, jointly extracts the target time sequence information and spatial information in the motion space, and updates the interactive information at each moment in the motion space; the forgetting unit is used to help the network clear irrelevant or outdated information so that the network only retains important feature information; the specific implementation steps of the dynamic graph structured cell long short-term memory network (TSGCell-LSTM) are: Step 5.1, feature input: input the implicit spatial motion structure information obtained in step 4 into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM); Step 5.2, spatial information screening: Part of the implicit interaction information in the motion space input in step 5.1 is screened and transmitted. The information screening method is as follows: Gr(t)=sigmoid(net G (t)) (5) Among them, G t is the spatial interaction information of the current frame, h t-1 is the information transmitted in the previous frame, and Gr(t) is the filtered spatial information; Step 5.3, information update: The spatial information Gr(t) filtered in step 5.2 enters the spatial information memory unit in the dynamic graph structure cell long short-term memory network (TSGCell-LSTM). The spatial information memory unit strengthens the memory of the current spatial structure information through an operation similar to a memory gate. The information transmission method is expressed as: C t =C t1 +I(t)*R(t)*Gr(t) (6) Among them, C t1 is the memory information after the standard memory unit and the forgetting unit, C t It is the memory information updated by spatial information; Step 5.4, output the updated result: add a bias term to the memory information updated in step 5.3, and output the iterative update result; Step 5.5: The updated result output in step 5.4 is input into the last fully connected layer, and the future 3D motion position information of the predicted moving target is obtained through regression.

7. A target long-term trajectory prediction system based on a dynamic graph structured cell long short-term memory network based on the method according to any one of claims 1 to 6, characterized in that: include: The 3D motion information feature acquisition module is used to detect the input video using a 3D object detector and a 3D object tracker to obtain the 3D motion information features of all objects in the video; A moving target information feature data set acquisition module is used to perform unified standardization processing on the 3D motion information features of all targets in the detected video to obtain a moving target information feature data set; A graph adjacency matrix construction module is used to extract each scene from the moving target information feature data set, and represent each moment in each scene as a graph, wherein the target ID at each moment is used as a node in the graph, the predicted moving target in the target ID is used as the main node in the graph, and the other moving targets in the target ID are used as other nodes in the graph, and the main node and other nodes are constructed into a graph adjacency matrix reflecting the motion space; The implicit spatial motion structure information acquisition module is used to use the target information features in each scene at each moment in the standardized motion target information feature dataset as the initial input feature matrix of the graph convolution network and the graph attention mechanism, provide 3D motion information features for the graph adjacency matrix reflecting the motion space, and use the spatial graph feature extraction module (SGFE) to update the features of each node in the graph adjacency matrix, and output the implicit spatial motion structure information, including the motion information between the targets in the motion space; The long-term trajectory information output module of the moving target is used to input the implicit spatial motion structure information into the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and simultaneously process the temporal information and spatial interaction information in the motion space through the dynamic graph structure cell long short-term memory network (TSGCell-LSTM), and finally output the long-term trajectory information of the predicted moving target, including the future 3D motion position information of the predicted moving target.

8. A target long-term trajectory prediction device based on dynamic graph structure cell long short-term memory network, characterized in that: include: Memory: a computer program storing a method for predicting a target long-term trajectory based on a dynamic graph structured cell long short-term memory network as described in any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement a target long-term trajectory prediction method based on a dynamic graph structured cell long short-term memory network as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a target long-term trajectory prediction method based on a dynamic graph structured cell long short-term memory network as described in any one of claims 1 to 6.

Images