An object entity trajectory prediction method based on air-ground fusion

Abstract

The application designs a target entity trajectory prediction method based on air-ground fusion, and belongs to the technical field of trajectory prediction; firstly, a spatial relationship connection graph is constructed for the historical trajectory of the ground target entity, and a weighted adjacency matrix of the graph is obtained according to the spatial relationship connection graph; then, spatial interaction feature extraction is carried out by using GCN to obtain a spatial interaction feature vector; secondly, an air perspective video of the ground target entity is obtained by using a UAV, and is converted into a video frame; the video frame is sent into U-Net to extract scene context information; the historical trajectory of the ground target entity and the context information are aggregated by MLP to obtain the feature of the current time scene which integrates the target entity context and the historical trajectory of the target entity; and the aggregated feature is calculated based on the U-Net model to obtain the terminal position probability distribution of the entity; the terminal position probability distribution of the entity, the historical trajectory of the ground target entity and the spatial interaction feature vector are fused, and the future trajectory of the final target entity is obtained based on the TCN model.

Description

Technical Field

[0001] This invention belongs to the field of trajectory prediction technology, specifically relating to a target entity trajectory prediction method based on air-ground fusion. Background Technology

[0002] With the rapid development of sixth-generation communication technology, the concept of an integrated air-ground network has emerged. As a crucial requirement for this network, air-ground information fusion technology has received widespread attention in recent years. Air-ground information fusion helps break down barriers to data sharing between air and ground network systems, maximizing information flow. Video, location, and various sensor data collected by airborne nodes supplement the information acquired by ground nodes. The fusion of different dimensions of ground and air information provides data support for solving various tasks such as surveying, forecasting, and monitoring. Due to the need for stability and efficiency in air-ground connectivity in collaborative scenarios, airborne machines and ground entities must maintain air-ground coordination and synchronized movements. Therefore, accurate prediction of the trajectories of ground entities is crucial.

[0003] The multimodal and complex spatiotemporal interactions inherent in trajectory prediction make constructing trajectory prediction algorithms highly challenging. Typically, trajectory prediction requires considering the spatial interactions of multiple entities in a scene, as well as the temporal dependence of predicted entity trajectories. The emergence and development of deep learning have provided solutions to this problem. In recent years, models such as generative adversarial networks, graph convolutional neural networks, convolutional neural networks, and recurrent neural networks have been extensively studied and applied to trajectory prediction, yielding numerous results. Xue H et al. proposed a staged method for predicting pedestrian trajectories. In the first stage, a bidirectional LSTM classification network predicts the destination region. Then, in the second stage, an LSTM-based architecture generates the trajectory corresponding to the predicted destination region. This method, combined with latent destination prediction, improves trajectory prediction accuracy.

[0004] From the perspective of multimodal and accuracy considerations in trajectory prediction, modeling only the time and space aspects is insufficient. With the continuous development of trajectory prediction technology, more and more scholars are considering incorporating aerial data into the input, attempting to improve trajectory prediction accuracy with more information input. Specifically, aerial data input is video input from an aerial perspective. Video input provides richer scene context information, which is lacking when relying solely on historical trajectory information of ground entities as input. Simultaneously, the broader scope of video information allows the model to consider the entire scene, thus enabling greater scene coverage when inferring the future trajectory of target entities, which helps improve the model's prediction accuracy. Shafiee N et al. introduced a sequence-to-sequence model for predicting pedestrian trajectories. They used a kinematic encoder and a 3D visual encoder to process the pedestrian's historical trajectory and the input video. The kinematic encoder used LSTM to process the historical trajectory, while the visual encoder used a conditional attention mechanism to extract dynamic scene context information from the input video. The decoder used LSTM to generate the trajectory distribution, thereby producing a multimodal trajectory output.

[0005] Currently, trajectory prediction algorithms have shifted from initial dynamics-based design approaches to deep learning-based model building. With the rapid development of deep learning, trajectory prediction algorithms have been extensively and deeply studied. However, previous research largely relied on ground-based entity trajectory information, neglecting the importance of aerial video input. Furthermore, prediction models based on Long Short-Term Memory (LSTM) networks have a large number of parameters and require storing intermediate calculation results, making it difficult to handle long-term trajectory dependencies. Additionally, most methods struggle to accurately capture complex spatial interactions between moving entities, leading to decreased prediction performance. Finally, many works have failed to effectively model the multimodal characteristics of the output trajectory, limiting the algorithm's application in real-world scenarios. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention designs a target entity trajectory prediction method based on air-ground fusion, solving the trajectory prediction problem of target entities in air-ground collaborative scenarios. This scenario mainly includes two aspects: one is the aerial vehicle, such as a drone, effectively collecting ground video information, which serves as the main input of the entire method; the other is the historical trajectory and current position information of various moving entities on the ground, which serve as the auxiliary input of the entire method. The air and ground input data are combined to infer the trajectory and position of the ground target entity in the next period of time.

[0007] A target entity trajectory prediction method based on air-to-ground fusion specifically includes the following steps:

[0008] Step 1: Construct a spatial relationship connection graph for the historical trajectories of ground target entities, and obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph; then use GCN to extract spatial interaction features and obtain spatial interaction feature vectors.

[0009] Step 2: Use a drone to acquire aerial video of ground target entities, convert it into video frames, and send the video frames into U-Net to extract scene context information;

[0010] First, the acquired video input V0 is converted into video frames. For a video with an input duration of T seconds, the frame rate is F frames / second, the frame number starts from 0, and there are a total of TF frames. The video contains the observation of the entire historical trajectory of the ground target entity. The observation time step of the historical trajectory is set to T0. Therefore, the video needs to be converted into T0 frames to cope with the time step of each observation of the historical trajectory. The mathematical form is shown in Formula 1.

[0011]

[0012] The obtained video frames will be fed into an encoder-decoder CNN for scene context information extraction, using the U-Net model, which consists of the following three parts:

[0013] (1) Encoder: Input an image, and gradually reduce the size and number of channels of the image through multiple convolutional layers and pooling layers to extract high-level features;

[0014] (2) Decoder: The decoder restores the feature maps extracted by the encoder to the original image size and gradually increases the number of channels to finally obtain the pixel-level segmentation results;

[0015] (3) SkipConnection: Connects features in the encoder with features in the decoder, thereby better preserving the image's detailed information and contextual information;

[0016] Step 3: Aggregate the historical trajectory of the ground target entity and the context information obtained in Step 2 using MLP to obtain the current scene features that combine the target entity context and the target entity historical trajectory.

[0017] The MLP method is used to aggregate contextual information and historical trajectory features. The MLP formula is shown in Equation 2.

[0018]

[0019] Where σ(·) is the activation function, W1 and W2 are weight matrices, and b is the bias vector. It is scene context information. It is the historical trajectory embedding of the target entity n, output This represents the aggregated feature vector, which is used as input for predicting the endpoint location.

[0020] Step 4: Calculate the probability distribution of entity endpoint locations based on the aggregated features using the U-Net model;

[0021] Based on the U-Net model, the system simultaneously analyzes historical trajectories and scene context information to predict the future destination of the target entity and outputs a probability distribution that reflects the multimodal nature of the future trajectory.

[0022] Specifically, the U-Net model contains L blocks, utilizing two convolutional layers with ReLU activation functions and max-pooling to reduce the spatial dimension of the input H×W, followed by each intermediate output H. l (1≤l≤L) are fed to the decoder via skip connections; the extended arm also consists of L blocks, utilizing bilinear upsampling and two convolutional layers containing ReLU activation functions to make H l The resolution is doubled, skip connections fuse tensors from the contracting and expanding arms, and finally, after convolutional layers and pixel-by-pixel sigmoid operations, the target entity in T is output. seq =T0+T p The spatial probability distribution of the endpoint position at time t is represented by a sigmoid function, where the output value is a real number between 0 and 1, indicating the probability value. The specific formula is shown in Equation 3.

[0023]

[0024] in This represents the probability distribution of the output, where n and t represent the entity number and the current time, respectively, and W... U-Net It is a weight matrix. This represents the mapping to the input, and σ(·) represents the sigmoid activation function;

[0025] The target entity endpoint prediction is trained using a binary cross-entropy loss function. Based on prior knowledge, the horizontal and vertical coordinates are independent. First, the target entity's position at point T is used as the reference point. seq =T0+T p A two-dimensional Gaussian distribution is constructed with the actual location at time as the center.

[0026] The loss function for this part is constructed by predicting the binary cross-entropy loss distributed in the true distribution, and the specific formula is shown in Equation 4.

[0027]

[0028] in, These are the current position coordinates. It is a predicted location. It is the true distribution. It is a predicted distribution;

[0029] Step 5: Integrate the probability distribution of the entity's endpoint location, the historical trajectory of the ground target entity, and the spatial interaction feature vector to obtain the future trajectory of the final target entity based on the TCN model;

[0030] By sampling the probability distribution of the endpoint location of entities, the predicted location of target entities in the scene is obtained; the distribution is represented as a two-dimensional heatmap, representing the location from t=0 to t=T. o Under the condition that the target entity is in T seq =T o +T p The probability of being in the final position at any given time, and the final predicted endpoint position. It was sampled from these probabilities;

[0031] The three input tensors are fused: interaction features, probability distribution of entity endpoint location, and historical trajectory of the target entity.

[0032] The fusion method adopted is to perform a linear transformation on each input feature tensor and use a fully connected approach to map them to the same dimension, as shown in Equation 5.

[0033]

[0034] Where X i Representing three input features, X' i It is the output of the linear transformation. W represents a linear transformation mapping. i It is the weight matrix, b i It is the bias vector;

[0035] Next, a weighted aggregation method is used; the three input feature tensors are multiplied by a weight coefficient, and the three results are added together to obtain a single input tensor.

[0036]

[0037] Where X input The aggregated input tensor is defined by α1, α2, and α3, which are the weight coefficients of the three input tensors. These three weight coefficients are optimized through training as linear transformation parameters.

[0038] After fusion, the data is input into the TCN model, which uses a sliding window to infer the motion characteristics of the target entity at future times. Based on these characteristics, the distribution of the target entity's position coordinates at future times is generated, and the trajectory coordinates at each future time are obtained through sampling.

[0039] Beneficial technical effects of the present invention:

[0040] This invention proposes a target entity trajectory prediction method based on air-ground fusion. Using historical trajectories of entities in a scene and aerial video as input, it extracts scene context information and predicts the endpoint position of entities, thereby improving the accuracy of trajectory prediction and assisting in solving the trajectory prediction problem of multiple entities in air-ground collaborative scenarios. This method effectively learns the distribution of trajectory positions, improving prediction accuracy and accurately predicting the future trajectory of entities, demonstrating its advanced nature and effectiveness. Joint analysis of video information and historical trajectory data further enhances prediction accuracy, providing methodological support for air-ground integration strategies. It can be deployed on aircraft in air-ground collaborative mission scenarios, with inputs fusing air and ground data and employing a weighted approach to process different input information; these weighting coefficients are learned through training.

[0041] This invention can predict the future movement paths of ground entities, thereby avoiding collisions and conflicts between them and ensuring the safe operation of air and ground equipment. Furthermore, trajectory prediction is also a crucial component of building smart cities. In urban traffic management, logistics delivery, and autonomous driving, it can help optimize the movement paths of intelligent devices, reducing traffic congestion and resource waste. By analyzing and predicting device trajectories, intelligent path planning strategies can be developed, improving traffic efficiency and resource utilization. Attached Figure Description

[0042] Figure 1 A flowchart of a target entity trajectory prediction method based on air-to-ground fusion according to an embodiment of the present invention;

[0043] Figure 2 A schematic diagram comparing the results of this invention embodiment with those of S-LSTM and G-GAN algorithms. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0045] A target entity trajectory prediction method based on air-to-ground fusion, as shown in the appendix. Figure 1 As shown, the specific steps include:

[0046] Step 1: Construct a spatial relationship connection graph for the historical trajectories of ground target entities, and obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph; then use GCN to extract spatial interaction features and obtain spatial interaction feature vectors.

[0047] This process serves as the backbone of the embodiments of the present invention; it models the temporal dependence of the trajectory and the spatial interaction between entities, that is, it is responsible for the modeling and processing of the time and spatial dimensions in the trajectory prediction problem. This part is the foundational work for solving the trajectory prediction problem.

[0048] First, the ground trajectory is embedded. A graph is constructed based on the spatial positions of entities in the scene at the current moment. This graph is then fed into a Graph Convolutional Network (GCN). Message propagation is performed using graph convolution, and the output represents the spatial relationships (I) of the entities at the current moment. t .

[0049] Since entities in a scene are typically in constant motion, their positions change over time, making it difficult to find a fixed topological structure to describe the scene content. Spatial relationship graphs can effectively solve this problem. In a spatial relationship graph, when two entities have a spatial relationship, an edge connects them. Graphs are very convenient for describing real-world scenes where the number and position of nodes change over time. While the relative positions of entities in the scene change at different times, the graph structure remains constant; only the spatial relationships between the three entities change. As the distance between the two entities at the bottom increases, their mutual trajectory influence strengthens, making their spatial relationship more compact. Similarly, the spatial relationship between the two entities at the top becomes more distant. Therefore, describing real-world trajectory prediction scenes using spatial relationship graphs is a very natural idea.

[0050] The strength of spatial relationships is quantified by the weights on the spatial relationship connection graph. The distance between entities is used as the basis for setting the weight. After the spatial relationship connection graph constructs the spatial location of the entities, it is input into GCN to further extract entity features.

[0051] Step 2: Use a drone to acquire aerial video of ground target entities, convert it into video frames, and send the video frames into U-Net to extract scene context information;

[0052] First, the acquired video input V0 is converted into video frames. For a video with an input duration of T seconds, the frame rate is F frames / second, the frame number starts from 0, and there are a total of TF frames. The video contains the observation of the entire historical trajectory of the ground target entity. The observation time step of the historical trajectory is set to T0. Therefore, the video needs to be converted into T0 frames to cope with the time step of each observation of the historical trajectory. The mathematical form is shown in Formula 1.

[0053]

[0054] The obtained video frames will be fed into an encoder-decoder CNN for scene context information extraction, using the U-Net model, which consists of the following three parts:

[0055] (1) Encoder: Input an image, and gradually reduce the size and number of channels of the image through multiple convolutional layers and pooling layers to extract high-level features;

[0056] (2) Decoder: The decoder restores the feature maps extracted by the encoder to the original image size and gradually increases the number of channels to finally obtain the pixel-level segmentation results;

[0057] (3) SkipConnection: Connects features in the encoder with features in the decoder, thereby better preserving the image's detailed information and contextual information;

[0058] video frames The data is fed into U-Net for processing. In U-Net, the encoder uses multiple convolutional and pooling layers for feature extraction and dimensionality reduction, while the decoder uses multiple upsampling and convolutional layers for feature restoration and expansion. At the same time, skip connections are used to connect the features in the encoder with the features in the decoder. After a series of convolution, pooling, upsampling, skip connections, concatenation, non-linear activation and dropout operations, the image's detailed information and contextual information can be better preserved.

[0059] Step 3: Aggregate the historical trajectory of the ground target entity and the context information obtained in Step 2 using MLP to obtain the current scene features that combine the target entity context and the target entity historical trajectory.

[0060] The MLP method is used to aggregate contextual information and historical trajectory features. The MLP formula is shown in Equation 2.

[0061]

[0062] Where σ(·) is the activation function, W1 and W2 are weight matrices, and b is the bias vector. It is scene context information. It is the historical trajectory embedding of the target entity n, output This represents the aggregated feature vector, which is used as input for predicting the endpoint location.

[0063] Step 4: Calculate the probability distribution of entity endpoint locations based on the aggregated features using the U-Net model;

[0064] This process, serving as the endpoint prediction part, predicts the endpoint position of the target entity at a specified time, supplementing the input information and handling the scene context modeling for the trajectory prediction problem. In the endpoint position prediction part, the video input is processed into video frames, which are then fed into an encoder-decoder CNN to extract scene information. This information, along with the target entity's historical trajectory information, is then fed into the endpoint position prediction part to obtain the endpoint position probability distribution. Finally, a reasonable endpoint position is obtained through sampling. Before the final inference, the outputs of these two parts need to be fused and fed into the trajectory inference part to infer the future trajectory of the target entity.

[0065] Convolutional Neural Networks (CNNs) are deep feedforward neural networks characterized by local connectivity and weight sharing. They are one of the representative algorithms of deep learning and are adept at handling image processing, especially image recognition and related machine learning problems. They have shown significant improvement in various visual tasks such as image classification, object detection, and image segmentation, and are currently one of the most widely used models.

[0066] Convolutional neural networks (CNNs) possess representation learning capabilities, enabling them to perform shift-invariant classification of input information according to their hierarchical structure. They can perform both supervised and unsupervised learning. The shared parameters of convolutional kernels within their hidden layers and the sparsity of inter-layer connections allow CNNs to learn grid-like topology features, such as pixels and audio, with relatively low computational cost, exhibiting stable performance and requiring no additional feature engineering. Consequently, they are widely used in fields such as computer vision and natural language processing.

[0067] Based on the U-Net model, the system simultaneously analyzes historical trajectories and scene context information to predict the future destination of the target entity and outputs a probability distribution that reflects the multimodal nature of the future trajectory.

[0068] Specifically, the U-Net model contains L blocks, utilizing two convolutional layers with ReLU activation functions and max-pooling to reduce the spatial dimension of the input H×W, followed by each intermediate output H. l (1≤l≤L) are fed to the decoder via skip connections; the extended arm also consists of L blocks, utilizing bilinear upsampling and two convolutional layers containing ReLU activation functions to make H l The resolution is doubled, skip connections fuse tensors from the contracting and expanding arms, and finally, after convolutional layers and pixel-by-pixel sigmoid operations, the target entity in T is output. seq =T0+T p The spatial probability distribution of the endpoint position at time t is represented by a sigmoid function, where the output value is a real number between 0 and 1, indicating the probability value. The specific formula is shown in Equation 3.

[0069]

[0070] in This represents the probability distribution of the output, where n and t represent the entity number and the current time, respectively, and W... U-Net It is a weight matrix. This represents the mapping to the input, and σ(·) represents the sigmoid activation function;

[0071] The target entity endpoint prediction is trained using a binary cross-entropy loss function. Based on prior knowledge, the horizontal and vertical coordinates are independent. First, the target entity's position at point T is used as the reference point. seq =T0+T p A two-dimensional Gaussian distribution is constructed with the actual location at time t as the center; the loss function for this part is constructed by predicting the binary cross-entropy loss that is distributed in the actual distribution, as shown in Equation 4.

[0072]

[0073] in, These are the current position coordinates. It is a predicted location. It is the true distribution. It is a predicted distribution;

[0074] Step 5: Integrate the probability distribution of the entity's endpoint location, the historical trajectory of the ground target entity, and the spatial interaction feature vector to obtain the future trajectory of the final target entity based on the TCN model;

[0075] This part requires integrating the output of the main body, the output of the entity endpoint prediction, and the historical trajectory of the current target entity over a past period as the overall input.

[0076] For inputs consisting of multiple feature tensors, a common approach is to use aggregation functions to combine these tensors into a single, unified input. Common aggregation functions include vector concatenation, weighted summation, and multilayer perceptrons (MLPs). For the three-part input mentioned above, vector concatenation or weighted summation can be considered for aggregation.

[0077] To address the common limitations of previous studies, a target entity trajectory prediction method based on air-ground fusion is proposed. This method utilizes video input from an aerial perspective and historical trajectory information of ground target entities to model scene context information and spatial interactions between ground target entities, effectively solving two major challenges in trajectory prediction: modeling time dependence and spatial interactivity.

[0078] By sampling the probability distribution of the endpoint location of entities, the predicted location of target entities in the scene is obtained; the distribution is represented as a two-dimensional heatmap, representing the location from t=0 to t=T. o Under the condition that the target entity is in T seq =T o +T pThe probability of being in the final position at any given time, and the final predicted endpoint position. It was sampled from these probabilities;

[0079] The three input tensors are fused: interaction features, probability distribution of entity endpoint location, and historical trajectory of the target entity.

[0080] The fusion method adopted is to perform a linear transformation on each input feature tensor and use a fully connected approach to map them to the same dimension, as shown in Equation 5.

[0081]

[0082] Where X i Representing three input features, X' i It is the output of the linear transformation. W represents a linear transformation mapping. i It is the weight matrix, b i It is the bias vector;

[0083] Next, a weighted aggregation method is used; the three input feature tensors are multiplied by a weight coefficient, and the three results are added together to obtain a single input tensor.

[0084]

[0085] Where X input The aggregated input tensor is defined by α1, α2, and α3, which are the weight coefficients of the three input tensors. These three weight coefficients are optimized through training as linear transformation parameters.

[0086] After fusion, the data is input into the TCN model, which uses a sliding window to infer the motion characteristics of the target entity at future times. Based on these characteristics, the distribution of the target entity's position coordinates at future times is generated, and the trajectory coordinates at each future time are obtained through sampling.

[0087] The method of this invention was tested on the public datasets ETH and Stanford Drone Dataset (SDD). The experimental results show that the method can effectively learn the distribution of trajectory positions and predict the future trajectory of target entities with relatively high accuracy.

[0088] The performance of the proposed method was validated on the public dataset ETH and its four subsets. The figure below shows the training loss.

[0089] To fully validate the method's performance, quantitative experimental results were verified on the SDD dataset, as shown in Table 1. The error data in the table are in meters; smaller values ​​indicate more accurate predictions.

[0090] Table 1 Comparison of prediction results of different algorithms on the SDD dataset (predicting 12 time steps);

[0091]

[0092] Compared to S-LSTM and G-GAN algorithms, this method achieves the best average prediction error and final displacement error on the ETH dataset, and performs best on most datasets. S-LSTM, a classic pedestrian trajectory prediction method using LSTM, produces the worst prediction results because the social pool approach cannot effectively model complex spatial interactions. G-GAN, as a graph-based method for modeling spatial interactions, achieves better results, demonstrating the effectiveness of graph-based methods for modeling scene spatial interactions. Specific results are shown in the appendix. Figure 2 As shown.

Claims

1. A target entity trajectory prediction method based on air-to-ground fusion, characterized in that, Specifically, the following steps are included: Step 1: Construct a spatial relationship connection graph for the historical trajectories of ground target entities, and obtain the weighted adjacency matrix of the graph based on the spatial relationship connection graph; then use GCN to extract spatial interaction features and obtain spatial interaction feature vectors. Step 2: Use a drone to acquire aerial video of ground target entities, convert it into video frames, and send the video frames into U-Net to extract scene context information; Step 3: Aggregate the historical trajectory of the ground target entity and the context information obtained in Step 2 using MLP to obtain the current scene features that combine the target entity context and the target entity historical trajectory. Step 4: Calculate the probability distribution of entity endpoint locations based on the aggregated features using the U-Net model; Step 5: Integrate the probability distribution of the entity's endpoint location, the historical trajectory of the ground target entity, and the spatial interaction feature vector to obtain the future trajectory of the final target entity based on the TCN model; Specifically, the probability distribution of the endpoint location of entities is sampled to obtain the predicted location of target entities in the scene; the distribution is represented as a two-dimensional heatmap, indicating the location of the target entities from the endpoint location. =0 to = Under the condition that the target entity is = + The probability of being in the final position at any given time, and the final predicted endpoint position. It was sampled from these probabilities; The three input tensors are fused: interaction features, probability distribution of entity endpoint location, and historical trajectory of the target entity. The fusion method adopted is to perform a linear transformation on each input feature tensor and use a fully connected approach to map them to the same dimension, as shown in Equation 5. （5） in Representing three input features, It is the output of the linear transformation. ( ) represents a linear transformation mapping. It is a weight matrix. It is the bias vector; Next, a weighted aggregation method is used; the three input feature tensors are multiplied by a weight coefficient, and the three results are added together to obtain a single input tensor. in The input tensor after aggregation. , , These are the weight coefficients of the three input tensors, which are optimized through training as linear transformation parameters. After fusion, the data is input into the TCN model, which uses a sliding window to infer the motion characteristics of the target entity at future times. Based on these characteristics, the distribution of the target entity's position coordinates at future times is generated, and the trajectory coordinates at each future time are obtained through sampling.

2. The target entity trajectory prediction method based on air-to-ground fusion according to claim 1, characterized in that, Step 1 is as follows: First, the acquired video The input is converted into video frames; for an input duration of... The video is 2 seconds long, with a frame rate of F frames per second, and the frame number starts from 0. There are a total of A frame in the video contains observations of the entire historical trajectory of a ground target entity, with the observation time step set to [number]. Therefore, the video needs to be converted. A frame is used to account for each time step of the historical trajectory, and its mathematical form is shown in Equation 1. (1) The obtained video frames will be fed into an encoder-decoder CNN for scene context information extraction, using the U-Net model.

3. The target entity trajectory prediction method based on air-to-ground fusion according to claim 2, characterized in that, The U-Net model consists of the following three parts: (1) Encoder: Input an image, and gradually reduce the size and number of channels of the image through multiple convolutional layers and pooling layers to extract high-level features; (2) Decoder: restores the feature maps extracted by the encoder to the original image size, and gradually increases the number of channels to finally obtain the pixel-level segmentation result; (3) SkipConnection: Connects features in the encoder with features in the decoder, thereby better preserving the details and context of the image.

4. The target entity trajectory prediction method based on air-to-ground fusion according to claim 1, characterized in that, Step 3 uses MLP to aggregate contextual information and historical trajectory features. The MLP formula is shown in Equation 2. (2) in, ( ) is the activation function. and It is a weight matrix. It is a bias vector. It is scene context information. It is the historical trajectory embedding of the target entity n, output This represents the aggregated feature vector, used as input for endpoint location prediction.

5. The target entity trajectory prediction method based on air-to-ground fusion according to claim 1, characterized in that, Step 4 specifically involves: Based on the U-Net model, the system simultaneously analyzes historical trajectories and scene context information to predict the future destination of the target entity and outputs a probability distribution that reflects the multimodal nature of the future trajectory. Specifically, the U-Net model contains L blocks, utilizing two convolutional layers with ReLU activation functions and max-pooling to reduce the spatial dimension of the input H×W, followed by each intermediate output... The data is fed to the decoder via skip connections; the extended arm also consists of L blocks, utilizing bilinear upsampling and two convolutional layers containing ReLU activation functions. The resolution is doubled, skip connections fuse tensors from the contracting and expanding arms, and finally, after convolutional layers and pixel-by-pixel sigmoid operations, the target entity is output. The spatial probability distribution of the endpoint position at time t is represented by a sigmoid function, where the output value is a real number between 0 and 1, indicating the probability value. The specific formula is shown in Equation 3. (3) in This represents the probability distribution of the output, where n and t represent the entity number and the current time, respectively. It is a weight matrix. ( () represents a mapping to the input. ( ) represents the sigmoid activation function; The binary cross-entropy loss function is used to train the target entity endpoint prediction.

6. The target entity trajectory prediction method based on air-to-ground fusion according to claim 5, characterized in that, The target entity endpoint prediction is trained using a binary cross-entropy loss function, specifically as follows: Based on prior knowledge, horizontal and vertical coordinates are independent. First, consider the target entity's position... A two-dimensional Gaussian distribution is constructed with the actual location at time as the center. The loss function for this part is constructed by predicting the binary cross-entropy loss distributed in the true distribution, and the specific formula is shown in Equation 4. (4) in, These are the current position coordinates. It is a predicted location. It is the true distribution. It is a predicted distribution.

Images