Floating object trajectory prediction method and system based on multi-modal space-time large model

Abstract

The invention relates to the technical field of crossing of artificial intelligence and meteorological engineering, in particular to a floating object track prediction method and system based on a multi-modal space-time large model, and the method comprises the steps: 1, collecting and preprocessing multi-modal data; step 2, sharing federal learning data; step 3, performing space-time alignment treatment; 4, predicting a physical constraint trajectory; 5, dynamically correcting the residual error; and step 6, visualization and decision support. According to the method, the graph attention space-time alignment network, the physical constraint large model and the federated learning-residual error correction double-closed-loop technology are comprehensively utilized, so that efficient fusion of multi-source heterogeneous data, physical rule driven trajectory deduction and millisecond-level dynamic response of burst interference are realized; technical guarantee is provided for reducing the trajectory prediction error of the floating object by 82% and improving the response speed by 93%, and high-reliability decision support is provided for the fields of meteorological monitoring, low-altitude unmanned aerial vehicle logistics, aviation safety early warning and the like.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and meteorological engineering, specifically to a method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model. Background Technology

[0002] With the rapid advancement of internet technology and the booming development of the low-altitude economy, we have witnessed a vibrant new era, but also some challenges, especially in the field of low-altitude safety. The widespread use of weather balloons, drones, and other aircraft has not only increased the complexity of air traffic management but also posed new challenges to aviation safety and the safety of ground personnel. Traditional airborne object trajectory prediction technology faces three core bottlenecks:

[0003] (1) Limited data fusion capability: Traditional methods often use a single meteorological data source (such as wind speed or air pressure) and fail to fully integrate multimodal spatiotemporal information (including three-dimensional wind field, topography, real-time GPS trajectory and other multi-dimensional data), resulting in incomplete environmental characterization.

[0004] (2) Long-term prediction error amplification effect: In predictions exceeding 3 hours based on traditional time series models such as RNN / LSTM, the error accumulates exponentially due to the recursive iteration mechanism, and the prediction reliability decreases sharply with the increase of time.

[0005] (3) The dynamic environment response is slow. Existing models lack an online learning mechanism for sudden meteorological events (such as sudden gusts and atmospheric turbulence), making it difficult to achieve real-time correction of the predicted trajectory.

[0006] To address the aforementioned problems, we propose a method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model. Summary of the Invention

[0007] The purpose of this invention is to provide a method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model includes the following steps:

[0010] Step 1: Multimodal data acquisition and preprocessing: Collect meteorological data, terrain data, and real-time GPS trajectory data, and perform standardized data processing, including data cleaning, normalization, missing value imputation, and unit conversion.

[0011] Step 2, Federated Learning Data Sharing: The pre-processed privacy data is encrypted and segmented, local model training is carried out, and the encrypted model parameters are uploaded to the federated learning server through a secure channel. After aggregation, the global model parameters are distributed to various departments to achieve cross-departmental data security sharing and collaborative model optimization.

[0012] Step 3, Spatiotemporal alignment processing: Initialize the spatiotemporal alignment network based on global model parameters, construct the graph structure of multimodal data using a graph attention network, calculate the attention weights between nodes, and align the multi-resolution data to a unified spatiotemporal coordinate system;

[0013] Step 4, Physical Constraint Trajectory Prediction: Based on the large physical constraint model, load the global model parameters, embed the discretized Navier-Stokes equations as physical constraint terms into the loss function, perform trajectory extrapolation on the aligned multimodal data, and output prediction results that conform to physical laws.

[0014] Step 5, Residual Dynamic Correction: The residual correction module calculates the residual between the predicted trajectory and the actual GPS trajectory in real time. When the residual exceeds the set threshold, the extended Kalman filter and online gradient descent algorithm are used to dynamically adjust the model parameters. When there is no GPS signal, the correction is switched to IMU inertial data.

[0015] Step 6, Visualization and Decision Support: Generate a 3D probability heatmap and obstacle avoidance suggestions based on the prediction results, and simultaneously achieve cross-departmental encrypted data sharing and model parameter aggregation and update through the federated learning server.

[0016] Preferably, the implementation process of the spatiotemporal alignment network in step 2 includes:

[0017] Data preprocessing: Standardize meteorological data, terrain data and real-time GPS tracks of different resolutions, convert meteorological data from Celsius temperature to Kelvin temperature, and unify wind speed data of different units to meters per second;

[0018] Constructing a graph structure: Representing multimodal data as a graph structure, where nodes represent the basic units of data and edges represent the spatial or temporal relationships between nodes;

[0019] Graph attention calculation: The constructed graph structure is encoded using a graph attention network. The relationship weights between each node and its neighboring nodes are calculated. For each node, its attention score with other nodes is calculated. The formula for calculating the attention score is: ,

[0020] in, It is the attention score between node i and node j. It is a learnable attention vector. It is a weight matrix. and These are the feature vectors of node i and node j, respectively. It is the activation function, and T is the transpose;

[0021] Feature fusion and alignment: Based on the calculated attention weights, the data features of different modalities are weighted and fused, and by introducing spatial and temporal coordinate information as additional feature dimensions, the multi-resolution data is aligned to a unified spatiotemporal coordinate system.

[0022] Preferably, in step 2, federated learning data sharing is used to achieve cross-departmental collaborative data training while protecting privacy data. The specific implementation process includes:

[0023] Data encryption and segmentation: Each department encrypts the private data it holds, and then segments the encrypted data into multiple subsets, each subset containing a portion of samples or features;

[0024] Local model training: Each department trains the base model locally using its own subset of data and updates the model parameters. During the training process, only local data is used for forward and backward propagation calculations, without transmitting the original data, thus further protecting data privacy.

[0025] Encrypted parameter aggregation: The model parameters trained locally by each department are encrypted and sent to the federated learning server through a secure communication channel. The server uses the encrypted aggregation method in the OpenMined framework to perform a weighted average of the received encrypted parameters to obtain the globally updated model parameters.

[0026] Global model update and distribution: The federated learning server decrypts the aggregated global model parameters and distributes them to each department. Each department uses the new global model parameters to replace the local model parameters and conducts the next round of local training.

[0027] Repeat the above process until the model converges, enabling cross-departmental collaborative data training, improving model performance, and protecting the data privacy of each department.

[0028] Preferably, the residual correction module in step 4 is used to dynamically calculate the error between the predicted trajectory and the actual trajectory, and to provide feedback to adjust the model parameters, thereby reducing the accumulation of long-term errors. The specific implementation process includes:

[0029] Error calculation and triggering mechanism: Real-time acquisition of the deviation between the actual GPS trajectory and the model-predicted trajectory of the airborne object, calculation of the residual vector, and position residual. velocity residual , It is the actual GPS location. It predicts GPS location. That's the actual speed. It is the prediction speed. Set a residual threshold. When the residual exceeds the threshold, the residual correction mechanism is triggered.

[0030] The Extended Kalman Filter (EKF) is combined with online gradient descent: EKF is used to estimate the state of the residuals, thus estimating the error state in the model prediction. Then, guided by the estimated error state, the online gradient descent algorithm is used to update the model parameters in real time. The parameter update formula for online gradient descent is as follows: ,

[0031] in, These are the model parameters at time t. These are the model parameters at time t+1. It's the learning rate. It is the gradient of the loss function with respect to the model parameters;

[0032] Auxiliary correction when there is no GPS signal: In the absence of GPS signal, inertial data provided by the inertial measurement unit is used for auxiliary correction. By constructing a dynamic model based on IMU data, the motion state of the airborne object is estimated and fused with the model's predicted state. The fused state information is then used to adjust the model parameters.

[0033] The system for implementing the above-described method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model includes:

[0034] Data acquisition terminal: Deployed at weather stations / drones, including high-precision sensors and Jetson AGX Orin edge nodes, used for multimodal data acquisition and preliminary processing;

[0035] Federated Learning Server: Hosted by China Telecom Cloud, it uses the OpenMined framework to achieve encrypted aggregation of cross-departmental data.

[0036] Central training platform: Equipped with an NVIDIA DGXA100 cluster for training and inference of physically constrained large models;

[0037] Airborne object prediction terminal: The airborne end uses Xilinx UltraScale + FPGA to accelerate inference, and the ground end uses a WebGL 3D visualization platform.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] This invention aims to provide a high-precision and robust solution for predicting the trajectory of airborne objects. By comprehensively utilizing Graph Attention Spatiotemporal Alignment Network (ST-AlignNet), PhysiCo-Transformer (a large model with physical constraints), and federated learning-residual correction dual-loop technology, it achieves efficient fusion of multi-source heterogeneous data, trajectory extrapolation driven by physical laws, and millisecond-level dynamic response to sudden disturbances. This provides technical support for reducing the trajectory prediction error of airborne objects by 82% and improving the response speed by 93%, and provides highly reliable decision support for fields such as meteorological monitoring, low-altitude UAV logistics, and aviation safety early warning.

[0040] This invention addresses the shortcomings of multi-source data fusion, where traditional methods cannot effectively align the spatiotemporal differences between meteorological, topographical, and trajectory data (such as the conflict between satellite data and ground sensor resolution).

[0041] This invention addresses the accumulation of long-term prediction errors, where existing AI models (such as LSTM) exhibit exponential increases in error with the step size during long-term predictions.

[0042] The present invention has a delayed response to sudden disturbances. Sudden weather changes (such as gusts and turbulence) can cause prediction failures, and traditional solutions require retraining the model. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of the present invention;

[0044] Figure 2 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0045] The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model includes the following steps:

[0046] Step 1: Multimodal data acquisition and preprocessing: Collect meteorological data, terrain data, and real-time GPS trajectory data, and perform standardized data processing, including data cleaning, normalization, missing value imputation, and unit conversion.

[0047] Step 2, Federated Learning Data Sharing: The pre-processed privacy data is encrypted and segmented, local model training is carried out, and the encrypted model parameters are uploaded to the federated learning server through a secure channel. After aggregation, the global model parameters are distributed to various departments to achieve cross-departmental data security sharing and collaborative model optimization.

[0048] Step 3, Spatiotemporal alignment processing: Initialize the spatiotemporal alignment network based on global model parameters, construct the graph structure of multimodal data using a graph attention network, calculate the attention weights between nodes, and align the multi-resolution data to a unified spatiotemporal coordinate system;

[0049] Step 4, Physical Constraint Trajectory Prediction: Based on the large physical constraint model, load the global model parameters, embed the discretized Navier-Stokes equations as physical constraint terms into the loss function, perform trajectory extrapolation on the aligned multimodal data, and output prediction results that conform to physical laws.

[0050] Step 5, Residual Dynamic Correction: The residual correction module calculates the residual between the predicted trajectory and the actual GPS trajectory in real time. When the residual exceeds the set threshold, the extended Kalman filter and online gradient descent algorithm are used to dynamically adjust the model parameters. When there is no GPS signal, the correction is switched to IMU inertial data.

[0051] Step 6, Visualization and Decision Support: Generate a 3D probability heatmap and obstacle avoidance suggestions based on the prediction results, and simultaneously achieve cross-departmental encrypted data sharing and model parameter aggregation and update through the federated learning server.

[0052] The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model specifically includes:

[0053] (1) Multimodal data fusion module

[0054] Spatiotemporal Alignment Network (ST-AlignNet): Employs a graph attention network (GAT) to perform spatiotemporal alignment of meteorological data, terrain data, and real-time GPS trajectories, resolving the resolution inconsistency issue.

[0055] Detailed steps:

[0056] Data preprocessing: Standardization processing is performed on meteorological data, terrain data, and real-time GPS tracks at different resolutions, including data cleaning and normalization. Meteorological data is converted from Celsius to Kelvin, and wind speed data in different units is uniformly converted to meters per second.

[0057] Constructing a graph structure: Represent multimodal data as a graph structure, where nodes represent the basic units of data and edges represent the spatial or temporal relationships between nodes. For example, for meteorological data, each meteorological monitoring point is treated as a node; for topographic data, each topographic grid cell is treated as a node; and for GPS tracks, each track point is treated as a node. Then, based on geographical location and time information, connections between nodes are established.

[0058] Graph Attention Calculation: A Graph Attention Network (GAT) is used to encode the constructed graph structure and calculate the relationship weights between each node and its neighbors. Specifically, for each node, an attention score is calculated between it and other nodes; a higher attention score indicates a stronger association between the two nodes. The formula for calculating the attention score is: ,

[0059] in, It is the attention score between node i and node j. It is a learnable attention vector. It is a weight matrix. and These are the feature vectors of node i and node j, respectively. is the activation function, and T is the transpose.

[0060] Feature fusion and alignment: Based on the calculated attention weights, data features from different modalities are weighted and fused. By introducing spatial and temporal coordinate information as additional feature dimensions, multi-resolution data are aligned to a unified spatiotemporal coordinate system. For example, for 0.25° meteorological grid data and 10m UAV wind field data, GAT's attention fusion and coordinate alignment operations enable them to be fused at the same spatiotemporal scale.

[0061] Federated learning data sharing: Enables cross-departmental collaborative training while protecting privacy data (such as military radar monitoring).

[0062] Detailed steps:

[0063] Data encryption and segmentation: Various departments (such as meteorological bureaus and air traffic control bureaus) encrypt the private data they hold, using encryption algorithms such as homomorphic encryption to ensure the confidentiality of the data during the sharing process. The encrypted data is then segmented into multiple subsets, each containing a portion of samples or features.

[0064] Local model training: Each department trains the base model (such as a pre-trained PhysiCo-Transformer model) and updates its parameters locally using its own subset of datasets. During training, only local data is used for forward and backward propagation calculations; the original data is not transmitted, further protecting data privacy.

[0065] Encrypted parameter aggregation: The model parameters trained locally by each department are encrypted and sent to the federated learning server via a secure communication channel. The server uses encrypted aggregation methods such as the federated averaging algorithm in the OpenMined framework to perform a weighted average of the received encrypted parameters to obtain globally updated model parameters.

[0066] Global Model Update and Distribution: The federated learning server decrypts the aggregated global model parameters (decryption is performed only on the server side, and the decrypted parameters do not contain the original data information) and distributes them to each department. Each department replaces its local model parameters with the new global model parameters and performs the next round of local training. This process is repeated until the model converges, enabling cross-departmental collaborative data training, improving model performance, and protecting the data privacy of each department.

[0067] (2) Physically constrained large model (PhysiCo-Transformer)

[0068] Physical equations embedded in the loss function: The Navier-Stokes fluid dynamics equations (existing technology) are added as constraints to the Transformer training process to ensure that the predictions conform to physical laws.

[0069] Detailed steps:

[0070] Discretization of the Navier-Stokes equations: The continuous form of the Navier-Stokes equations is discretized, transforming it into a form suitable for computation on a discrete spatiotemporal grid. The finite difference method is used to discretize and approximate each term in the equations, yielding an algebraic equation representation at discrete points. Assuming the equations are in a planar region... Discretize the data and divide the region into... The grid has a grid spacing of [value]. , .use Represents a grid point, and the corresponding position is ,

[0071] Discretization of the continuity equation at grid points At this point, the continuity equation can be discretized as: ,

[0072] Discretization of the momentum equation in the x-direction at grid points At this point, the momentum equation in the x-direction can be discretized as: ,

[0073] Discretization of the momentum equation in the y-direction at grid points At this point, the momentum equation in the y-direction can be discretized as: ,

[0074] in, and They are fluids in and The velocity component in the direction; It's pressure; It is the fluid density; It is kinematic viscosity. , , The subscript indicates the velocity component or pressure value at a specific grid point.

[0075] By discretizing the equations using the finite difference method, the continuous Navier-Stokes equations are transformed into a system of algebraic equations at discrete grid points. These equations can then be solved using numerical methods and embedded into the loss function of the PhysiCo-Transformer model to achieve physical constraints.

[0076] Constructing the loss function: Add the discretized form of the Navier-Stokes equation as a constraint term to the training loss function of the Transformer model. The combined loss function is: ,

[0077] in, It is the loss due to prediction error. λ is the physical constraint loss, and λ is the balancing weight parameter.

[0078] Model Training and Optimization: The comprehensive loss function is optimized using the backpropagation algorithm, while the parameters of the Transformer model and the physical constraint terms are updated. The optimization goal is to enable the model to accurately fit historical data and satisfy the basic physical laws of fluid dynamics when predicting the trajectory of airborne objects.

[0079] Residual Correction Module (ResCorrector): Dynamically calculates (if a new algorithm is used, please list the new algorithm formula in detail and provide a detailed explanation) the error between the predicted trajectory and the actual trajectory, and feeds back to adjust model parameters to reduce the accumulation of long-term errors.

[0080] Detailed steps:

[0081] Error Calculation and Triggering Mechanism: The deviation between the actual GPS trajectory of the airborne object and the model's predicted trajectory is acquired in real time, and the residual vector is calculated. Position Residual velocity residual , It is the actual GPS location. It predicts GPS location. That's the actual speed. This refers to the prediction speed. A residual threshold is set; when the residual exceeds the threshold, a residual correction mechanism is triggered.

[0082] The Extended Kalman Filter (EKF) is combined with online gradient descent: EKF is used to estimate the state of the residuals, thus estimating the error state in the model prediction. Then, guided by the estimated error state, the online gradient descent algorithm is used to update the model parameters in real time. The parameter update formula for online gradient descent is as follows: ,

[0083] in, These are the model parameters at time t+1. These are the model parameters at time t. It's the learning rate. It is the gradient of the loss function with respect to the model parameters.

[0084] Auxiliary correction in the absence of GPS signal: In the absence of GPS signal, inertial data provided by the inertial measurement unit (IMU) is used for auxiliary correction. By constructing a dynamic model based on IMU data, the motion state of the airborne object is estimated, and this estimate is fused with the model's predicted state. The fused state information is then used to adjust the model parameters.

[0085] The implementation process of the dynamic model based on IMU data is as follows:

[0086] 1. Data Acquisition and Preprocessing

[0087] Data Acquisition: An IMU sensor is installed on the airborne object to collect triaxial acceleration and triaxial angular velocity data at a high frequency, while also acquiring real-time timestamp information.

[0088] Preprocessing: The acquired IMU data undergoes necessary preprocessing, including data filtering, noise reduction, and time synchronization, to improve data quality. For example, low-pass filtering can be used to remove high-frequency noise interference, and linear interpolation can be used to achieve data time synchronization.

[0089] 2. Establishment of the dynamic model

[0090] Kinematic equations: Based on the physical characteristics and motion laws of the airborne object, a dynamic model is established to describe the motion state of the airborne object in three-dimensional space, including the relationships between parameters such as position, velocity, and attitude. The model can be represented as: ,

[0091] in, Indicates the time of the airborne object The state vector includes position, velocity, attitude, etc. Indicates the time of the airborne object The state vector; Indicates time The control inputs mainly come from IMU data; The kinematic equations representing the dynamic model.

[0092] State transition matrix: Construct a state transition matrix to describe the state change relationship of the airborne object between adjacent time points. The specific form of the state transition matrix is ​​determined based on the kinematic equations and the sampling period.

[0093] Process noise covariance matrix: Considering various uncertainties in the motion of airborne objects, such as airflow disturbances and sensor errors, the process noise covariance matrix is ​​determined to reflect the uncertainty of the model.

[0094] 3. State estimation and correction

[0095] State prediction: Based on the state estimate and IMU data from the previous moment, the state of the airborne object at the current moment is predicted using a dynamic model.

[0096] Measurement update: Compare the current GPS measurement data with the predicted state and calculate the measurement residual.

[0097] State correction: Based on the measurement residuals and Kalman gain, the predicted state is corrected to obtain the optimal state estimate at the current time.

[0098] (3) Real-time interactive prediction system

[0099] Sudden environmental response mechanism: When abnormal meteorological data (such as sudden wind speed changes) is detected, the model is automatically fine-tuned (how to fine-tune) and the predicted trajectory is updated.

[0100] Model fine-tuning triggering and update strategy: The latest abnormal meteorological data is used as input to rapidly fine-tune the PhysiCo-Transformer model. During fine-tuning, only some parameters of the model (such as those related to meteorological conditions) are updated to reduce computational load and time latency. Simultaneously, the local model is appropriately adjusted based on the global model parameters on the federated learning server to ensure that the model can update the predicted trajectory in a timely manner when responding to sudden meteorological events, improving the accuracy and real-time performance of predictions.

[0101] Multi-trajectory simulation visualization: Based on the probability distribution of airborne object trajectories predicted by the PhysiCo-Transformer model, the WebGL 3D engine is used for visualization rendering to generate a probability heatmap and display the possible landing point distribution in the next 10 minutes, 1 hour, and 3 hours.

[0102] Example 1: Drone trajectory prediction in clear weather

[0103] Hardware equipment and operation

[0104] Data acquisition terminal: A high-precision weather sensor (Vaisala WXT530 weather instrument) and a Jetson AGXOrin edge node are deployed on the drone. After takeoff, the weather sensor collects meteorological data under clear weather conditions at a frequency of 0.1Hz, including temperature, humidity, wind speed, and air pressure. Simultaneously, the Jetson AGXOrin edge node collects the drone's GPS trajectory data at a frequency of 1Hz. This data is uploaded to the central training platform in real time via the MQTT protocol.

[0105] Central training platform: Equipped with an NVIDIA DGXA100 cluster for training large-scale physically constrained models. The model inference process begins when 1000 time-series samples are received. First, ST-AlignNet aligns NOAA 0.25° meteorological grid data with UAV 10m wind field data, controlling the error within 0.15m / s to eliminate resolution differences and improve fusion accuracy. Then, the PhysiCo-Transformer model predicts the UAV's 6-hour trajectory based on the aligned data, embedding Navier-Stokes constraints with a loss weight β set to 0.3, outputting a trajectory prediction result that conforms to physical laws, with a 6-hour prediction error controlled within 47m.

[0106] Airborne object prediction terminal: The airborne end uses Xilinx UltraScale + FPGA to accelerate inference, while the ground end uses a WebGL 3D visualization platform. During model inference, the airborne FPGA acquires the deviation between the UAV's actual GPS trajectory and the model's predicted trajectory in real time and calculates the residual vector. When the residual exceeds 50m, ResCorrector triggers online gradient descent to adjust model parameters. The learning rate η is set to 0.001, and the error is converged to within 10m within 3 seconds. Simultaneously, the ground-based WebGL 3D visualization platform renders a 3D probability heatmap based on the prediction results, marks the 95% confidence region, and displays the possible landing point distribution for the next 10 minutes, 1 hour, and 3 hours, assisting users in formulating flight plans.

[0107] Data calculation process:

[0108] Spatiotemporal alignment: Assuming the wind speed in the NOAA 0.25° meteorological grid data is The wind speed in the 10m wind field data from the drone is The attention weights calculated using GAT are: The aligned wind speed data It can be represented as: ,

[0109] in The calculation formula is: ,

[0110] The attention score between node i and node j is calculated using the following formula: ,

[0111] It is a learnable attention vector. It is a weight matrix. and These are the feature vectors of node i and node j, respectively. is the activation function, and T is the transpose.

[0112] Training a large model with physical constraints: Constructing a comprehensive loss function using discretized Navier-Stokes equations as physical constraints. ,

[0113] in, It is the loss due to prediction error. It is a loss due to physical constraints. This is the balancing weight parameter, set to 0.3. The comprehensive loss function is optimized using the backpropagation algorithm, and the parameters of the Transformer model are updated.

[0114] Residual Correction: When the residual exceeds 50m, the EKF is used to perform state estimation on the residual to obtain the error state. Then, the online gradient descent algorithm is used to update the model parameters: ,

[0115] in, These are model parameters, subscripts. , Indicates before and after the update. This is the learning rate, set to 0.001. It is an error state. The gradient of the model parameters.

[0116] Technical effect comparison:

[0117] Improved Prediction Accuracy: The traditional LSTM model has a 6-hour trajectory prediction error of 214m under clear weather conditions, while the PhysiCo-Transformer model of this invention has a prediction error of only 47m, representing an 82% reduction. This is because in clear weather, meteorological conditions are relatively stable, but the traditional LSTM model relies solely on data-driven approaches and cannot effectively capture physical laws, easily leading to large errors. This invention, by embedding the Navier-Stokes equations into the Transformer loss function, makes the model output conform to physical laws, thereby significantly improving prediction accuracy.

[0118] Breakthrough in Real-Time Performance: Traditional solutions have a response delay of 3.2 seconds when encountering sudden disturbances (such as airflow disturbances encountered by drones). In contrast, the residual correction module (ResCorrector) of this invention, combining EKF and online gradient descent algorithm, can complete the correction within 0.2 seconds, improving efficiency by 93%. This is because in traditional solutions, encountering sudden disturbances requires retraining the model or waiting for updates at fixed intervals, resulting in a long response delay. The residual correction module of this invention can detect residuals in real time and make dynamic adjustments, significantly shortening the response time.

[0119] Example 2: Prediction of weather probe trajectory during heavy rain

[0120] Hardware equipment and operation:

[0121] Data acquisition terminal: High-precision meteorological sensors and Jetson AGX Orin edge nodes deployed at the weather station. After the weather balloon is launched, the meteorological sensors collect meteorological data under heavy rain conditions at a frequency of 0.1Hz, including rain intensity, wind speed, and air pressure. At the same time, the Jetson AGX Orin edge nodes collect GPS trajectory data of the weather balloon at a frequency of 1Hz. This data is uploaded to the federated learning server in real time via the MQTT protocol.

[0122] Federated Learning Server: Hosted by China Telecom Cloud, using the OpenMined framework, it enables encrypted aggregation of cross-departmental data (meteorological bureau, air traffic control bureau, etc.). Once a certain amount of data is received, the federated learning process is initiated. Each department encrypts its private data using homomorphic encryption and other encryption algorithms to ensure confidentiality during data sharing. The encrypted data is then divided into multiple subsets, and each department trains its own subset locally to use the base model (a pre-trained PhysiCo-Transformer model), updating the model parameters. The encrypted model parameters trained locally by each department are then sent to the federated learning server via a secure communication channel. The server uses encrypted aggregation methods such as federated averaging to perform a weighted average of the received encrypted parameters, obtaining globally updated model parameters, which are then distributed to each department.

[0123] Central training platform: Equipped with an NVIDIA DGXA100 cluster, it uses globally aggregated model parameters from a federated learning server to train a large-scale physical constraint model. Once the model converges, the trained model parameters are distributed to the airborne object prediction terminals.

[0124] Airborne object prediction terminal: The airborne end uses Xilinx UltraScale + FPGA to accelerate inference, while the ground end uses a WebGL 3D visualization platform. During model inference, the airborne FPGA acquires the deviation between the actual GPS trajectory of the weather balloon and the model's predicted trajectory in real time and calculates the residual vector. When the residual exceeds 50m, ResCorrector triggers online gradient descent to adjust model parameters and the learning rate. Setting it to 0.001, the error will converge to within 10m within 3 seconds. Simultaneously, the ground-based WebGL 3D visualization platform renders a 3D probability heatmap based on the prediction results, marks the 95% confidence area, and displays the possible landing point distribution for the next 10 minutes, 1 hour, and 3 hours, assisting the meteorological bureau in developing a balloon recovery plan.

[0125] Data calculation process:

[0126] Spatiotemporal alignment: Assuming the rainfall intensity of the NOAA 0.25° meteorological grid data is... The GPS trajectory data of the weather balloon is The attention weights calculated using GAT are: Then the aligned rainstorm intensity data and GPS trajectory data It can be represented as: , ,

[0127] in, and These are, respectively, rainfall intensity data from weather balloons and GPS trajectory data corresponding to the NOAA meteorological grid. The calculation formula is the same as that in Example 1.

[0128] Training a large-scale model with physical constraints: Under heavy rain conditions, the discretized form of the Navier-Stokes equations needs to consider the impact of heavy rain on fluid motion. When constructing the comprehensive loss function, the weights of the physical constraint loss are adjusted. To better adapt to the physical laws under heavy rain conditions, the comprehensive loss function is optimized using the backpropagation algorithm to update the parameters of the Transformer model.

[0129] Residual correction: When the residual exceeds 50m, the EKF is used to estimate the state of the residual to obtain the error state.

[0130] Example 3: Prediction of sounding balloon trajectory in typhoon scenarios

[0131] The detailed implementation steps are shown in the table below.

[0132]

[0133] Alternative solutions

[0134] Step 2, Spatiotemporal Alignment: If the network latency is >200ms, you can switch to the local lightweight GAT model (parameters compressed by 50%).

[0135] Step 4, Residual Correction: When there is no GPS signal, IMU inertial data is used to assist in correction (accuracy decreases to ±20m).

[0136] Through comparative testing, the obstacle avoidance success rate and emergency obstacle avoidance response time of the present invention are significantly improved compared with traditional methods. The specific improvement is shown in the table below.

[0137]

[0138] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model, characterized in that, Includes the following steps: Step 1: Multimodal data acquisition and preprocessing: Collect meteorological data, terrain data, and real-time GPS trajectory data, and perform standardized data processing, including data cleaning, normalization, missing value imputation, and unit conversion. Step 2, Federated Learning Data Sharing: The pre-processed privacy data is encrypted and segmented, local model training is carried out, and the encrypted model parameters are uploaded to the federated learning server through a secure channel. After aggregation, the global model parameters are distributed to various departments to achieve cross-departmental data security sharing and collaborative model optimization. Step 3, Spatiotemporal alignment processing: Initialize the spatiotemporal alignment network based on global model parameters, construct the graph structure of multimodal data using a graph attention network, calculate the attention weights between nodes, and align the multi-resolution data to a unified spatiotemporal coordinate system; Step 4, Physical Constraint Trajectory Prediction: Based on the large physical constraint model, load the global model parameters, embed the discretized Navier-Stokes equations as physical constraint terms into the loss function, perform trajectory extrapolation on the aligned multimodal data, and output prediction results that conform to physical laws. Step 5, Residual Dynamic Correction: The residual correction module calculates the residual between the predicted trajectory and the actual GPS trajectory in real time. When the residual exceeds the set threshold, the extended Kalman filter and online gradient descent algorithm are used to dynamically adjust the model parameters. When there is no GPS signal, the correction is switched to IMU inertial data. Step 6, Visualization and Decision Support: Generate a 3D probability heatmap and obstacle avoidance suggestions based on the prediction results, and simultaneously achieve cross-departmental encrypted data sharing and model parameter aggregation and update through the federated learning server.

2. The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model according to claim 1, characterized in that, The implementation process of the spatiotemporal alignment network in step 2 includes: Data preprocessing: Standardize meteorological data, terrain data and real-time GPS tracks of different resolutions, convert meteorological data from Celsius temperature to Kelvin temperature, and unify wind speed data of different units to meters per second; Constructing a graph structure: Representing multimodal data as a graph structure, where nodes represent the basic units of data and edges represent the spatial or temporal relationships between nodes; Graph attention calculation: The constructed graph structure is encoded using a graph attention network. The relationship weights between each node and its neighboring nodes are calculated. For each node, its attention score with other nodes is calculated. The formula for calculating the attention score is:

3. Among them, It is the attention score between node i and node j. It is a learnable attention vector. It is a weight matrix. and These are the feature vectors of node i and node j, respectively. It is the activation function, and T is the transpose; Feature fusion and alignment: Based on the calculated attention weights, the data features of different modalities are weighted and fused, and by introducing spatial and temporal coordinate information as additional feature dimensions, the multi-resolution data is aligned to a unified spatiotemporal coordinate system.

4. The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model according to claim 1, characterized in that, Step 2, where federated learning data sharing is used to achieve cross-departmental collaborative training while protecting privacy, includes the following specific implementation process: Data encryption and segmentation: Each department encrypts the private data it holds, and then segments the encrypted data into multiple subsets, each subset containing a portion of samples or features; Local model training: Each department trains the base model locally using its own subset of data and updates the model parameters. During the training process, only local data is used for forward and backward propagation calculations, without transmitting the original data, thus further protecting data privacy. Encrypted parameter aggregation: The model parameters trained locally by each department are encrypted and sent to the federated learning server through a secure communication channel. The server uses the encrypted aggregation method in the OpenMined framework to perform a weighted average of the received encrypted parameters to obtain the globally updated model parameters. Global model update and distribution: The federated learning server decrypts the aggregated global model parameters and distributes them to each department. Each department uses the new global model parameters to replace the local model parameters and conducts the next round of local training. Repeat the above process until the model converges, enabling cross-departmental collaborative data training, improving model performance, and protecting the data privacy of each department.

5. The method for predicting the trajectory of airborne objects based on a multimodal spatiotemporal large model according to claim 1, characterized in that, The residual correction module in step 4 is used to dynamically calculate the error between the predicted trajectory and the actual trajectory, and to adjust the model parameters accordingly to reduce the accumulation of long-term errors. The specific implementation process includes: Error calculation and triggering mechanism: Real-time acquisition of the deviation between the actual GPS trajectory and the model-predicted trajectory of the airborne object, calculation of the residual vector, and position residual. velocity residual , It is the actual GPS location. It predicts GPS location. That's the actual speed. It is the prediction speed. Set a residual threshold. When the residual exceeds the threshold, the residual correction mechanism is triggered. Extended Kalman Filtering (EKF) combined with online gradient descent: EKF is used to estimate the state of the residuals, thus estimating the error state in the model prediction. Then, guided by the estimated error state, the online gradient descent algorithm is used to update the model parameters in real time. The parameter update formula for online gradient descent is:

6. Among them, These are the model parameters at time t. These are the model parameters at time t+1. It's the learning rate. It is the gradient of the loss function with respect to the model parameters; Auxiliary correction when there is no GPS signal: In the absence of GPS signal, inertial data provided by the inertial measurement unit is used for auxiliary correction. By constructing a dynamic model based on IMU data, the motion state of the airborne object is estimated and fused with the model's predicted state. The fused state information is then used to adjust the model parameters.

7. A system for implementing the airborne object trajectory prediction method based on a multimodal spatiotemporal large model as described in any one of claims 1-4, characterized in that, include: Data acquisition terminal: Deployed at weather stations / drones, including high-precision sensors and Jetson AGX Orin edge nodes, used for multimodal data acquisition and preliminary processing; Federated Learning Server: Hosted by China Telecom Cloud, it uses the OpenMined framework to achieve encrypted aggregation of cross-departmental data. Central training platform: Equipped with an NVIDIA DGXA100 cluster for training and inference of physically constrained large models; Airborne object prediction terminal: The airborne end uses Xilinx UltraScale + FPGA to accelerate inference, and the ground end uses a WebGL 3D visualization platform.

Images