Multi-unmanned aerial vehicle intention recognition method and system based on video frame completion and multi-modal feature learning
By combining video frame completion and multimodal feature learning with various robust mechanisms and neural networks, the accuracy and stability issues of intent recognition in adversarial environments for UAVs were addressed, achieving efficient intent recognition and prediction and improving the success rate of UAV mission execution.
Patent Information
- Application Number
- CN202510987742.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-17
- Publication Date
- 2025-10-31
AI Technical Summary
In drone warfare environments, traditional intent recognition methods struggle to effectively handle incomplete input data, noise interference, and dynamic disturbances, which limits the accuracy and stability of intent recognition in complex and uncertain environments.
A multi-UAV intent recognition system is constructed by employing video frame completion and multimodal feature learning methods, combined with extended Kalman filter (EKF), temporal generative adversarial network (T-GAN), convolutional neural network (CNN), graph neural network (GNN), and transformer network (Transformer). By filling in missing frames, denoising, extracting spatial features, and performing temporal dynamic modeling, the system predicts the future trajectory of UAVs and infers their operational intent.
It significantly improves the accuracy and stability of intent recognition in complex and uncertain environments, enhances the model's ability to understand complex behaviors, improves the success rate and adaptability of UAV mission execution, suppresses overfitting, and maintains good performance.
Smart Images
Figure CN120876875A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method and system for recognizing the intent of multiple unmanned aerial vehicles (UAVs), and more particularly to a method and system for recognizing the intent of multiple UAVs based on video frame completion and multimodal feature learning. Background Technology
[0002] In recent years, unmanned aerial vehicle (UAV) technology has developed rapidly and has been widely used in both civilian and military fields, including search and rescue, precision agriculture, and communication relay, greatly improving work efficiency and reducing operating costs. Especially in the application of UAV swarms, the ability of UAVs to cooperate and self-organize has been continuously enhanced, which not only improves mission execution efficiency but also opens up more possibilities for more complex mission scenarios. However, these advancements have also placed higher demands on the intelligence of airborne command and control systems. In this context, situational awareness has become the foundation of command and control. Its core tasks include not only analyzing and identifying the complex and concealed relationships between adversarial operational intentions and tactical measures, but also ensuring the rapid flow of information within command structures at all levels to achieve efficient decision-making and execution.
[0003] In UAV combat environments, perceiving the operational intentions of individual UAVs is a crucial component of situational awareness. However, changes in spatial location, flight parameters, or other factors can complicate and challenge the perception of individual UAV operational intentions. This is especially true in the intention recognition of UAV formations, where various interferences and environmental uncertainties are prevalent, including incomplete input data, noise interference, and dynamic interference. These issues severely limit the effectiveness of traditional intention recognition methods in practical applications, making the establishment of robust mechanisms particularly important.
[0004] The core objective of robust mechanisms is to ensure the accuracy and stability of situational awareness and intent recognition even under conditions of incomplete input data, noise interference, or anomalies. Traditional methods often overlook the impact of missing data, sensor noise, and dynamic interference, limiting their effectiveness in real-world scenarios. Summary of the Invention
[0005] Purpose of the invention: The purpose of this invention is to provide a multi-UAV intent recognition method based on video frame completion and multimodal feature learning to improve the accuracy and stability of intent recognition in complex and uncertain environments. On the other hand, it provides a multi-UAV intent recognition system based on video frame completion and multimodal feature learning.
[0006] Technical solution: The multi-UAV intent recognition method of the present invention includes the following steps:
[0007] S1. Collect simulated video data of the drone swarm, fill in the missing frames in the video data and remove noise;
[0008] S2. Convolutional Neural Network (CNN) is used to extract the spatial features of UAVs in video frames. At the same time, Graph Neural Network (GNN) is used to model the cooperative relationship between UAVs and extract the spatial features between UAVs.
[0009] S3. Input the extracted features into the Transformer network for time-series dynamic modeling, build an intent prediction model, and predict the future trajectory of the drone.
[0010] S4. The UAV's operational intent is inferred from the predicted trajectory using a multilayer perceptron (MLP) classifier.
[0011] Preferably, in step S1, filling in missing frames and removing noise from the video data involves processing the sequential video data using an Extended Kalman Filter (EKF) and a Temporal Generative Adversarial Network (T-GAN).
[0012] Preferably, S1 includes:
[0013] S11. Collect simulated video data of the drone swarm and construct the original video dataset;
[0014] S12. Set frame missing criteria, identify and fill in missing frames in the data and remove noise, and define state vectors;
[0015] S13. Construct the motion state of the UAV according to the discrete-time dynamic model and the observation model.
[0016] Preferably, the motion state of the UAV follows a discrete-time dynamic model as follows:
[0017] X t+1 =AX t +W t W t ~N(0,Q);
[0018] Among them, X t+1 Let A represent the state vector at time t+1, and let W represent the state transition matrix. t The process noise is represented by a Gaussian distribution with a mean of 0 and a covariance of Q.
[0019] The observation model is as follows:
[0020] Z t =HX t +V t V t ~N(0,R);
[0021] Among them, Z t Let H be the observation vector, H be the observation matrix, and V be the observation vector.t The observed noise follows a Gaussian distribution with a mean of 0 and a covariance of R.
[0022] Preferably, S2 includes: extracting frame information related to the location of the UAV from the processed video data; performing feature extraction on the extracted frame information through a convolutional neural network (CNN), and converting the feature representation into a vector form.
[0023] Preferably, the construction of the intent prediction model in S3 includes:
[0024] The extracted multimodal features are input into the transformer network to capture the dynamic characteristics of the drone trajectory changing over time.
[0025] An intent prediction model is constructed and trained. The trained model predicts the operational intent of the drone, and the results are compared with actual situations to optimize the model parameters. The predicted intent categories are:
[0026]
[0027] in, Let p represent the predicted intent category, Y represent the set of all possible intent categories, and p represent the predicted intent category. i (y) represents the probability that drone i predicts the intention as y.
[0028] Preferably, S4 includes:
[0029] S41. The cross-entropy loss function is used to measure the difference between the predicted result and the actual operational intention, and the model is trained using the error backpropagation algorithm. The training objective uses cross-entropy loss.
[0030]
[0031] Where L intent The loss value represents the intention prediction, where N represents the number of samples, C represents the number of intention categories, and y represents the number of intention categories. i,c This represents the actual class label of sample i; it is 1 if it is class c, and 0 otherwise. i (c) represents the probability that sample i is predicted to be of class c;
[0032] S42. Validate the model in multiple test scenarios, evaluate its performance under different conditions, and use Huber loss to handle outliers:
[0033]
[0034] in, This represents the Huber loss function, where x is the actual value. δ represents the predicted value and the threshold value.
[0035] The multi-UAV intent recognition system of the present invention includes:
[0036] The data processing module is used to collect simulated video data of drone swarms, fill in missing frames in the video data, and remove noise.
[0037] The video frame feature extraction module is used to extract spatial features between UAVs from the processed video frames using convolutional neural network (CNN) technology.
[0038] The intent prediction model module is used to construct and train a model for predicting intent based on pre-extracted UAV spatial features using graph neural network (GNN) technology.
[0039] The temporal dynamic modeling module is used to perform temporal dynamic modeling of the extracted multimodal features with the help of a converter network to predict the future trajectory of the UAV.
[0040] The intent classification module is used to infer the specific operational intent of the drone based on the predicted drone trajectory using a multilayer perceptron (MLP) classifier.
[0041] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: 1. By integrating multiple robust mechanisms such as the Extended Kalman Filter (EKF) and the Temporal Generative Adversarial Network (T-GAN), it can effectively handle missing data, sensor noise, and dynamic interference, significantly improving the accuracy and stability of intent recognition in complex and uncertain environments; 2. By combining Convolutional Neural Networks (CNN), Graph Neural Networks (GNN), and Transformers, it can comprehensively extract and fuse spatial features, group motion features, and temporal dynamic features, enhancing the model's understanding of complex behaviors and further improving the accuracy of intent recognition; 3. By predicting the future trajectory of the UAV and combining it with a Multilayer Perceptron (MLP) classifier, it can quickly and accurately infer the UAV's operational intent, enabling the UAV to better adapt to dynamic environments when performing tasks, thereby improving the success rate of task execution; 4. Through advanced training mechanisms such as adversarial training and structural regularization, it effectively suppresses overfitting and improves the model's adaptability under different interference conditions, allowing the model to maintain good performance even when facing various complex scenarios. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0043] Figure 2 This is a schematic diagram of timing feature capture according to the present invention;
[0044] Figure 3 This is a schematic diagram of the unmanned swarm flight of the present invention;
[0045] Figure 4This is a schematic diagram of the multi-UAV intent recognition framework for video frame completion and multimodal feature learning according to the present invention. Detailed Implementation
[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0047] A multi-UAV intent recognition method based on video frame completion and multimodal feature learning includes the following steps:
[0048] S1. Construct the original video dataset and complete the video data. The specific steps are as follows:
[0049] (1) Obtain and process the original video dataset to facilitate the application of the subsequent framework;
[0050] (2) Set appropriate frame missing criteria, identify and fill in missing frames in the data to ensure the integrity and continuity of the video sequence. The state vector is defined as:
[0051]
[0052] Among them, X t It is a state vector, representing the state of the UAV at time t, including position and velocity information; x t y t z t These represent the coordinates of the UAV in three-dimensional space, v and v respectively. x v y v z These represent the velocity components of the UAV in the three coordinate axes.
[0053] The motion state of the UAV follows a discrete-time dynamic model as follows:
[0054] X t+1 =AX t +W t W t ~N(0,Q);
[0055] Among them, X t+1 Let A represent the state vector at time t+1, and let W represent the state transition matrix, which describes the change of state over time. t The process noise is represented by a Gaussian distribution with mean 0 and covariance Q. The observation model is as follows:
[0056] Z t =HX t +V t V t ~N(0,R);
[0057] Among them, Z tThe observation vector represents the value observed at time t, H represents the observation matrix, which describes the relationship between the state and the observation, and V... t The observation noise is represented by a Gaussian distribution with a mean of 0 and a covariance of R.
[0058] S2. Extract spatial features between drones. The specific steps are as follows:
[0059] (1) Extract frame information related to the drone's location from the processed video dataset;
[0060] (2) Feature extraction is performed on the extracted frame information using a CNN network, and the feature representation is transformed into a vector form suitable for subsequent analysis:
[0061] F CNN (I t )=φ CNN (I t W CNN );
[0062] Among them, F CNN (I t I represents the feature vector extracted by CNN. t φ represents the input image at time t. NN W represents the feature extraction function of a convolutional neural network. CNN This represents the weight parameters of the CNN.
[0063] S3. Construct an intent prediction model. The specific steps are as follows:
[0064] (1) Input the extracted multimodal features into the transformer network to capture the dynamic characteristics of the UAV trajectory changing over time;
[0065] (2) Construct and train an intent prediction model. Predict the UAV's operational intent using the trained model, compare it with the actual situation, and optimize the model parameters. The predicted intent categories are:
[0066]
[0067] in, Let p represent the predicted intent category, Y represent the set of all possible intent categories, and p represent the predicted intent category. i (y) represents the probability that drone i predicts the intention as y.
[0068] S4. Model training and evaluation, the specific steps are as follows:
[0069] (1) The cross-entropy loss function is used to measure the difference between the prediction result and the actual operation intention, and the model is trained by the error backpropagation algorithm. The training objective uses the cross-entropy loss as follows:
[0070]
[0071] Among them, L intent The loss value represents the intention prediction, where N represents the number of samples, C represents the number of intention categories, and y represents the number of intention categories. i,c This represents the actual class label of sample i; it is 1 if it is class c, and 0 otherwise. i (c) represents the probability that sample i is predicted to be class c.
[0072] (2) Verify the accuracy of the model in multiple test scenarios, evaluate the model's performance under different conditions, achieve more accurate intent recognition, and use Huber loss to handle outliers:
[0073]
[0074] in, This represents the Huber loss function, where x represents the actual value. δ represents the predicted value, and δ represents the threshold, which determines the switching between the linear and squared parts of the loss function.
[0075] A multi-UAV intent recognition system based on video frame completion and multimodal feature learning includes:
[0076] The data processing module is used to collect simulated video data of drone swarms, fill in missing frames in the video data, and remove noise.
[0077] The video frame feature extraction module is used to extract spatial features between UAVs from the processed video frames using convolutional neural network (CNN) technology.
[0078] The intent prediction model module is used to construct and train a model for predicting intent based on pre-extracted UAV spatial features using graph neural network (GNN) technology.
[0079] The temporal dynamic modeling module is used to perform temporal dynamic modeling of the extracted multimodal features with the help of a converter network to predict the future trajectory of the UAV.
[0080] The intent classification module is used to infer the specific operational intent of the drone based on the predicted drone trajectory using a multilayer perceptron (MLP) classifier.
[0081] In this embodiment, the ratio of training, validation, and test sets is set to 7:2:1 to ensure the reliability of model performance evaluation. All experiments are repeated three times to calculate average accuracy and stability. The results of this embodiment on one dataset are shown in Table 1.
[0082] Table 1 Intent Recognition Results
[0083] intention accuracy attack 0.968 retreat 0.971 patrol 0.946 Investigation 0.936 feint attack 0.981
[0084] In summary, this invention effectively combines video frame completion with multimodal feature learning, utilizing Extended Kalman Filter (EKF) and Temporal Generative Adversarial Network (T-GAN) as core technologies to accurately fill in missing data. Furthermore, it employs Convolutional Neural Network (CNN) and Graph Neural Network (GNN) to extract spatial and group phenomena, improving the performance of UAV intent recognition. This invention excels in handling data sparsity and real-time video processing, significantly improving the accuracy and robustness of intent recognition, and laying a solid foundation for multi-UAV collaborative tasks.
Claims
1. A multi-UAV intent recognition method based on video frame completion and multimodal feature learning, characterized in that, Includes the following steps: S1. Collect simulated video data of the drone swarm, fill in the missing frames in the video data and remove noise; S2. Convolutional Neural Network (CNN) is used to extract the spatial features of UAVs in video frames. At the same time, Graph Neural Network (GNN) is used to model the cooperative relationship between UAVs and extract the spatial features between UAVs. S3. Input the extracted features into the Transformer network for time-series dynamic modeling, build an intent prediction model, and predict the future trajectory of the drone. S4. The UAV's operational intent is inferred from the predicted trajectory using a multilayer perceptron (MLP) classifier.
2. The multi-UAV intent recognition method according to claim 1, characterized in that, S1 describes filling in missing frames and removing noise from video data by processing sequential video data using an Extended Kalman Filter (EKF) and a Temporal Generative Adversarial Network (T-GAN).
3. The multi-UAV intent recognition method according to claim 1, characterized in that, S1 includes: S11. Collect simulated video data of the drone swarm and construct the original video dataset; S12. Set frame missing criteria, identify and fill in missing frames in the data and remove noise, and define state vectors; S13. Construct the motion state of the UAV according to the discrete-time dynamic model and the observation model.
4. The multi-UAV intent recognition method according to claim 3, characterized in that, The motion state of the UAV follows a discrete-time dynamic model as follows: X t+1 =AX t +W t ,W t ~N(0,Q); Among them, X t+1 Let A represent the state vector at time t+1, and let W represent the state transition matrix. t The process noise is represented by a Gaussian distribution with a mean of 0 and a covariance of Q. The observation model is as follows: Z t =HX t +V t V t ~N(0,R); Among them, Z t Let H be the observation vector, H be the observation matrix, and V be the observation vector. t The observed noise follows a Gaussian distribution with a mean of 0 and a covariance of R.
5. The multi-UAV intent recognition method according to claim 1, characterized in that, S2 includes: extracting frame information related to the drone's location from the processed video data; performing feature extraction on the extracted frame information using a convolutional neural network (CNN), and converting the feature representation into a vector form.
6. The multi-UAV intent recognition method according to claim 1, characterized in that, The intention prediction model described in S3 includes: The extracted multimodal features are input into the transformer network to capture the dynamic characteristics of the drone trajectory changing over time. An intent prediction model is constructed and trained. The trained model predicts the operational intent of the drone, and the results are compared with actual situations to optimize the model parameters. The predicted intent categories are: in, Let p represent the predicted intent category, Y represent the set of all possible intent categories, and p represent the predicted intent category. i (y) represents the probability that drone i predicts the intention as y.
7. The multi-UAV intent recognition method according to claim 1, characterized in that, S4 includes: S41. The cross-entropy loss function is used to measure the difference between the predicted result and the actual operational intention, and the model is trained using the error backpropagation algorithm. The training objective uses cross-entropy loss. Where L intent The loss value represents the intention prediction, where N represents the number of samples, C represents the number of intention categories, and y represents the number of intention categories. i,c This represents the actual class label of sample i; it is 1 if it is class c, and 0 otherwise. i (c) represents the probability that sample i is predicted to be class c; S42. Validate the model in multiple test scenarios, evaluate its performance under different conditions, and use Huber loss to handle outliers: in, This represents the Huber loss function, where x is the actual value. δ represents the predicted value and the threshold value.
8. A multi-UAV intent recognition system based on video frame completion and multimodal feature learning, characterized in that, include: The data processing module is used to collect simulated video data of drone swarms, fill in missing frames in the video data, and remove noise. The video frame feature extraction module is used to extract spatial features between UAVs from the processed video frames using convolutional neural network (CNN) technology. The intent prediction model module is used to construct and train a model for predicting intent based on pre-extracted UAV spatial features using graph neural network (GNN) technology. The temporal dynamic modeling module is used to perform temporal dynamic modeling of the extracted multimodal features with the help of a converter network to predict the future trajectory of the UAV. The intent classification module is used to infer the specific operational intent of the drone based on the predicted drone trajectory using a multilayer perceptron (MLP) classifier.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the multi-UAV intent recognition method based on video frame completion and multimodal feature learning as described in any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-UAV intent recognition method based on video frame completion and multimodal feature learning as described in any one of claims 1 to 7.