Unmanned aerial vehicle operation skill training system based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle operation skill training system based on deep reinforcement learning, and the system comprises the following modules: an operation behavior data collection module which is used for constructing an operation behavior data set; the data preprocessing module is used for generating a time sequence joint input tensor; the behavior transformation module is used for outputting an operation behavior representation vector; the parameter updating module is used for carrying out non-momentum mode iteration updating on the training parameters of the behavior transformation Transform module based on a Lion optimizer; the disturbance adjusting module is used for obtaining disturbance behavior representation; the decoding module is used for decoding the disturbance behavior representation into a control action sequence; the reversible motivation generation network module is used for reasoning to obtain a reconstructed motivation vector; the motivation comparison module is used for identifying target cognitive deviation; and the training scheduling module is used for outputting a new training scene and a control strategy to the training simulation environment. According to the method, deep reinforcement learning and motivation modeling methods are fused, and personalized training of unmanned aerial vehicle operation skills is realized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control and unmanned aerial vehicle (UAV) application technology, and in particular to a UAV operation skill training system based on deep reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, deep reinforcement learning (DL) is increasingly being applied to complex control tasks, particularly in the field of intelligent drones. Traditional drone training relies heavily on instruction and field practice by human flight instructors. However, this approach presents challenges such as long training cycles, high costs, significant safety risks, and difficulty adapting to individual differences. In recent years, researchers have attempted to automate and intelligentize drone training by combining virtual simulation environments with AI technologies. Deep reinforcement learning, with its capabilities for self-learning, adaptive adjustment, and long-term strategy optimization, has become a key technology for addressing the challenges of intelligent drone training.

[0003] In existing technologies, deep reinforcement learning primarily models drone control behaviors by constructing policy networks, enabling the model to continuously trial and error and optimize in a virtual environment, ultimately achieving superior flight control capabilities. However, these approaches generally suffer from the following limitations: First, most methods focus solely on the output of flight control strategies, ignoring the cognitive intent and motivational information behind student behavior. This lacks the ability to explain why students perform certain actions, resulting in the system's inability to provide categorized guidance for different cognitive errors. Second, the optimization process for training strategies typically utilizes a fixed optimizer and a uniform loss function, lacking the ability to adapt flexibly to individual operational deviations, learning styles, or erroneous habits. This makes it difficult to generalize training strategies to real-world application scenarios characterized by the uncertainty of human operation.

[0004] During actual training, different trainees exhibit significant differences in task understanding, motivational reasoning, and operational methods. For example, some trainees may inaccurately locate the target area, while others are prone to attitude control deviations during complex flight maneuvers. Existing methods are mostly based on imitation learning based on a single policy network or expert demonstration trajectories. However, they lack the ability to extract and hierarchically model individual behavioral characteristics, and cannot effectively support personalized training needs that vary from person to person. Furthermore, training feedback mechanisms are often crude, lacking identification and feedback on the root causes of incorrect behaviors, and are unable to provide trainees with clear improvement suggestions.

[0005] Some existing research has attempted to incorporate the Transformer architecture to model time series data, but these efforts have primarily focused on flight path prediction or image recognition. Systematic mechanisms for modeling UAV operational behavior, such as motivational information fusion, target alignment control, and cognitive bias identification, have yet to be established. Furthermore, in terms of optimization algorithms, traditional SGD or Adam optimizers are prone to falling into local optima when training in dynamic perturbation scenarios. These optimizers lack the ability to flexibly adapt strategies to the characteristics of perturbed behavior and lack mechanisms for regulating error tolerance and feedback sensitivity.

[0006] Therefore, how to provide a drone operation skill training system based on deep reinforcement learning is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0007] One objective of the present invention is to propose a drone operating skills training system based on deep reinforcement learning. This system integrates deep reinforcement learning, motivation modeling, and disturbance regulation methods to construct a training system based on the behavior transformer and the Lion optimizer, enabling a personalized, closed-loop learning process for drone operating skills. The system not only accurately models the relationship between flight behavior and mission motivation, but also identifies cognitive biases and dynamically schedules training strategies. This system offers significant advantages, including strong comprehension, high adaptability, excellent training efficiency, and high error correction accuracy. It effectively addresses the issues of poor adaptability to individual differences and crude feedback mechanisms in existing training methods.

[0008] A drone operation skill training system based on deep reinforcement learning according to an embodiment of the present invention includes the following modules:

[0009] The operation behavior data acquisition module is used to collect the flight state sequence, control instruction sequence and task motivation data of UAV operation trainees in a simulation training environment and construct an operation behavior dataset;

[0010] A data preprocessing module, configured to perform trajectory encoding and motivation encoding on the operation behavior dataset to generate a time series joint input tensor;

[0011] A behavior transformation module is used to input the time series joint input tensor into the behavior transformation Transformer model, model the interaction between trajectory features and task motivation in the time series dimension by introducing a motivation-constrained multi-head attention mechanism, and output an operation behavior representation vector;

[0012] A parameter update module, based on the Lion optimizer, for iteratively updating the training parameters of the behavior transformation Transformer module in a momentum-free manner;

[0013] a disturbance adjustment module, connected to the parameter update module and configured with a nonlinear mapping layer and a gating mechanism, for generating a direction-aware and amplitude-adjustable disturbance vector according to the operation behavior representation vector and the disturbance intensity factor, and adding the disturbance vector to the operation behavior representation vector to obtain a disturbance behavior representation;

[0014] A decoding module, configured to decode the disturbance behavior representation into a control action sequence;

[0015] A reversible motivation generation network module, comprising an encoder, a motivation reasoner, and a reconstruction verifier, is used to bidirectionally encode the control action sequence, generate a potential control strategy representation vector, and infer a reconstruction motivation vector;

[0016] a motivation comparison module, configured to compare the reconstructed motivation vector with the original task motivation vector generated in the task initialization phase by cosine similarity, calculate the motivation difference, and identify the target cognitive bias accordingly;

[0017] A training scheduling module is used to generate personalized training scheduling instructions based on the target cognitive bias and the disturbance behavior representation, update the task target parameters, environment difficulty parameters and action accuracy parameters in the training task configuration, and output new training scenarios and control strategies to the training simulation environment.

[0018] A method for training drone operation skills based on deep reinforcement learning according to an embodiment of the present invention includes the following steps:

[0019] S1. Collect flight control data of UAV operators in a simulation training environment and construct an operation behavior dataset.

[0020] S2. Preprocess the operation behavior dataset to construct a time series joint input tensor;

[0021] S3. Input the time series joint input tensor into the behavior transformation Transformer model, introduce the motivation-constrained multi-head attention mechanism, model the interaction between trajectory features and task motivation in the time series dimension, and output the operation behavior representation vector;

[0022] S4. During the generation of the action behavior representation, the parameters of the behavior transformer model are updated based on the Lion optimizer, and a disturbance adjustment function is introduced to add non-momentum disturbance values ​​to the action behavior representation vector to form a disturbance behavior representation.

[0023] S5. Decode the disturbance behavior representation into a control action sequence and send it to a reversible motivation generation network. Encode the control action sequence through a bidirectional structure and output a reconstructed motivation vector at the motivation prediction end.

[0024] S6. Compare the reconstructed motivation vector with the original task motivation vector, and calculate the target cognitive bias in the identification operation behavior based on the motivation difference;

[0025] S7. Based on the target cognitive bias and disturbance behavior representation, construct personalized training scheduling instructions, update the training task configuration, including task target parameters, environment difficulty parameters and action accuracy parameters, and generate new training scenarios and control strategies.

[0026] Optionally, the operational behavior data set specifically includes a flight state sequence, a control instruction sequence and mission motivation data.

[0027] Optionally, the preprocessing specifically includes using a trajectory encoding method to encode the flight state sequence and the control instruction sequence into a trajectory embedding vector, and encoding the mission motivation data into a mission motivation vector.

[0028] Optionally, the behavior transformation Transformer model includes a motivation-constrained multi-head attention mechanism for modeling the interactive relationship between trajectory features and task motivation, specifically including:

[0029] Construct a multi-head self-attention substructure for processing trajectory embedding vectors, perform attention weighting on the input temporal trajectory features, and extract the dependency of operation behaviors in the time dimension;

[0030] Construct a motivation-guided attention substructure for processing task motivation vectors, copy and extend the task motivation vectors to the same time step dimension as the trajectory sequence, and participate in the calculation of attention weights;

[0031] In each attention head, the attention weights between trajectory features and the interactive attention weights between the trajectory embedding vector and the task motivation vector are simultaneously calculated;

[0032] The representation vectors output by each attention head are concatenated and input into the feedforward fully connected network for dimensional mapping and nonlinear transformation;

[0033] Through residual connection and layer normalization operations, the output is transformed into an operation behavior representation vector.

[0034] Optionally, the S4 specifically includes:

[0035] S41. The action representation vector output by the behavior transformer model at each time step in the time series is used as the basic input for policy learning, where each action representation vector contains a high-dimensional representation of the current flight state and control intent.

[0036] S42, setting the training parameters of the behavior transformation Transformer model, and performing gradient calculation on the training parameters based on the loss function in each training round to obtain the gradient information of the current round;

[0037] S43. Use the Lion optimizer to update the training parameters, using a momentum-free method that does not rely on the accumulated gradient history to iterate the training parameters. The Lion optimizer directly modifies the training parameters of the behavior transformation Transformer model according to the direction of the current gradient in each iteration, thereby achieving rapid convergence.

[0038] S44. Constructing a disturbance adjustment function, wherein the disturbance adjustment function takes the operation behavior representation vector and the disturbance intensity factor as input, first transforms the operation behavior representation vector through a set of nonlinear mapping functions, then determines the channel activation intensity of the disturbance through a gating mechanism, and finally generates a disturbance vector with the same dimension as the operation behavior representation, wherein the disturbance generation process is associated with the original feature direction and has direction perception and amplitude adjustability;

[0039] S45, adding the generated disturbance vector to the original operation behavior representation vector to form a disturbance behavior representation, so as to enhance the adaptability and correction capability of the behavior transformation Transformer model to abnormal control behavior;

[0040] S46. During the entire training process, the disturbance intensity factor in the disturbance adjustment function is dynamically adjusted according to the change in strategy performance, so that the degree of disturbance influence changes intelligently with the training stage, and the disturbance is gradually controlled from enhanced exploration to convergence and contraction.

[0041] Optionally, the S5 specifically includes:

[0042] S51. Input the disturbance behavior representation into a decoder to convert it into a control action sequence, where the control action sequence is a multi-dimensional time series vector including throttle, pitch, roll, and yaw flight control instructions at each time step.

[0043] S52, inputting the control action sequence into a reversible motivation generation network, wherein the reversible motivation generation network includes an encoder and a motivation reasoner;

[0044] S53: bidirectionally encoding the control action sequence using a bidirectional temporal neural network in the encoder, extracting forward and backward temporal context features, and obtaining a compressed potential control strategy representation vector for representing the overall decision trend of the trainee's behavior;

[0045] S54. Input the potential control strategy representation vector into the motivation reasoner, which includes at least one fully connected neural network layer and a nonlinear activation unit, and is used to map the potential control strategy representation vector into a representation vector in the task motivation space to form a reconstructed motivation vector.

[0046] Optionally, the S6 specifically includes:

[0047] S61, obtaining a reconstructed motivation vector, wherein the reconstructed motivation vector is a potential task intention representation after encoding and reasoning the control action sequence;

[0048] S62, extracting an original task motivation vector corresponding to the current training task, wherein the original task motivation vector is generated during the task initialization phase and represents an expected execution goal;

[0049] S63, performing a positional comparison between the reconstructed motivation vector and the original task motivation vector, and calculating the motivation difference by cosine similarity to indicate the degree of deviation between the current behavior intention and the target task;

[0050] S64: Compare the motivation difference with a set cognitive bias determination threshold. If the motivation difference exceeds the cognitive bias determination threshold, it is determined that there is a target cognitive bias; otherwise, it is considered that the intentions are aligned.

[0051] S65. Label samples with target cognitive bias and classify them according to the degree of motivation difference to form bias type labels, which are used to distinguish different categories of cognitive misunderstandings, including target confusion, path misunderstanding, and task strategy misjudgment.

[0052] Optionally, the S7 specifically includes:

[0053] S71. Input the identified target cognitive bias and the perturbation behavior representation vector into the training scheduling analysis module, where the target cognitive bias is used to characterize the type of deviation in the trainee's understanding of the task, and the perturbation behavior representation is used to characterize the execution deviation characteristics in the operation strategy;

[0054] S72. In the training scheduling analysis module, matching policy rules is performed based on the deviation type and the behavior characteristics, and generating personalized training scheduling instructions according to the multi-condition decision logic, wherein the personalized training scheduling instructions include a set of modification instructions for multiple training configuration parameters;

[0055] S73. Adjust the task objective parameters of the training task based on the degree of target cognitive bias, including the expression of the task description, the spatial configuration of the task objective location, and the density of sequential prompts for the execution steps;

[0056] S74. Adjusting the difficulty parameters of the training environment based on the operational stability characteristics reflected in the disturbance behavior representation, including the number of obstacles in the environment, the intensity of dynamic disturbances, and external interference factors;

[0057] S75. Adjust the action accuracy parameters based on the target cognitive bias and the degree of operational disturbance, including the flight control command tolerance error range, control rhythm requirements, dynamic feedback frequency, and fault tolerance threshold settings;

[0058] S76. Based on the updated task objective parameters, environmental difficulty parameters, and action accuracy parameters, generate training scenarios and control strategies tailored to the current individual ability status of the trainee as input for the next stage of training.

[0059] The beneficial effects of the present invention are:

[0060] First, by introducing a motivation-constrained behavior transformation Transformer model, this paper breaks through the limitations of traditional deep learning models that make decisions based solely on state-action sequences. By deeply integrating flight trajectory characteristics with task motivation information, this model dynamically captures the correlation between student operational behaviors and target tasks in the temporal dimension through a motivation-guided multi-head attention mechanism. This effectively improves the target alignment of operational strategies and enhances the model's understanding of student flight intentions.

[0061] Secondly, the present invention achieves efficient updating of strategy parameters and dynamic disturbance generation during the training process by adopting a momentum-free disturbance control method based on the Lion optimizer. Unlike the fixed-step parameter update method of traditional optimization algorithms, the Lion optimizer has directional sensitivity and convergence acceleration characteristics, which can quickly adjust strategies in complex disturbance environments and improve the robustness of the model to unstable control behaviors. At the same time, combined with the nonlinear mapping and gating mechanism in the disturbance adjustment function, the disturbance value can be directionally sensed and amplitude controlled according to the characteristics of the operating behavior, effectively realizing a disturbance adjustment closed loop from early training exploration to later stable convergence.

[0062] In addition, the present invention constructs a reversible motivation generation network module, which for the first time realizes a bidirectional mapping structure that reversely infers task motivation from flight behavior. By bidirectionally encoding the control action sequence and inferring the potential motivation vector, and then comparing it with the original task goal, the network can effectively identify the trainee's target cognitive bias during execution. Compared with the existing technology that can only analyze problems at the level of action errors, the present invention introduces judgment criteria at the cognitive level, so that the training system not only knows what the trainee did wrong, but also why the student did it wrong, with significant interpretability and intervention.

[0063] Finally, the present invention implements personalized training configuration updates based on target cognitive biases and perturbation behavior characteristics through a training scheduling module. Based on the trainee's specific cognitive misunderstanding type, operational bias characteristics, and flight control stability, the system dynamically adjusts the target parameters, environmental complexity, and motion accuracy requirements of the training task. This generates training scenarios and control strategies tailored to the individual's ability status, enabling differentiated and precise guidance for trainees of varying levels and abilities, significantly improving training efficiency and the stability of operational skill acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0065] Figure 1 This is a structural diagram of a UAV operation skill training system based on deep reinforcement learning proposed by the present invention;

[0066] Figure 2 This is an overall flow chart of the drone operation skills training method based on deep reinforcement learning proposed in this invention. DETAILED DESCRIPTION

[0067] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0068] refer to Figure 1 , a drone operation skill training system based on deep reinforcement learning, including the following modules:

[0069] The operation behavior data acquisition module is used to collect the flight state sequence, control instruction sequence and task motivation data of UAV operation trainees in a simulation training environment and construct an operation behavior dataset;

[0070] A data preprocessing module, configured to perform trajectory encoding and motivation encoding on the operation behavior dataset to generate a time series joint input tensor;

[0071] A behavior transformation module is used to input the time series joint input tensor into the behavior transformation Transformer model, model the interaction between trajectory features and task motivation in the time series dimension by introducing a motivation-constrained multi-head attention mechanism, and output an operation behavior representation vector;

[0072] A parameter update module, based on the Lion optimizer, for iteratively updating the training parameters of the behavior transformation Transformer module in a momentum-free manner;

[0073] a disturbance adjustment module, connected to the parameter update module and configured with a nonlinear mapping layer and a gating mechanism, for generating a direction-aware and amplitude-adjustable disturbance vector according to the operation behavior representation vector and the disturbance intensity factor, and adding the disturbance vector to the operation behavior representation vector to obtain a disturbance behavior representation;

[0074] A decoding module, configured to decode the disturbance behavior representation into a control action sequence;

[0075] A reversible motivation generation network module, comprising an encoder, a motivation reasoner, and a reconstruction verifier, is used to bidirectionally encode the control action sequence, generate a potential control strategy representation vector, and infer a reconstruction motivation vector;

[0076] a motivation comparison module, configured to compare the reconstructed motivation vector with the original task motivation vector generated in the task initialization phase by cosine similarity, calculate the motivation difference, and identify the target cognitive bias accordingly;

[0077] A training scheduling module is used to generate personalized training scheduling instructions based on the target cognitive bias and the disturbance behavior representation, update the task target parameters, environment difficulty parameters and action accuracy parameters in the training task configuration, and output new training scenarios and control strategies to the training simulation environment.

[0078] This system achieves comprehensive modeling and dynamic training of drone operating skills by constructing a deep reinforcement learning architecture consisting of modules such as acquisition, preprocessing, modeling, optimization, perturbation generation, intent reasoning, and personalized scheduling. The modules form a collaborative closed loop, which not only enables high-dimensional encoding of students' flight behaviors and alignment with target motivations, but also improves the robustness of the model in unstable operating situations with the help of perturbation mechanisms. At the same time, the system introduces a motivation back-inference structure that supports reverse semantic interpretation of control action sequences, significantly enhancing the interpretability and adjustability of the training process. The overall architecture supports on-demand adjustment of training tasks and strategy content, solving the problems of traditional flight training systems such as lack of personalized adaptation, ambiguous feedback, and shallow understanding of behavior. It has cognitive intervention capabilities and full-process adaptive adjustment capabilities.

[0079] refer to Figure 2 , a UAV operation skill training method based on deep reinforcement learning, including the following steps:

[0080] S1. Collect flight control data of UAV operators in a simulation training environment and construct an operation behavior dataset.

[0081] S2. Preprocess the operation behavior dataset to construct a time series joint input tensor;

[0082] S3. Input the time series joint input tensor into the behavior transformation Transformer model, introduce the motivation-constrained multi-head attention mechanism, model the interaction between trajectory features and task motivation in the time series dimension, and output the operation behavior representation vector;

[0083] S4. During the generation of the action behavior representation, the parameters of the behavior transformer model are updated based on the Lion optimizer, and a disturbance adjustment function is introduced to add non-momentum disturbance values ​​to the action behavior representation vector to form a disturbance behavior representation.

[0084] S5. Decode the disturbance behavior representation into a control action sequence and send it to a reversible motivation generation network. Encode the control action sequence through a bidirectional structure and output a reconstructed motivation vector at the motivation prediction end.

[0085] S6. Compare the reconstructed motivation vector with the original task motivation vector, and calculate the target cognitive bias in the identification operation behavior based on the motivation difference;

[0086] S7. Based on the target cognitive bias and disturbance behavior representation, construct personalized training scheduling instructions, update the training task configuration, including task target parameters, environment difficulty parameters and action accuracy parameters, and generate new training scenarios and control strategies.

[0087] The proposed training process achieves a complete closed loop from data collection and model building to feedback scheduling. By introducing a motivation-guidance mechanism into the process, behavioral modeling becomes more accurate, improving the semantic matching capability of strategy generation. The perturbation-guided strategy diversification during training improves the model's generalization ability in flight control. At the same time, based on a reversible motivation-based reasoning network and a difference comparison mechanism, it is able to identify deviations in students' understanding of mission objectives and dynamically adjust subsequent training content accordingly, thus combining cognitive bias identification with targeted training. The entire training process is scientific, rigorous, and dynamically adaptable, significantly improving the accuracy, efficiency, and intelligent feedback capabilities of operational training.

[0088] In this embodiment, the operation behavior data set specifically includes a flight state sequence, a control instruction sequence and mission motivation data.

[0089] In this embodiment, the preprocessing specifically includes using a trajectory encoding method to encode the flight state sequence and the control instruction sequence into a trajectory embedding vector, and encoding the mission motivation data into a mission motivation vector.

[0090] In this embodiment, the behavior transformation Transformer model includes a motivation-constrained multi-head attention mechanism for modeling the interaction between trajectory features and task motivation, specifically including:

[0091] Construct a multi-head self-attention substructure for processing trajectory embedding vectors, perform attention weighting on the input temporal trajectory features, and extract the dependency of operation behaviors in the time dimension;

[0092] Construct a motivation-guided attention substructure for processing task motivation vectors, copy and extend the task motivation vectors to the same time step dimension as the trajectory sequence, and participate in the calculation of attention weights;

[0093] In each attention head, the attention weights between trajectory features and the interactive attention weights between the trajectory embedding vector and the task motivation vector are simultaneously calculated;

[0094] The representation vectors output by each attention head are concatenated and input into the feedforward fully connected network for dimensional mapping and nonlinear transformation;

[0095] Through residual connection and layer normalization operations, the output is transformed into an operation behavior representation vector.

[0096] The motivation-constrained multi-head attention mechanism employed overcomes the limitations of traditional self-attention models, which rely solely on feature extraction based on internal correlations within the input sequence. By introducing the task motivation vector as an interaction signal and integrating trajectory features with the motivation signal in the attention calculation, the model can more accurately focus on and model time series segments that are strongly correlated with the task objective, thereby enhancing the policy's ability to perceive task semantics. This mechanism significantly improves the model's task alignment accuracy and the rationality of policy generation, effectively reducing the risk of task deviation and incorrect execution.

[0097] In this embodiment, the S4 specifically includes:

[0098] S41. The action representation vector output by the behavior transformer model at each time step in the time series is used as the basic input for policy learning, where each action representation vector contains a high-dimensional representation of the current flight state and control intent.

[0099] S42, setting the training parameters of the behavior transformation Transformer model, and performing gradient calculation on the training parameters based on the loss function in each training round to obtain the gradient information of the current round;

[0100] S43. Use the Lion optimizer to update the training parameters, using a momentum-free method that does not rely on the accumulated gradient history to iterate the training parameters. The Lion optimizer directly modifies the training parameters of the behavior transformation Transformer model according to the direction of the current gradient in each iteration, thereby achieving rapid convergence.

[0101] S44. Constructing a disturbance adjustment function, wherein the disturbance adjustment function takes the operation behavior representation vector and the disturbance intensity factor as input, first transforms the operation behavior representation vector through a set of nonlinear mapping functions, then determines the channel activation intensity of the disturbance through a gating mechanism, and finally generates a disturbance vector with the same dimension as the operation behavior representation, wherein the disturbance generation process is associated with the original feature direction and has direction perception and amplitude adjustability;

[0102] S45, adding the generated disturbance vector to the original operation behavior representation vector to form a disturbance behavior representation, so as to enhance the adaptability and correction capability of the behavior transformation Transformer model to abnormal control behavior;

[0103] S46. During the entire training process, the disturbance intensity factor in the disturbance adjustment function is dynamically adjusted according to the change in strategy performance, so that the degree of disturbance influence changes intelligently with the training stage, and the disturbance is gradually controlled from enhanced exploration to convergence and contraction.

[0104] The Lion optimizer's momentum-free parameter update mechanism and adjustable perturbation function design significantly improve the stability and exploration capabilities of the policy training process. The optimizer achieves efficient convergence through gradient direction updates, overcoming the response lag that can result from reliance on gradient history in momentum optimization. The perturbation adjustment function, combining nonlinear mapping with a gating mechanism, automatically adjusts the perturbation amplitude based on the directionality of the behavior representation and the training phase, effectively ensuring both policy diversity and training stability, and enhancing the model's adaptability to individual operational differences and misoperation.

[0105] In this embodiment, the S5 specifically includes:

[0106] S51. Input the disturbance behavior representation into a decoder to convert it into a control action sequence, where the control action sequence is a multi-dimensional time series vector including throttle, pitch, roll, and yaw flight control instructions at each time step.

[0107] S52, inputting the control action sequence into a reversible motivation generation network, wherein the reversible motivation generation network includes an encoder and a motivation reasoner;

[0108] S53: bidirectionally encoding the control action sequence using a bidirectional temporal neural network in the encoder, extracting forward and backward temporal context features, and obtaining a compressed potential control strategy representation vector for representing the overall decision trend of the trainee's behavior;

[0109] S54. Input the potential control strategy representation vector into the motivation reasoner, which includes at least one fully connected neural network layer and a nonlinear activation unit, and is used to map the potential control strategy representation vector into a representation vector in the task motivation space to form a reconstructed motivation vector.

[0110] By constructing a reversible motivation generation network with an encoder and motivation reasoner, the system achieves the ability to reversely infer task intent from action execution. This network utilizes bidirectional temporal modeling to extract the contextual associations of control behaviors and, through a nonlinear structure, generates interpretable motivation representations that can be verified for consistency with the original task motivation. This mechanism provides theoretical support for the identification of cognitive biases and significantly enhances the system's ability to understand the underlying intentions behind complex behavioral sequences, providing a cognitive-level regulatory basis for personalized training.

[0111] In this embodiment, S6 specifically includes:

[0112] S61, obtaining a reconstructed motivation vector, wherein the reconstructed motivation vector is a potential task intention representation after encoding and reasoning the control action sequence;

[0113] S62, extracting an original task motivation vector corresponding to the current training task, wherein the original task motivation vector is generated during the task initialization phase and represents an expected execution goal;

[0114] S63, performing a positional comparison between the reconstructed motivation vector and the original task motivation vector, and calculating the motivation difference by cosine similarity to indicate the degree of deviation between the current behavior intention and the target task;

[0115] S64: Compare the motivation difference with a set cognitive bias determination threshold. If the motivation difference exceeds the cognitive bias determination threshold, it is determined that there is a target cognitive bias; otherwise, it is considered that the intentions are aligned.

[0116] S65. Label samples with target cognitive bias and classify them according to the degree of motivation difference to form bias type labels, which are used to distinguish different categories of cognitive misunderstandings, including target confusion, path misunderstanding, and task strategy misjudgment.

[0117] By comparing the reconstructed motivation vector with the original task motivation vector for similarity and introducing deviation thresholds and difference analysis, the system can effectively identify specific deviations in the trainee's task understanding, such as goal confusion or path misunderstanding. This deviation identification mechanism establishes a connection between operational behavior and cognitive judgment, resolving the problem that traditional training systems only provide feedback on action execution and fail to identify underlying errors in intent. This module not only improves the accuracy of training feedback but also provides a core judgment basis for dynamic adjustment of training tasks.

[0118] In this embodiment, the S7 specifically includes:

[0119] S71. Input the identified target cognitive bias and the perturbation behavior representation vector into the training scheduling analysis module, where the target cognitive bias is used to characterize the type of deviation in the trainee's understanding of the task, and the perturbation behavior representation is used to characterize the execution deviation characteristics in the operation strategy;

[0120] S72. In the training scheduling analysis module, matching policy rules is performed based on the deviation type and the behavior characteristics, and generating personalized training scheduling instructions according to the multi-condition decision logic, wherein the personalized training scheduling instructions include a set of modification instructions for multiple training configuration parameters;

[0121] S73. Adjust the task objective parameters of the training task based on the degree of target cognitive bias, including the expression of the task description, the spatial configuration of the task objective location, and the density of sequential prompts for the execution steps;

[0122] S74. Adjusting the difficulty parameters of the training environment based on the operational stability characteristics reflected in the disturbance behavior representation, including the number of obstacles in the environment, the intensity of dynamic disturbances, and external interference factors;

[0123] S75. Adjust the action accuracy parameters based on the target cognitive bias and the degree of operational disturbance, including the flight control command tolerance error range, control rhythm requirements, dynamic feedback frequency, and fault tolerance threshold settings;

[0124] S76. Based on the updated task objective parameters, environmental difficulty parameters, and action accuracy parameters, generate training scenarios and control strategies tailored to the current individual ability status of the trainee as input for the next stage of training.

[0125] By inputting the identified target cognitive biases and perturbation behavior characteristics into the training scheduling module, the system generates multi-dimensional personalized training scheduling instructions and automatically adjusts key parameters such as task objectives, environmental difficulty, and movement accuracy. Compared to traditional fixed-process training, this approach enables on-demand customized strategy optimization. The system can generate training content more closely aligned with the operational capabilities of trainees of varying ability levels and error types, effectively improving the targeted nature of training and the efficiency of operational corrections, offering the advantages of adaptive, differentiated, and closed-loop control.

[0126] Example 1:

[0127] To verify the feasibility of this invention, a simulation training platform at a national-level UAV flight training center was used to conduct a four-week flight operations training program. This platform, designed for junior UAV operators in military and police units, includes a variety of simulated mission scenarios, such as fixed-point patrol, target identification, complex obstacle traversal, and emergency response. The training center has long used fixed mission templates and a rule-based feedback mechanism for operational training, which has resulted in a monotonous training schedule, difficulty identifying cognitive biases, and a limited feedback dimension.

[0128] In this implementation, the system uses trainee flight control data as its core input, collecting flight state sequences such as throttle control, attitude angle, and control surface adjustments generated during simulated flight, as well as flight motivation data set by the trainee before the mission, including target location, mission type, and flight path preference. Data collection covers 30 trainees, with each trainee completing three different mission types per day. Each type of mission is repeated five times, for a total training period of 28 days.

[0129] The data is first preprocessed using a combined trajectory encoding and motivation encoding method to generate a time series joint input tensor. This tensor is then fed into the proposed motivation-constrained behavior transformer model for modeling. The model dynamically analyzes the association between trajectory changes and motivation information in the time series using a multi-head attention mechanism, thereby obtaining a high-dimensional representation vector of the action behavior.

[0130] During training, the system uses the Lion optimizer to efficiently update Transformer parameters, ensuring training stability under perturbation conditions. Furthermore, a perturbation adjustment function is introduced to automatically adjust the perturbation intensity based on policy performance at each iteration. This perturbation intensity is then incorporated into the action representation vector to form a directionally aware perturbation behavior representation, thereby increasing policy diversity and enhancing the model's ability to identify abnormal behavior.

[0131] For example, student A034 repeatedly experienced unstable altitude control during obstacle traversal and fixed-point landing missions. The system identified a cosine similarity of only 0.68 between the reconstructed motivation vector and the original mission motivation, indicating a significant cognitive bias. Further analysis revealed that the student misjudged the timing demarcation points of the mission phases, resulting in premature flight phase transitions. The system therefore identified this as a cognitive bias caused by misjudgment of mission strategy.

[0132] The system then generated customized training scheduling instructions based on A034's target cognitive bias and perturbation behavior. These included simplifying phase transition instructions in the mission description, increasing the proportion of training during the altitude hold phase, reducing the number of dynamic obstacles in the training environment, and relaxing the accuracy threshold for flight maneuvers by 10%. After 15 rounds of personalized scheduling training over three consecutive days, the student's mission achievement rate increased from 65% to 88%, and the altitude control error decreased from ±8.5% to ±3.2%, significantly improving training efficiency and performance stability.

[0133] Fifteen of the 30 participants in the experimental group were tested against the proposed system and the traditional training process. The results showed that participants using the proposed system saw their average task completion rate increase by 12.6 percentage points, reaching 91.2%. The recognition rate for cognitive biases reached 94.5%, with an average correction time of just 17 minutes, compared to over 45 minutes using the traditional method. Furthermore, the average training period required to achieve a 90% task completion rate was shortened to 2.3 weeks, nearly half the time of traditional training.

[0134] This example fully demonstrates the core advantages of the present invention: by constructing a motivation-guided behavior modeling network and disturbance regulation mechanism, the system effectively enhances its ability to understand trainees' operational behaviors and correct erroneous behaviors; by identifying cognitive biases through a reversible motivational reasoning and comparison mechanism, the feedback dimension is enhanced; and finally, through personalized training scheduling, a more adaptive training task configuration is generated, achieving an efficient and accurate path for improving flight skills. The practicality, scientificity, and intelligence of the system constructed by this invention in drone operation skills training far surpass existing traditional training models, and it has significant application value and promotion prospects.

[0135] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A UAV operation skill training system based on deep reinforcement learning, characterized by: Includes the following modules: The operation behavior data acquisition module is used to collect the flight state sequence, control instruction sequence and task motivation data of UAV operation trainees in a simulation training environment and construct an operation behavior dataset; A data preprocessing module, configured to perform trajectory encoding and motivation encoding on the operation behavior dataset to generate a time series joint input tensor; A behavior transformation module is used to input the time series joint input tensor into the behavior transformation Transformer model, model the interaction between trajectory features and task motivation in the time series dimension by introducing a motivation-constrained multi-head attention mechanism, and output an operation behavior representation vector; A parameter update module, based on the Lion optimizer, for iteratively updating the training parameters of the behavior transformation Transformer module in a momentum-free manner; a disturbance adjustment module, connected to the parameter update module and configured with a nonlinear mapping layer and a gating mechanism, for generating a direction-aware and amplitude-adjustable disturbance vector according to the operation behavior representation vector and the disturbance intensity factor, and adding the disturbance vector to the operation behavior representation vector to obtain a disturbance behavior representation; A decoding module, configured to decode the disturbance behavior representation into a control action sequence; A reversible motivation generation network module, comprising an encoder and a motivation reasoner, for bidirectionally encoding the control action sequence, generating a potential control strategy representation vector, and inferring a reconstruction motivation vector; a motivation comparison module, configured to compare the reconstructed motivation vector with the original task motivation vector generated in the task initialization phase by cosine similarity, calculate the motivation difference, and identify the target cognitive bias accordingly; A training scheduling module is used to generate personalized training scheduling instructions based on the target cognitive bias and the disturbance behavior representation, update the task target parameters, environment difficulty parameters and action accuracy parameters in the training task configuration, and output new training scenarios and control strategies to the training simulation environment.

2. The UAV operation skill training system based on deep reinforcement learning according to claim 1 is characterized in that: The modules are implemented as follows: S1. Collect flight control data of UAV operators in a simulation training environment and construct an operation behavior dataset. S2. Preprocess the operation behavior dataset to construct a time series joint input tensor; S3. Input the time series joint input tensor into the behavior transformation Transformer model, introduce the motivation-constrained multi-head attention mechanism, model the interaction between trajectory features and task motivation in the time series dimension, and output the operation behavior representation vector; S4. During the generation of the action behavior representation, the parameters of the behavior transformer model are updated based on the Lion optimizer, and a disturbance adjustment function is introduced to add non-momentum disturbance values ​​to the action behavior representation vector to form a disturbance behavior representation. S5. Decode the disturbance behavior representation into a control action sequence and send it to a reversible motivation generation network. Encode the control action sequence through a bidirectional structure and output a reconstructed motivation vector at the motivation prediction end. S6. Compare the reconstructed motivation vector with the original task motivation vector, and calculate the target cognitive bias in the identification operation behavior based on the motivation difference; S7. Based on the target cognitive bias and disturbance behavior representation, construct personalized training scheduling instructions, update the training task configuration, including task target parameters, environment difficulty parameters and action accuracy parameters, and generate new training scenarios and control strategies.

3. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The operational behavior data set specifically includes a flight state sequence, a control instruction sequence, and mission motivation data.

4. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The preprocessing specifically includes using a trajectory encoding method to encode the flight state sequence and the control instruction sequence into a trajectory embedding vector, and encoding the mission motivation data into a mission motivation vector.

5. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The behavior transformation Transformer model includes a motivation-constrained multi-head attention mechanism for modeling the interaction between trajectory features and task motivation, specifically including: Construct a multi-head self-attention substructure for processing trajectory embedding vectors, perform attention weighting on the input temporal trajectory features, and extract the dependency of operation behaviors in the time dimension; Construct a motivation-guided attention substructure for processing task motivation vectors, copy and extend the task motivation vectors to the same time step dimension as the trajectory sequence, and participate in the calculation of attention weights; In each attention head, the attention weights between trajectory features and the interactive attention weights between the trajectory embedding vector and the task motivation vector are simultaneously calculated; The representation vectors output by each attention head are concatenated and input into the feedforward fully connected network for dimensional mapping and nonlinear transformation; Through residual connection and layer normalization operations, the output is transformed into an operation behavior representation vector.

6. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The S4 specifically includes: S41. The action representation vector output by the behavior transformer model at each time step in the time series is used as the basic input for policy learning, where each action representation vector contains a high-dimensional representation of the current flight state and control intent. S42, setting the training parameters of the behavior transformation Transformer model, and performing gradient calculation on the training parameters based on the loss function in each training round to obtain the gradient information of the current round; S43. Use the Lion optimizer to update the training parameters, using a momentum-free method that does not rely on the accumulated gradient history to iterate the training parameters. The Lion optimizer directly modifies the training parameters of the behavior transformation Transformer model according to the direction of the current gradient in each iteration, thereby achieving rapid convergence. S44. Constructing a disturbance adjustment function, wherein the disturbance adjustment function takes the operation behavior representation vector and the disturbance intensity factor as input, first transforms the operation behavior representation vector through a set of nonlinear mapping functions, then determines the channel activation intensity of the disturbance through a gating mechanism, and finally generates a disturbance vector with the same dimension as the operation behavior representation, wherein the disturbance generation process is associated with the original feature direction and has direction perception and amplitude adjustability; S45, adding the generated disturbance vector to the original operation behavior representation vector to form a disturbance behavior representation, so as to enhance the adaptability and correction capability of the behavior transformation Transformer model to abnormal control behavior; S46. During the entire training process, the disturbance intensity factor in the disturbance adjustment function is dynamically adjusted according to the change in strategy performance, so that the degree of disturbance influence changes intelligently with the training stage, and the disturbance is gradually controlled from enhanced exploration to convergence and contraction.

7. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The S5 specifically includes: S51. Input the disturbance behavior representation into a decoder to convert it into a control action sequence, where the control action sequence is a multi-dimensional time series vector including throttle, pitch, roll, and yaw flight control instructions at each time step. S52, inputting the control action sequence into a reversible motivation generation network, wherein the reversible motivation generation network includes an encoder and a motivation reasoner; S53: bidirectionally encoding the control action sequence using a bidirectional temporal neural network in the encoder, extracting forward and backward temporal context features, and obtaining a compressed potential control strategy representation vector for representing the overall decision trend of the trainee's behavior; S54. Input the potential control strategy representation vector into the motivation reasoner, which includes at least one fully connected neural network layer and a nonlinear activation unit, and is used to map the potential control strategy representation vector into a representation vector in the task motivation space to form a reconstructed motivation vector.

8. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The S6 specifically includes: S61, obtaining a reconstructed motivation vector, wherein the reconstructed motivation vector is a potential task intention representation after encoding and reasoning the control action sequence; S62, extracting an original task motivation vector corresponding to the current training task, wherein the original task motivation vector is generated during the task initialization phase and represents an expected execution goal; S63, performing a positional comparison between the reconstructed motivation vector and the original task motivation vector, and calculating the motivation difference by cosine similarity to indicate the degree of deviation between the current behavior intention and the target task; S64: Compare the motivation difference with a set cognitive bias determination threshold. If the motivation difference exceeds the cognitive bias determination threshold, it is determined that there is a target cognitive bias; otherwise, it is considered that the intentions are aligned. S65. Label samples with target cognitive bias and classify them according to the degree of motivation difference to form bias type labels, which are used to distinguish different categories of cognitive misunderstandings, including target confusion, path misunderstanding, and task strategy misjudgment.

9. The UAV operation skill training system based on deep reinforcement learning according to claim 2 is characterized in that: The S7 specifically includes: S71. Input the identified target cognitive bias and the perturbation behavior representation vector into the training scheduling analysis module, where the target cognitive bias is used to characterize the type of deviation in the trainee's understanding of the task, and the perturbation behavior representation is used to characterize the execution deviation characteristics in the operation strategy; S72. In the training scheduling analysis module, matching policy rules is performed based on the deviation type and the behavior characteristics, and generating personalized training scheduling instructions according to the multi-condition decision logic, wherein the personalized training scheduling instructions include a set of modification instructions for multiple training configuration parameters; S73. Adjust the task objective parameters of the training task based on the degree of target cognitive bias, including the expression of the task description, the spatial configuration of the task objective location, and the density of sequential prompts for the execution steps; S74. Adjusting the difficulty parameters of the training environment based on the operational stability characteristics reflected in the disturbance behavior representation, including the number of obstacles in the environment, the intensity of dynamic disturbances, and external interference factors; S75. Adjust the action accuracy parameters based on the target cognitive bias and the degree of operational disturbance, including the flight control command tolerance error range, control rhythm requirements, dynamic feedback frequency, and fault tolerance threshold settings; S76. Based on the updated task objective parameters, environmental difficulty parameters, and action accuracy parameters, generate training scenarios and control strategies tailored to the current individual ability status of the trainee as input for the next stage of training.

Images