An agent aircraft construction method for cloud box system security simulation verification

Abstract

This invention discloses a method for constructing an intelligent agent aircraft for security simulation and verification of cloud-based subsystems, belonging to the technical fields of cloud-based subsystem simulation, aircraft motion state simulation, task behavior simulation, and data-driven intelligent agent modeling. The method divides the intelligent agent aircraft into two levels: motion state simulation and task behavior simulation. It can describe the temporal evolution of low-level states such as aircraft attitude and speed, and also express task behaviors such as waypoint tracking, flight phase transitions, and flight constraint maintenance, improving the completeness of aircraft behavior simulation. A random forest method is used to select key input features, and an Attention-BiGRU model is combined for temporal modeling of motion states. This reduces the impact of redundant inputs on model training, improves the accuracy and stability of aircraft motion state simulation, and enables the model to inherit basic behavioral patterns from historical flight samples while further enhancing its continuous task execution capabilities through reinforcement learning.

Description

Technical Field

[0001] This invention belongs to the technical fields of cloud-based system simulation, aircraft motion state simulation, mission behavior simulation, and data-driven intelligent agent modeling, specifically a method for constructing an intelligent agent aircraft for the safety simulation and verification of cloud-based systems. Background Technology

[0002] With the continuous expansion of my country's civil aviation transport scale and route network, the interaction between aircraft, airports, air traffic management, and operational support elements has become more complex, leading to increasing demands for safety monitoring, operational analysis, and simulation within the civil aviation system. As the core dynamic carrier in the system's operation, the aircraft's state changes and mission execution processes directly impact the results of safety evolution analysis. To enhance the acquisition, continuous monitoring, and safety analysis capabilities of aircraft operational data, the "cloud box" system, encompassing the entire aircraft operation process, has gradually become a crucial technological direction for supporting civil aviation operational safety. The "cloud box" continuously collects, transmits, stores, and analyzes aircraft operational status, flight parameters, mission processes, and related environmental information, providing a data foundation for aircraft operation monitoring, risk identification, safety assessment, and simulation. However, the construction and application of the cloud box system require verification of its data access, state identification, operational analysis, and safety simulation capabilities. Relying entirely on real aircraft for verification is not only costly and organizationally challenging but also fails to cover verification needs related to abnormal operations, complex routes, flight phase transitions, and high-risk scenarios. Therefore, constructing a simulation model that can characterize the aircraft's operation process is an important foundation for carrying out safety simulation verification of the "cloud box" system.

[0003] Existing aircraft process simulation methods mainly include simplified trajectory simulation methods and system-level complex simulation methods. The former is mostly based on trajectory, position, or a small number of motion parameters, with low computational cost, but its ability to express aircraft attitude response, control processes, and mission execution behavior is limited, making it difficult to fully reflect the relationship between state evolution and maneuvering behavior. The latter can describe multi-agent operational relationships and complex interaction processes, but the model construction cost is high, the state dimension is large, and the deployment is complex, resulting in a high threshold for engineering application. Therefore, existing methods still have a contradiction between behavioral expression capability and engineering feasibility, making it difficult to simultaneously consider model credibility and implementation cost, and also making it difficult to directly meet the requirements of cloud box system simulation verification for the continuity, stability, and mission adaptability of dynamic flight objects.

[0004] Based on this, the present invention provides a method for constructing an intelligent agent aircraft for security simulation verification of cloud box subsystems. Summary of the Invention

[0005] This invention provides a method for constructing an intelligent agent aircraft for safety simulation verification of a cloud-based system. Based on QAR flight data, this method divides the intelligent agent aircraft into two levels: motion state simulation and task behavior simulation. Through state-time modeling, expert prior guidance, and reinforcement learning optimization, it achieves continuous generation of aircraft flight behavior, thereby enabling safety simulation verification of the "cloud-based" system.

[0006] The technical solution adopted by this invention to solve its technical problem is: a method for constructing an intelligent agent aircraft for security simulation verification of a cloud-based subsystem, comprising the following steps: S1 Intelligent Agent Aircraft Motion State Simulation Model Construction: Based on real civil aviation flight data recorded by the airborne QAR system, motion state training samples are constructed; pitch angle, roll angle, heading angle, indicated airspeed, vertical speed, and ground speed are selected as motion state representations; the random forest feature importance evaluation method is used to screen key input features; based on the screened key input features, an Attention-BiGRU motion state simulation model is constructed to describe the temporal evolution relationship of flight state; S2 intelligent agent aircraft mission behavior simulation model construction: Based on historical flight records, an expert dataset is constructed, the behavior cloning method is used to complete the pre-training of the mission policy, and dynamic consistency constraints are introduced to optimize the pre-training objective function; based on the pre-trained policy, the PPO reinforcement learning algorithm is used to optimize the policy and construct the mission behavior simulation model. S3 Intelligent Agent Aircraft Integration: Integrates motion state simulation models with task behavior simulation models to generate intelligent agent aircraft that can be used for waypoint tracking, flight phase transitions, and route constraint maintenance. It is then connected to the cloud box system simulation scenario to conduct safety simulation verification.

[0007] Preferably, the specific meaning of the motion state representation in step S1 is as follows: pitch angle, roll angle, and heading angle are used to describe changes in aircraft attitude, while airspeed, vertical speed, and ground speed are used to describe the aircraft speed state and trajectory propulsion process.

[0008] Preferably, the specific process of the random forest feature importance assessment method in step S1 is as follows: Calculate the out-of-bag sample error: ,in This is the actual value. The value is the predicted value, and n is the number of samples. After perturbing the j-th candidate feature, recalculate the out-of-bag error (OOBError). j,k , where k represents the kth repeated perturbation, repeated m times; Importance score of the j-th candidate feature: ; Based on the feature importance scores, parameters that rank highly or have scores above a preset threshold are selected as key input features.

[0009] Preferably, the construction process of the Attention-BiGRU motion state simulation model in step S1 is as follows: Using the time window sequence of key input features as input, bidirectional temporal information is extracted through forward GRU and backward GRU to obtain the hidden state. ; A self-attention mechanism is introduced to weight the hidden states at different time steps and output attention features; The attention features are input into the fully connected layer, which outputs the six-dimensional motion state at the next time step.

[0010] Preferably, the expert dataset in step S2 is organized in a one-step state transition form of "current observation state, expert action, and next moment state"; the observation state includes aircraft attitude and speed state, current position, target waypoint information, and control reference quantities.

[0011] Preferably, the action vector of the task behavior simulation model in step S2 is divided into an incremental adjustment channel and an absolute target channel; during the training phase, the actions of each dimension are scaled to a uniform numerical range, and during the execution phase, they are scaled back to physical actions.

[0012] Preferably, the pre-training objective function for introducing dynamic consistency constraints in step S2 is: , where L action Huber loss is used for motion fitting loss; L dyn is the dynamic consistency loss, and is the standardized error between the predicted state and the actual state; These are the weighting coefficients.

[0013] Preferably, the reward function of the PPO reinforcement learning algorithm in step S2 is composed of a weighted average of the trajectory tracking reward, the state constraint cost, and the action constraint cost: Rtrack includes proximity reward, distance reduction reward, and trajectory matching reward; Cstate includes envelope crossover cost and state change rate exceedance cost; Caction includes incremental control amplitude cost, control jitter cost, action saturation cost, and gear shifting cost. , These are the weighting coefficients.

[0014] Preferably, the PPO reinforcement learning algorithm in step S2 employs a pruning objective function: ,in Let be the probability ratio of the new and old strategies, and At be the generalized advantage estimate. These are the trimming parameters.

[0015] Preferably, the operation process of the intelligent agent aircraft integration in step S3 is as follows: The mission behavior simulation model outputs control actions based on the current flight status and route mission constraints; The motion state simulation model updates the aircraft's attitude and velocity states based on control actions and key input features; Position status is determined by kinematic propagation based on velocity, heading, and sampling period; The above process is repeated to generate continuous flight process data containing position, attitude, velocity, and trajectory information.

[0016] The advantages of this invention are: (1) This invention divides the intelligent aircraft into two levels: motion state simulation and task behavior simulation. It can describe the temporal evolution of the aircraft's attitude, speed and other underlying states, and express task behaviors such as waypoint tracking, flight phase transition and flight constraint maintenance, thereby improving the completeness of aircraft behavior simulation.

[0017] (2) The present invention uses the random forest method to screen key input features and combines the Attention-BiGRU model to perform temporal modeling of motion state, which can reduce the impact of redundant input on model training and improve the accuracy and stability of aircraft motion state simulation.

[0018] (3) In the simulation of task behavior, the present invention adopts a combination of expert prior pre-training and PPO strategy optimization, so that the model can inherit the basic behavior rules in historical flight samples and further improve the continuous task execution capability through reinforcement learning.

[0019] (4) The present invention can generate continuous and stable intelligent aircraft flight behavior with low engineering deployment cost, and can provide model support for the safety simulation and verification of the "cloud box" system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a general framework diagram of an intelligent agent aircraft construction method for security simulation verification of cloud box system according to the present invention; Figure 2 This is a diagram of the Attention-enhanced BiGRU network architecture in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] The specific technical solution is as follows: I. Simulation of the Motion State of Intelligent Agent Aircraft 1. Representation of motion state Based on real civil aviation flight data recorded by an airborne QAR system, pitch angle, roll angle, yaw angle, indicated airspeed, vertical speed, and ground speed are selected as the representations of the aircraft's motion state by the intelligent agent. Pitch angle, roll angle, and yaw angle are used to describe changes in aircraft attitude, while indicated airspeed, vertical speed, and ground speed are used to describe the aircraft's velocity state and trajectory propulsion process. Through the joint representation of attitude and velocity quantities, a set of output variables is formed to describe the evolution of the aircraft's motion state.

[0024] 2. Key Feature Filtering To address the issue of numerous dimensions and significant redundancy in raw flight parameters, this paper selects candidate input parameters from flight control variables, attitude-related variables, and velocity-related variables based on aircraft motion mechanisms. A random forest feature importance assessment method is employed to analyze the correlation between candidate input parameters and the six motion state variables, and key input features with strong relationships to changes in aircraft motion state are selected based on feature importance scores. This key feature selection reduces the dimensionality of the model input and minimizes the interference of low-relevance parameters on motion state modeling.

[0025] 3. Motion state temporal modeling based on Attention-BiGRU Based on the key input features obtained through screening, an Attention-BiGRU motion state simulation model is constructed. This model takes the time window sequence of key input features as input, uses bidirectional gated recurrent units to extract the sequential information of the flight state sequence, and highlights the influence of key time steps on the evolution of the current state through an attention mechanism. Finally, it outputs the motion state at the next moment, thereby constructing an intelligent agent aircraft motion state simulation model and providing a state response basis for subsequent task behavior simulation.

[0026] II. Simulation of Intelligent Agent Aircraft Mission Behavior 1. Expert-guided strategy pre-training To meet the mission requirements of waypoint tracking, flight phase transitions, and flight constraint maintenance, an expert dataset is constructed based on historical flight records. The expert data is organized in a one-step state transition format of "current observation state, expert action, and next moment state". The observation state includes the aircraft attitude and velocity state, current position, target waypoint information, and mission reference information. The expert action represents the adjustment results of control and target variables during mission execution.

[0027] Based on this, a behavior cloning method is used to pre-train the task behavior simulation model, learning the mapping relationship between observed states and control actions in expert samples, so that the model has basic task execution capabilities before entering reinforcement learning training. To reduce the error accumulation problem that may occur in continuous task execution due to simple action fitting, a dynamic consistency constraint is introduced, so that the policy output action not only closely resembles the expert action, but also corresponds to a more reasonable one-step state response.

[0028] 2. PPO-based strategy optimization Based on expert prior training, a proximal policy optimization algorithm is used to optimize the task behavior simulation model. The optimization phase adopts the observation space, action semantics, and scale transformation rules from the pre-training phase, ensuring consistency between the policy pre-training and PPO policy optimization in terms of input and output.

[0029] During interactive training, the motion state variables are updated one step at a time by the Attention-BiGRU motion state simulation model, and the position state is kinematically advanced based on speed, heading, and sampling period. When the aircraft reaches the vicinity of the current target waypoint, it automatically switches to the next waypoint. During training, a reward function integrating trajectory tracking, flight safety, and control stability is constructed to guide the strategy to improve waypoint tracking capability, flight phase adaptability, and flight constraint maintenance capability, thereby forming an intelligent agent aircraft mission behavior simulation model capable of continuously executing flight missions.

[0030] III. Intelligent Agent Aircraft Integration The motion state simulation model and the task behavior simulation model of the intelligent agent aircraft are integrated to form a complete intelligent agent aircraft. The task behavior simulation model outputs control actions based on the current flight state and route task constraints, while the motion state simulation model updates the aircraft's attitude and velocity states based on the control actions and key input features, and the position state progresses synchronously according to kinematic relationships. Through the above collaborative modeling process, continuous flight process data containing position, attitude, velocity, and trajectory information is generated, enabling the intelligent agent aircraft to complete tasks such as waypoint tracking, flight phase transitions, and route constraint maintenance in the "cloud box" system simulation scenario.

[0031] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0032] I. A Method for Constructing Intelligent Agent Aircraft for Security Simulation Verification of Cloud Box System The intelligent agent aircraft construction method involved in this invention is as follows: Figure 1 As shown. Figure 1 In this process, the construction of an intelligent agent aircraft mainly includes two modeling branches and an integration process. One branch is the construction of a simulation model for the intelligent agent aircraft's motion state, and the other branch is the construction of a simulation model for the intelligent agent aircraft's mission behavior. After both branches are trained, they are integrated through state input, control action output, and state propagation processes to form a complete intelligent agent aircraft model.

[0033] In the motion state simulation branch, pitch angle, roll angle, heading angle, indicated airspeed, vertical speed, and ground speed are first selected as the aircraft motion state representations. Then, a random forest feature importance evaluation method is used to select key input features that are strongly related to the six motion state variables from the original flight parameters. Finally, an Attention-BiGRU motion state simulation model is constructed using the selected key input features to describe the temporal evolution of the aircraft motion state as a function of control input and flight process.

[0034] In the task behavior simulation branch, an expert dataset is first constructed based on historical flight records. This dataset is organized using a one-step state transition format: "current observation state, expert action, next moment state." Next, a task behavior action interface is designed, incorporating incremental adjustment and absolute target channels. Then, a behavior cloning method is used to pre-train the task policy, and dynamic consistency constraints are introduced to ensure that the pre-trained policy not only fits the expert actions but also maintains the rationality of the one-step state response. Finally, based on the pre-trained policy, the PPO reinforcement learning method is used to optimize the policy, resulting in a task behavior simulation model capable of waypoint tracking, flight phase transitions, and flight constraint maintenance.

[0035] During the model integration phase, the mission behavior simulation model outputs control actions based on the current flight state and route mission constraints; the motion state simulation model updates the aircraft's attitude and velocity states based on the control actions and key input features; and the position state is propagated kinematically based on velocity, heading, and sampling period. Through this process, the intelligent agent aircraft can continuously generate flight process data such as position, attitude, velocity, and trajectory, and integrate them into the "cloud box" system simulation scenario for display and verification.

[0036] II. Construction Process of Intelligent Agent Aircraft Motion State Simulation Model The motion state simulation model involved in this invention is used to describe the continuous evolution of aircraft attitude and velocity states. The model construction process includes three parts: motion state representation, key input feature selection, and Attention-BiGRU temporal modeling.

[0037] 1. Representation of motion state In this invention, six state variables—pitch angle, roll angle, heading angle, indicated airspeed, vertical speed, and ground speed—are selected as the representations of the aircraft's motion state. Specifically, pitch angle describes the aircraft's attitude change around its lateral axis, roll angle describes its attitude change around its longitudinal axis, and heading angle describes its heading change; indicated airspeed reflects the aircraft's relative airflow speed, vertical speed reflects its climb or descent, and ground speed reflects its propulsion speed along its flight path.

[0038] The six state variables mentioned above are used for joint characterization because the aircraft flight process is not only characterized by attitude changes, but is also closely related to velocity state, longitudinal motion, and trajectory propulsion. Using only attitude variables such as pitch angle, roll angle, and yaw angle is insufficient to fully describe the coupling relationship between attitude changes and velocity response. Therefore, this invention uses both attitude and velocity variables as motion state output variables, thus forming a state expression method more suitable for task behavior generation and flight process simulation.

[0039] 2. Key Input Feature Filtering After determining the six motion state output variables, this invention selects input features from the original flight parameters for motion state modeling. Directly using all the original flight parameters as model input can easily introduce redundant information and low-correlation noise, increasing model training complexity and affecting the model's learning of the main state change patterns.

[0040] In this invention, candidate parameters related to attitude changes, speed response, longitudinal control, and lateral control are first retained based on the aircraft's motion mechanism. Then, a random forest feature importance evaluation method is used to filter the candidate input parameters. For the regression tree in the random forest, let the actual value be y. i The predicted value is If the sample size is n, then the out-of-bag sampling error can be expressed as:

[0041] After perturbing the j-th candidate feature, the out-of-bag error is recalculated and denoted as OOBError. j,k , where k represents the k-th repeated perturbation. If the number of repeated evaluations is m, then the importance score of the j-th candidate feature can be expressed as:

[0042] Among them, Importance jThe larger the value, the greater the increase in model error after the j-th candidate feature is perturbed, and the more significant the contribution of this feature to the aircraft motion state simulation model. Based on the feature importance score ranking, parameters with high rankings or importance scores above a preset threshold are selected as key input features for the motion state simulation model. This key feature selection process reduces the interference of low-relevance and redundant parameters on model training, enabling the subsequent motion state simulation model to more effectively learn the main patterns in aircraft motion state changes.

[0043] 3. Motion state temporal modeling based on Attention-BiGRU After selecting key input features, the selected features are used to construct a time-window input sequence, and an Attention-BiGRU motion state simulation model is built. The network structure of this model is as follows: Figure 2 As shown.

[0044] Let X be the input feature vector at time t. t The hidden state is h t The GRU model controls the transmission of temporal information through reset gates, update gates, and candidate states. Its calculation process is as follows:

[0045] Where, r t To reset the door, z t To update the door, In the candidate hidden state, It is the Sigmoid activation function. This indicates element-wise multiplication.

[0046] BiGRU consists of a forward GRU and a backward GRU. Let the forward hidden state be... The backward hidden state is ,but:

[0047] The final hidden state is represented as:

[0048] Here, [.;.] represents vector concatenation. Through the bidirectional structure, the model can simultaneously extract historical information and subsequent trends from the flight state sequence, thereby better characterizing the temporal continuity of the aircraft's motion state.

[0049] Based on the BiGRU output, a self-attention mechanism is introduced to weight the hidden states at different time steps. Let the query matrix, key matrix, and value matrix be WQ, WK, and WV, respectively, then the attention output is:

[0050] Where dk is the dimension of the key vector. If a multi-head attention mechanism is used, then the i-th attention head is:

[0051] The multi-head attention output is:

[0052] Through the self-attention mechanism, the model can adaptively allocate weights based on the correlation between different time steps and the current state change, highlighting key time segments that contribute more to the evolution of the current motion state.

[0053] Finally, the attention-weighted temporal features are input into the fully connected layer, which outputs the six-dimensional motion state for the next time step:

[0054] Where fABG represents the Attention-BiGRU motion state simulation model, T represents the time window length, and s t+1 This represents the predicted six-dimensional motion state at the next moment. Thus, a simulation model of the intelligent agent's aircraft motion state is formed. This model can describe the temporal evolution of the aircraft's attitude and velocity states as a function of control inputs and flight processes, based on key flight parameters within a historical time window, and provides a state response basis for subsequent mission behavior simulation.

[0055] Thus, an intelligent agent aircraft motion state simulation model is formed. This model can describe the temporal evolution of the aircraft's attitude and velocity states as a function of control inputs and flight processes, based on key flight parameters within a historical time window, and provides a state response basis for subsequent mission behavior simulation.

[0056] III. Construction Process of Intelligent Agent Aircraft Mission Behavior Simulation Model The task behavior simulation model involved in this invention is used to generate the task behavior of an intelligent agent aircraft during waypoint tracking, flight phase transitions, and course constraint maintenance. The model construction process includes two parts: expert-guided policy pre-training and policy optimization based on PPO.

[0057] 1. Expert-guided strategy pre-training In this embodiment, the intelligent agent aircraft task behavior simulation mainly addresses task requirements such as waypoint tracking, flight phase transitions, and flight constraint maintenance. Since these tasks involve policy learning in continuous state and action spaces, directly using reinforcement learning for interactive training can easily lead to a large amount of ineffective exploration in the initial stage, resulting in significant fluctuations during the training process. Therefore, this invention introduces expert priors before reinforcement learning optimization. An expert dataset is constructed using historical flight records, and a behavior cloning method is used to pre-train the policy, enabling the task behavior simulation model to possess basic task execution capabilities before entering PPO policy optimization.

[0058] The expert samples are derived from historical flight records and organized using one-step state transitions as the basic unit. The expert dataset is defined as follows:

[0059] Where N is the number of expert samples, st is the observed state at time t, at is the control action taken by the expert in that state, and st+1 is the observed state at the next time step. By organizing the samples into "state, action, and next state," the relationship between behavioral choices and state evolution during task execution can be uniformly incorporated into the pre-training process. To balance aircraft motion state evolution information and spatial position information, this invention designs the observed state of the task behavior simulation model as a combination of attitude and velocity states, current position, target waypoint information, and control reference quantities, which can be expressed as:

[0060] Where, x dyn pos represents a subset of states related to the current attitude and velocity. t Indicates the current position, wp t Indicates the current target waypoint position, ref t This represents the current control reference quantity. By incorporating waypoint information and control reference quantities into the observation state, the policy network can simultaneously perceive the current flight state, target constraints, and behavioral adjustment references within the same input framework. Waypoint advancement is determined by the horizontal distance between the current position and the target waypoint. Let the current position be (lat). t ,lon t The target waypoint is (lat) wp ,lon wp (where lat represents latitude, lon represents longitude, and the Earth's radius is Re), then the great circle distance between two points can be expressed as:

[0061] When d is less than the preset waypoint capture radius, the system switches the current target waypoint to the next waypoint. Regarding the action interface design, considering that the task behavior output includes both continuous correction of the current manipulation quantity and direct setting of the target command, this invention divides the action vector into an incremental adjustment channel and an absolute target channel. The action vector is represented as:

[0062] in, This indicates the incremental adjustment channel, used for continuous correction based on the control reference at the previous moment; a abs,tThis represents the absolute target channel, used to directly set the target instruction for the current stage. To reduce the differences in scale and value range between different action dimensions, this invention scales each action dimension to a uniform numerical range during the training phase. For the i-th action dimension, if its physical boundary is [a min,i ,a max,i The scaling relationship is as follows:

[0063] The inverse scaling relationship is:

[0064] in, For normalized actions, a i For physical actions, during the training phase, the policy network outputs normalized actions, and the loss function is calculated in the normalized space. During the execution phase, the network output is first inversely scaled to the physical action, and then the control variable is updated according to the semantics of the action channel. For the incremental adjustment channel, the control variable is updated as follows:

[0065] For the absolute target channel, the control variable is updated as follows:

[0066] The above action interface design ensures consistency in the physical meaning of actions between the training and execution phases, reducing control bias caused by inconsistent action interpretations. After obtaining the expert dataset and action interface, the task behavior policy is pre-trained using a behavior cloning method. Let the parameterized policy network be... , in state s t The next output action is The expert's action is a t * The objective of minimizing the empirical risk of behavioral cloning can then be expressed as:

[0067] Among them, L H This is the action fitting loss function. In this embodiment, the action fitting term uses the Huber loss, which has the following form:

[0068] in, This represents the error between the predicted action and the expert's action. 0 represents the Huber loss threshold. The Huber loss maintains a quadratic penalty in the small error range and transforms into a linear penalty in the large error range, which can reduce the excessive influence of outliers on gradient updates. Since simply fitting expert actions is insufficient to constrain local closed-loop responses in continuous task execution, this invention further introduces dynamic consistency constraints. Let the dynamic surrogate model be f. dyn Given the current state s t and strategy output action at Under these conditions, the one-step prediction state is:

[0069] The dynamic consistency loss is defined as:

[0070] in, This indicates state standardization. Standardization uses the z-score form:

[0071] in, and These are the mean and standard deviation, calculated based on expert data. To avoid dividing by zero for extremely small positive numbers.

[0072] The overall objective function in the pre-training phase is a weighted sum of the action fitting loss and the dynamic consistency loss:

[0073] in, The weighting coefficients are used to adjust the relative effects of action fitting constraints and dynamic consistency constraints. The pre-trained strategy obtained through training with the above objective function can not only learn the basic task behavior patterns from expert samples, but also maintain good rationality at the one-step state response level, providing a stable initialization foundation for subsequent PPO strategy optimization.

[0074] PPO-based strategy optimization After completing expert-guided policy pre-training, this invention employs the Proximal Policy Optimization (PPO) algorithm to optimize the task behavior simulation model. The PPO policy optimization phase retains the observation space, action semantics, and scaling rules defined in the pre-training phase. That is, the observed inputs continue to be processed using the same standardized parameters, the policy network output is still generated in the normalized action space, and is inversely scaled to physical control variables using the same action boundary parameters. This approach ensures consistency in input / output between the policy pre-training phase and the PPO policy optimization phase.

[0075] In the interactive environment, the intelligent agent aircraft receives the current observation state s at each time step. t The policy network then outputs the action:

[0076] Subsequently, based on the progress of this action, the environment obtains the next moment's observation state, immediate reward, and round termination flag. For motion state variables such as pitch angle, roll angle, yaw angle, indicated airspeed, vertical velocity, and ground speed, the Attention-BiGRU motion state simulation model is used for one-step updates:

[0077] Among them, X t-T+1 ,...,X t f represents the sequence of key input features within the current time window. ABG This represents the Attention-BiGRU motion state simulation model. For the position state, kinematic propulsion is performed based on velocity, heading, and sampling period:

[0078] Where, pos t v represents the current position. t The velocity vector is determined by the velocity and heading. The sampling period is specified. When the distance between the aircraft and the current target waypoint is less than the preset capture radius, the system automatically switches to the next waypoint.

[0079] To simultaneously consider mission completion effectiveness, flight safety, and behavioral stability, this invention constructs a reward function under multi-objective constraints. The overall reward is expressed as:

[0080] Among them, R track Indicates a reward for tracking a flight path, C state C represents the cost of state constraints. action This represents the cost of action constraints.

[0081] Tracking rewards are used to encourage the agent aircraft to approach the target waypoint and maintain close alignment with the reference track. These rewards may include proximity rewards related to the distance to the target waypoint.

[0082] Distance reduction reward:

[0083] And trajectory matching rewards:

[0084] Where d is the distance from the current position to the target waypoint. e is the distance scale parameter. lat This is due to lateral trajectory deviation. This is the lateral deviation scale parameter.

[0085] State constraint costs are used to limit attitude, velocity, altitude, and their rates of change to remain within reasonable flight ranges, and can be expressed as:

[0086] Among them, C envelope C represents the cost of exceeding the envelope limit. rate This represents the cost of exceeding the rate of change of state. For weighting coefficients, the motion constraint cost is used to suppress problems such as excessively large control amplitude, excessive jitter in control quantities between adjacent time steps, and excessively frequent switching of motion channels. It can be expressed as:

[0087] Among them, C mag C represents the cost of incremental control amplitude. jitter C represents the cost of controlling jitter. sat C represents the cost of action saturation. switch This indicates the cost of shifting gears. These are the weighting coefficients.

[0088] In terms of policy optimization, PPO limits the update range between the old and new policies by pruning the objective function. Let the new policy be... The old strategy was Then the probability ratio is:

[0089] The objective function for PPO pruning is:

[0090] Among them, A t This represents the estimation of the advantage function. Here are the pruning parameters. The dominance function uses generalized dominance estimation and is expressed as:

[0091] in:

[0092] As a discount factor, Let V(s) be the smoothing parameter for the GAE, and V(s) be the value function. The overall loss function for training the policy network can be expressed as:

[0093] Among them, L VF For the value function loss, S[ ](s t Let c1 and c2 be the policy entropy regularization term, and c1 and c2 be the weight coefficients. The value function loss can be expressed as:

[0094] Among them, R t This indicates an estimated return.

[0095] During training, the Actor network loads the policy parameters obtained in the behavior clone pre-training phase, while the Critic network is used to estimate state values. Through PPO interactive training, the policy can further correct the biases in the pre-trained policy in the long-term reward context, improving the closed-loop execution capability of the task behavior simulation model in waypoint tracking, flight phase transitions, and route constraint maintenance tasks.

[0096] IV. Intelligent Agent Aircraft Integration After both the motion state simulation model and the task behavior simulation model have been trained, they are integrated to form a complete intelligent aircraft.

[0097] During integrated runtime, the mission behavior simulation model outputs control actions based on the current flight state, current position, target waypoint, and mission constraint information; the motion state simulation model updates the aircraft's attitude and velocity states based on the control actions and key input features; and the position state is synchronously advanced based on velocity, heading, and sampling period. By repeatedly executing the above process, a continuous sequence of flight states can be generated, which includes information such as longitude, latitude, altitude, pitch angle, roll angle, heading angle, indicated airspeed, vertical speed, and ground speed.

[0098] In this embodiment, the intelligent agent aircraft is validated using a Singapore-Shanghai reference route tracking task as an example. The reference route is used as the target route, and the simulated trajectory generated by the intelligent agent aircraft is compared with the reference route. If the intelligent agent aircraft can continuously advance along the reference route and remain within the channel constraints in the altitude, longitude, and latitude directions, it indicates that the constructed intelligent agent aircraft possesses good waypoint tracking and channel constraint maintenance capabilities.

[0099] During the verification process, the simulated trajectory generated by the intelligent agent aircraft can be mapped to a civil aviation system simulation scenario through front-end and back-end data connections. The front-end scenario continuously updates the aircraft model's position, attitude, and trajectory based on the longitude, latitude, altitude, and attitude angle information output from the back-end. This process allows for a visual demonstration of the intelligent agent aircraft's flight process throughout a complete flight route mission, and verifies its trajectory maintenance, altitude changes, and spatial position changes. The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A method for constructing an agent aircraft for security simulation verification of a cloud box system, characterized in that, Includes the following steps: S1 Intelligent Agent Aircraft Motion State Simulation Model Construction: Based on real civil aviation flight data recorded by the airborne QAR system, motion state training samples are constructed. Pitch angle, roll angle, heading angle, indicated airspeed, vertical speed, and ground speed are selected as motion state representations; key input features are selected using the random forest feature importance evaluation method; and an Attention-BiGRU motion state simulation model is constructed based on the selected key input features to describe the temporal evolution relationship of flight state. S2 intelligent agent aircraft mission behavior simulation model construction: Based on historical flight records, an expert dataset is constructed, the behavior cloning method is used to complete the pre-training of the mission policy, and dynamic consistency constraints are introduced to optimize the pre-training objective function; based on the pre-trained policy, the PPO reinforcement learning algorithm is used to optimize the policy and construct the mission behavior simulation model. S3 Intelligent Agent Aircraft Integration: Integrates motion state simulation models with task behavior simulation models to generate intelligent agent aircraft that can be used for waypoint tracking, flight phase transitions, and route constraint maintenance. It is then connected to the cloud box system simulation scenario to conduct safety simulation verification.

2. The intelligent agent aircraft construction method according to claim 1, characterized in that, The specific meanings of the motion state representations in step S1 are as follows: pitch angle, roll angle, and heading angle are used to describe changes in aircraft attitude, while airspeed, vertical speed, and ground speed are used to describe the aircraft speed state and trajectory propulsion process.

3. The intelligent agent aircraft construction method according to claim 1, characterized in that, The specific process of the random forest feature importance assessment method described in step S1 is as follows: Calculate the out-of-bag sample error: ,in This is the actual value. The value is the predicted value, and n is the number of samples. Recalculating out-of-bag error (OOBError) after perturbing the jth candidate feature j,k where k represents the kth repeated perturbation, repeated m times. Importance score of the j-th candidate feature: ; Based on the feature importance scores, parameters that rank highly or have scores above a preset threshold are selected as key input features.

4. The intelligent agent aircraft construction method according to claim 1, characterized in that, The construction process of the Attention-BiGRU motion state simulation model in step S1 is as follows: Using the time window sequence of key input features as input, bidirectional temporal information is extracted through forward GRU and backward GRU to obtain the hidden state. ; A self-attention mechanism is introduced to weight the hidden states at different time steps and output attention features; The attention features are input into the fully connected layer, which outputs the six-dimensional motion state at the next time step.

5. The intelligent agent aircraft construction method according to claim 1, characterized in that, The expert dataset mentioned in step S2 is organized in a one-step state transition form of "current observation state, expert action, and next moment state"; the observation state includes aircraft attitude and speed state, current position, target waypoint information, and control reference quantities.

6. The intelligent agent aircraft construction method according to claim 1, characterized in that, In step S2, the action vector of the task behavior simulation model is divided into an incremental adjustment channel and an absolute target channel; During the training phase, the actions in each dimension are scaled to a uniform numerical range, and during the execution phase, they are scaled back to physical actions.

7. The method for constructing an intelligent agent aircraft according to claim 1, characterized in that, The pre-training objective function for introducing dynamic consistency constraints in step S2 is: , where L action Huber loss is used for motion fitting loss; L dyn is the dynamic consistency loss, and is the standardized error between the predicted state and the actual state; These are the weighting coefficients.

8. The intelligent agent aircraft construction method according to claim 1, characterized in that, The reward function of the PPO reinforcement learning algorithm described in step S2 is composed of a weighted average of the trajectory tracking reward, the state constraint cost, and the action constraint cost: Rtrack includes proximity reward, distance reduction reward, and trajectory matching reward; Cstate includes envelope crossover cost and state change rate exceedance cost; Caction includes incremental control amplitude cost, control jitter cost, action saturation cost, and gear shifting cost. , These are the weighting coefficients.

9. The method for constructing an intelligent aircraft according to claim 1, characterized in that, The PPO reinforcement learning algorithm described in step S2 uses a pruning objective function: ,in Let be the probability ratio of the new and old strategies, and At be the generalized advantage estimate. These are the trimming parameters.

10. The method for constructing an intelligent aircraft according to claim 1, characterized in that, The operation process of the intelligent agent aircraft integration described in step S3 is as follows: The mission behavior simulation model outputs control actions based on the current flight status and route mission constraints; The motion state simulation model updates the aircraft's attitude and velocity states based on control actions and key input features; Position status is determined by kinematic propagation based on velocity, heading, and sampling period; The above process is repeated to generate continuous flight process data containing position, attitude, velocity, and trajectory information.

Images