Air battle intelligent decision-making method based on time sequence reinforcement learning

Abstract

The invention relates to the technical field of unmanned aerial vehicle pursuit intelligent decision-making, in particular to an air combat intelligent decision-making method based on time sequence reinforcement learning, and the method comprises the steps: constructing a high-fidelity unmanned aerial vehicle pursuit simulation environment, carrying out the confrontation simulation, and obtaining a multi-source situation observation vector of each simulation decision-making step, extracting time sequence dependence features of the pursuit process from the observation vector through a time sequence feature extraction module; inputting the time sequence characteristics into a pursuit autonomous decision-making model, and outputting a pursuit strategy at the current moment, including a tracking or attacking or avoiding instruction; and controlling the unmanned aerial vehicle to execute a corresponding pursuit action according to the strategy, updating the environment situation at the next moment, guiding model learning through a composite reward function, iteratively updating model network parameters in combination with trajectory data of each adversarial round, and finally realizing autonomous control of the pursuit action of the unmanned aerial vehicle by using the trained model. Therefore, the problem of poor generalization ability of pursuit decision in a complex dynamic environment and the like in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of intelligent decision-making technology for drone pursuit, and in particular to an intelligent decision-making method for air combat based on temporal reinforcement learning. Background Technology

[0002] With the development of unmanned autonomous technology, drone pursuit and escape games have shifted from close-range dogfights to beyond-line-of-sight confrontations. Among related technologies, drone pursuit and escape decision-making methods based on deep reinforcement learning often rely solely on observation information at the current moment, making it difficult to effectively capture the temporal dependencies in the pursuit and escape process. This leads to short-sightedness in decision-making and makes it difficult to adapt to complex adversarial scenarios spanning long periods of time. Summary of the Invention

[0003] This application provides a method, device, equipment, and medium for intelligent air combat decision-making based on temporal reinforcement learning, in order to solve the problem of poor generalization ability of pursuit and escape decision-making in complex dynamic environments in related technologies.

[0004] The first aspect of this application provides an intelligent air combat decision-making method based on temporal reinforcement learning, comprising the following steps: performing adversarial simulation on a UAV model in a UAV pursuit and escape simulation environment, obtaining the observation vector of the UAV model at each simulation decision step, and extracting temporal features from the observation vector; inputting the temporal features into a pursuit and escape autonomous decision-making model, and outputting the pursuit and escape strategy at the current moment through the pursuit and escape autonomous decision-making model, the pursuit and escape strategy including at least one of tracking instructions, attack instructions, and evasion instructions; controlling the UAV model to execute pursuit and escape actions using the pursuit and escape strategy at the current moment, updating the environmental situation of the UAV pursuit and escape simulation environment at the next moment, guiding the autonomous decision-making model to learn the pursuit and escape strategy using a reward function, iteratively updating the network parameters of the autonomous decision-making model based on the data of each adversarial round, and controlling the UAV to execute pursuit and escape actions using the trained pursuit and escape autonomous decision-making model.

[0005] Based on the aforementioned technical means, this application embodiment extracts the temporal features of the observation vectors in the simulation environment and inputs them into the autonomous decision-making model for pursuit and escape. This effectively captures the dynamic dependencies in the pursuit and escape process, avoids the short-sightedness of decision-making caused by relying solely on information at the current moment, and thus improves the rationality and adaptability of decision-making under long-term confrontation. By outputting the intentions of strategies such as tracking, attacking, and evading, and combining them with reward functions to guide learning, the UAV can autonomously optimize its decision-making behavior in complex dynamic environments, enabling it to have stronger autonomous decision-making capabilities and confrontational advantages in beyond-visual-range pursuit and escape games.

[0006] Optionally, adversarial simulation is performed on the UAV model in a UAV pursuit simulation environment, including: constructing the UAV's flight dynamics model and load kinematics model; constructing a UAV pursuit simulation environment based on the flight dynamics model and load kinematics model, the UAV pursuit simulation environment including the UAV model and a target intelligent agent based on fixed rule strategies, both the UAV model and the target intelligent agent acquire each other's position, speed and attitude information in real time.

[0007] Based on the above technical means, the embodiments of this application construct a high-fidelity aircraft dynamics model and load kinematics model, and build a complete information pursuit and escape simulation environment containing a target intelligent agent with fixed rules. This can realistically reproduce the dynamic confrontation process between the UAV and the target, providing a reliable data foundation and simulation support for subsequent temporal feature extraction and strategy learning. At the same time, it ensures the consistency between the training environment and the real confrontation scenario, and improves the generalization ability and physical credibility of the decision model.

[0008] Optionally, the aircraft dynamics model includes translational equations and rotational equations, wherein the translational equations are:

[0009]

[0010] in, For aircraft quantity, and The three-axis velocity components of the body shaft system. For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the roll angular velocity, The pitch angular velocity, Yaw angular velocity, , and This represents the total external force components under the machine body's axis system; The equation of rotation is:

[0011] in, , and Principal moment of inertia, For inertial product, , and This represents the total external torque component under the machine body axis. For roll acceleration, For pitch acceleration, Yaw acceleration; The load kinematic equations of the load kinematic model in the inertial coordinate system are described as follows:

[0012] in, , and The three-axis velocity components of the load in the inertial coordinate system. For load speed, The pitch angle of the flight path. The yaw angle of the track; The load kinematic equations of the load kinematic model are described in the ballistic coordinate system as follows:

[0013] in, It is the acceleration due to gravity. For tangential overload, and To control overload laterally, Let be the tangential acceleration of the load. Yaw angular velocity, It represents the pitch angular velocity.

[0014] Optionally, extracting temporal features from the observation vector includes: inputting the observation vector into a policy network, outputting temporal features through the policy network, wherein the policy network includes a feature extraction network and a gated recurrent unit, extracting features from the observation vector through the feature extraction network, inputting the features from the observation vector into the gated recurrent unit, and extracting temporal features containing historical trajectory information during the pursuit process through the gated recurrent unit.

[0015] Based on the above technical means, the embodiments of this application introduce a feature extraction network and a gated recurrent unit into the policy network. This enables the full mining of historical trajectory information and temporal dependencies in the pursuit process, based on effective feature encoding of the observation vector. This provides a more comprehensive situational awareness basis for the decision-making model, effectively improving the short-sightedness of decision-making caused by relying solely on single-step observation in related technical methods, and enhancing the rationality and foresight of decision-making in long-term confrontation scenarios.

[0016] Optionally, the pursuit and escape strategy at the current moment is output through the autonomous pursuit and escape decision-making model, including: defining the output space of the policy network as a discretized policy intention through a hierarchical decision-making mechanism; constructing the autonomous pursuit and escape decision-making model based on the discretized policy intention; sampling the temporal features using the autonomous pursuit and escape decision-making model based on probability distribution; and determining the pursuit and escape strategy at the current moment based on the sampling results; the probability distribution sampling mechanism of the autonomous pursuit and escape decision-making model is as follows:

[0017] in, For time series feature vectors, and These are the weight matrix and bias vector of the top-level output layer of the policy network, respectively. This is the action logic value. and The first The and the first The exponent term of each action logic value is used to calculate the action probability. This represents the action probability distribution output by the policy network. The policy intent commands are sampled from the current policy distribution of the UAV model. It is a set of discrete action spaces.

[0018] Optionally, the reward function can be expressed as:

[0019] in, The total reward value, Rewards for the final outcome of the game. Rewards are provided to guide users from a forward-looking perspective. As a reward for crash punishment, As a reward for attack launch, Payload threat penalty and reward; Endgame and security constraints rewards:

[0020]

[0021] in, The crash penalty coefficient is h, where h is the flight altitude. For the crash height threshold, For safety alarm height, The descent speed is the velocity of the ground. The descent speed threshold, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. Positional advantage and attack bonus:

[0022]

[0023] in, Forward perspective, As a reward for attack launch, Penalty for the number of launches, Rewards for advantageous attack positions; The payload threat is:

[0024] in, For the first The relative approach speed of each load, As the velocity normalization factor, This is a truncation function.

[0025] Optionally, the network parameters of the autonomous decision-making model are iteratively updated based on the data from each adversarial round, including: identifying the algorithmic framework of the autonomous decision-making model, which includes a policy network and a value network; calculating the temporal difference error based on the evaluation value of the value network and recursively calculating the advantage function; calculating the policy loss of the policy network and the value loss of the value network based on the data from each adversarial round, and calculating the total loss function based on the policy loss, value loss, and entropy regularization term; and iteratively updating the network parameters of the policy network based on the advantage function and the total loss function.

[0026] Based on the above technical means, the embodiments of this application adopt an algorithm framework that includes a policy network and a value network, combines temporal difference error and advantage function calculation, and constructs a total loss function based on policy loss, value loss and entropy regularization term. This can improve the efficiency of policy exploration while ensuring training stability, and achieve efficient iterative updates of network parameters, so that the autonomous decision-making model can converge quickly and obtain better decision performance in complex chase-and-escape game scenarios.

[0027] A second aspect of this application provides an intelligent air combat decision-making device based on temporal reinforcement learning, comprising: an acquisition module for performing adversarial simulation on a UAV model in a UAV pursuit and escape simulation environment, acquiring the observation vector of the UAV model at each simulation decision step, and extracting temporal features from the observation vector; an output module for inputting the temporal features into a pursuit and escape autonomous decision-making model, outputting the pursuit and escape strategy at the current moment through the pursuit and escape autonomous decision-making model, the pursuit and escape strategy including at least one of a tracking command, an attack command, and an evasion command; and a control module for controlling the UAV model to execute pursuit and escape actions using the pursuit and escape strategy at the current moment, updating the environmental situation of the UAV pursuit and escape simulation environment at the next moment, guiding the autonomous decision-making model to learn the pursuit and escape strategy using a reward function, iteratively updating the network parameters of the autonomous decision-making model based on the data of each adversarial round, and controlling the UAV to execute pursuit and escape actions using the trained pursuit and escape autonomous decision-making model.

[0028] Optionally, the acquisition module is further used to: construct the UAV's aircraft dynamics model and payload kinematics model; construct a UAV pursuit simulation environment based on the aircraft dynamics model and payload kinematics model, the UAV pursuit simulation environment including the UAV model and the target intelligent agent based on fixed rule strategies, both the UAV model and the target intelligent agent acquire each other's position, speed and attitude information in real time.

[0029] Optionally, the aircraft dynamics model includes translational equations and rotational equations, wherein the translational equations are:

[0030] in, For aircraft quantity, and The three-axis velocity components of the body shaft system. For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the roll angular velocity, The pitch angular velocity, Yaw angular velocity, , and This represents the total external force components under the machine body's axis system; The equation of rotation is:

[0031] in, , and Principal moment of inertia, For inertial product, , and This represents the total external torque component under the machine body axis. For roll acceleration, For pitch acceleration, Yaw acceleration; The load kinematic equations of the load kinematic model in the inertial coordinate system are described as follows:

[0032] in, , and The three-axis velocity components of the load in the inertial coordinate system. For load speed, The pitch angle of the flight path. The yaw angle of the track; The load kinematic equations of the load kinematic model are described in the ballistic coordinate system as follows:

[0033] in, It is the acceleration due to gravity. For tangential overload, and To control overload laterally, Let be the tangential acceleration of the load. Yaw angular velocity, It represents the pitch angular velocity.

[0034] Optionally, the acquisition module is further configured to: input the observation vector into the policy network, and output temporal features through the policy network, wherein the policy network includes a feature extraction network and a gated recurrent unit, the feature extraction network extracts features of the observation vector, the features of the observation vector are input into the gated recurrent unit, and the gated recurrent unit extracts temporal features containing historical trajectory information during the pursuit process.

[0035] Optionally, the output module is further used to: define the output space of the policy network as a discretized policy intent through a hierarchical decision-making mechanism; construct an autonomous pursuit decision-making model based on the discretized policy intent; sample the temporal features using the autonomous pursuit decision-making model based on probability distribution; and determine the pursuit strategy at the current moment based on the sampling results; the probability distribution sampling mechanism of the autonomous pursuit decision-making model is as follows:

[0036] in, For time series feature vectors, and These are the weight matrix and bias vector of the top-level output layer of the policy network, respectively. This is the action logic value. and The first The and the first The exponent term of each action logic value is used to calculate the action probability. This represents the action probability distribution output by the policy network. The policy intent commands are sampled from the current policy distribution of the UAV model. It is a set of discrete action spaces.

[0037] Optionally, the reward function can be expressed as:

[0038] in, The total reward value, Rewards for the final outcome of the game. Rewards are provided to guide users from a forward-looking perspective. As a reward for crash punishment, As a reward for attack launch, Payload threat penalty and reward; Endgame and security constraints rewards:

[0039]

[0040] in, The crash penalty coefficient is h, where h is the flight altitude. For the crash height threshold, For safety alarm height, The descent speed is the velocity of the ground. The descent speed threshold, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. Positional advantage and attack bonus:

[0041]

[0042] in, Forward perspective, As a reward for attack launch, Penalty for the number of launches, Rewards for advantageous attack positions; The payload threat is:

[0043] in, For the first The relative approach speed of each load, As the velocity normalization factor, This is a truncation function.

[0044] Optionally, the control module is further configured to: identify the algorithmic framework of the autonomous decision-making model, the algorithmic framework including a policy network and a value network; calculate the temporal difference error based on the evaluation value of the value network, and recursively calculate the advantage function; calculate the policy loss of the policy network and the value loss of the value network based on the data of each adversarial round, and calculate the total loss function based on the policy loss, value loss and entropy regularization term; and iteratively update the network parameters of the policy network based on the advantage function and the total loss function.

[0045] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the program to implement the air combat intelligent decision-making method based on temporal reinforcement learning as described in the above embodiments.

[0046] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the air combat intelligent decision-making method based on temporal reinforcement learning as described in the above embodiments.

[0047] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0048] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an air combat intelligent decision-making method based on temporal reinforcement learning provided according to an embodiment of this application; Figure 2 This is a flowchart of an air combat intelligent decision-making method based on temporal reinforcement learning according to an embodiment of this application; Figure 3 This is a schematic diagram of the strategy network architecture and data flow that integrates temporal memory and hierarchical control according to the embodiments of this application; Figure 4 This is a schematic diagram of the average round reward convergence curve during the training process of the agent model provided according to the embodiments of this application; Figure 5 This is a schematic diagram illustrating the evolution trend of the win rate and loss rate of an intelligent agent against a fixed expert strategy according to an embodiment of this application; Figure 6 This is a block diagram of an air combat intelligent decision-making device based on temporal reinforcement learning, according to an embodiment of this application. Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0050] Currently, with the development of unmanned autonomous systems, multi-source heterogeneous sensing, and long-range high-dynamic interaction technologies, the game mode of UAVs has gradually shifted from close-range dogfights within line of sight to beyond-line-of-sight pursuit and escape games under incomplete observation. In such long-distance, highly dynamic game tasks, the environmental state space has extremely strong uncertainty and nonlinear characteristics. Intelligent decision-making systems not only need to process massive amounts of sensor observation streams and inter-machine communication data in real time, but also need to complete OODA (Observation-Orientation-Decision-Action) closed-loop control from environmental perception and situation assessment to strategy planning within millisecond time scales. Related technologies often use DRL (Deep Reinforcement Learning) to achieve intelligent decision-making in UAV pursuit and escape games.

[0051] The related technologies have the following shortcomings when training agents and making decisions in 6-DOF (6 Degrees of Freedom) drone pursuit environments: First, most reinforcement learning methods in related technologies are based on the MDP (Markov Decision Process) assumption, which assumes that the current observation contains all the information needed for decision-making. However, the real drone pursuit situation has significant "partial observability" and "non-Markov" characteristics. Using a simple feedforward neural network that only relies on the current frame input, the agent cannot capture the strategic intent in the historical trajectory, resulting in short-sighted decision-making and difficulty in adapting to long-term games. Secondly, most of the reinforcement learning research on pursuit and escape in related technologies is based on simplified point mass model 3-DOF (3 Degrees of Freedom) training, ignoring the rotational inertia, aerodynamic coupling and attitude response delay of the aircraft. After introducing a six-degree-of-freedom nonlinear dynamic model, the state space and action space expand rapidly. When the reinforcement learning algorithm directly outputs the deflection of the bottom continuous aerodynamic control surface, the model is extremely difficult to converge because the mapping from "strategic intention" to "physical execution" is complex and nonlinear. The generated actions have high-frequency oscillations and do not have engineering flightability. Third, when training against opponents with fixed rules based on expert knowledge, the pursuit-escape game involves long distances and durations, with sparse win / loss signals. In the early stages of training, the agent struggles to obtain positive feedback, resulting in low exploration efficiency. Furthermore, fixed expert strategies are highly effective under their preferred conditions. Without effective strategy guidance and hierarchical constraints, the agent struggles to spontaneously evolve methods to counter these strategies.

[0052] In summary, when using reinforcement learning for drone pursuit game decision training, the relevant technologies still face problems such as strong observability of the state space, difficulty in searching the non-Markovian and continuous low-level control space, and difficulty in convergence due to sparse rewards for expert strategies. They cannot adapt to the complex adversarial requirements of six-degree-of-freedom high-fidelity drone pursuit.

[0053] The following description, with reference to the accompanying drawings, outlines an air combat intelligent decision-making method, apparatus, device, and medium based on temporal reinforcement learning, according to embodiments of this application. It addresses the problems mentioned in the background art, such as poor generalization ability of pursuit and escape decisions in complex dynamic environments, insufficient capture of temporal dependencies in six-degree-of-freedom high-fidelity nonlinear mechanical environments, difficulty in model convergence due to complex continuous low-level control mapping, and difficulty in convergence when facing fixed expert strategies. This application presents an intelligent air combat decision-making method based on temporal reinforcement learning, applicable to autonomous strategy generation and deciphering of fixed expert strategies in a high-fidelity nonlinear mechanics environment. It utilizes a Gated Recurrent Unit (GRU) to address temporal dependencies in game theory. Here, the long-short dependency relationship refers to the fact that in UAV pursuit games, the current decision depends not only on the current situation but also on historical trajectories such as preceding maneuvers, target motion trends, and payload launch timing, resulting in a decision-making process that does not satisfy Markov property. Furthermore, it employs a hierarchical architecture of "discrete intention-continuous control" to decouple the complex six-degree-of-freedom control and reduce the decision-making dimensionality. Specifically, it constructs a full-element adversarial simulation environment integrating six-degree-of-freedom aircraft dynamics and three-degree-of-freedom payload kinematics. Under complete information situational conditions, it establishes a high-dimensional observation space containing all friendly flight states, target relative motion parameters, and target payload information. A gated neural network (GRU) is used to process the observation data, introducing hidden states... Historical information is summarized to extract temporal features from the pursuit-escape game process. These features are obtained by fusing and encoding multi-step observation vectors, aggregating historical trajectory information, and characterizing the dynamic dependencies and trend intentions during the pursuit-escape process. Since the sequential decision-making process of UAV game adversarial does not satisfy the Markov property, the current decision is not only related to the current state but also influenced by historical states. Therefore, GRU is used to represent this temporal characteristic. A policy network is constructed using the PPO (Proximal Policy Optimization) algorithm. Discrete maneuver commands such as tracking, attacking, and evading are output through a hierarchical decision-making mechanism. These commands are then calculated into aerodynamic control surface variables by the underlying flight control system. A composite reward function is designed, incorporating forward-view guidance, payload range constraints, and survival assessment, to maximize payload efficiency while ensuring survival. Finally, policy convergence is achieved through iterative training with an expert rule system, resulting in a high win rate suppression of the agent against fixed expert policies.

[0054] Specifically, Figure 1 This is a flowchart illustrating an intelligent air combat decision-making method based on temporal reinforcement learning, provided as an embodiment of this application.

[0055] like Figure 1 As shown, this air combat intelligent decision-making method based on temporal reinforcement learning includes the following steps: In step S101, the UAV model is subjected to adversarial simulation in the UAV pursuit simulation environment to obtain the observation vector of the UAV model at each simulation decision step, and the temporal features are extracted from the observation vector.

[0056] It is understood that the embodiments of this application, by conducting adversarial simulation in a drone pursuit simulation environment and obtaining the observation vectors of each decision step, and then extracting temporal features from the observation vectors, can make full use of historical trajectory information to characterize the dynamic dependencies in the pursuit process, provide a more comprehensive decision basis for subsequent strategy generation, effectively avoid the short-sighted decision problem caused by relying solely on single-step observation, and improve the rationality and adaptability of the model's decision-making in complex and highly dynamic pursuit scenarios.

[0057] Specifically, this embodiment first constructs a high-fidelity pursuit simulation environment integrating a six-degree-of-freedom aircraft dynamics model and a three-degree-of-freedom payload kinematics model. This environment is built based on JSBSim (an open-source flight dynamics simulation engine). The agent to be trained is a drone model, and the target agent is a fixed-rule agent that operates according to an expert knowledge base and carries a standard three-degree-of-freedom payload. Both parties acquire each other's position, velocity, and attitude information in real time. After environment initialization, adversarial simulation is carried out. At each simulation decision step, a full observation vector containing the drone model's flight state, the target agent's relative motion parameters, and the target agent's payload information is read and assembled in real time through the environment interface. Then, the observation vector is input into a policy network containing a feature extraction network and a gated recurrent unit. The observation vector is first mapped to a high-dimensional hidden space through the fully connected layer of the feature extraction network, and then processed by ReLU (Rectified Linear Array). The activation function of the linear rectifier unit (GRU) introduces nonlinear features, and then layer normalization eliminates the numerical distribution differences caused by different physical dimensions, generating a normalized instantaneous feature vector. Finally, the instantaneous feature vector is input into the GRU, and the update gate and reset gate inside the GRU control the memory flow. The hidden state of the previous moment is fused with the current instantaneous feature vector to calculate the current hidden state containing historical trajectory information, that is, the temporal features containing historical trajectory information during the pursuit process, thus realizing the extraction of temporal features.

[0058] In this embodiment of the application, adversarial simulation of a drone model is performed in a drone pursuit simulation environment, including: constructing a drone flight dynamics model and a payload kinematics model; constructing a drone pursuit simulation environment based on the flight dynamics model and the payload kinematics model, the drone pursuit simulation environment including a drone model and a target intelligent agent based on a fixed rule strategy, both the drone model and the target intelligent agent acquire each other's position, speed and attitude information in real time.

[0059] It is understood that the embodiments of this application, by constructing a high-fidelity aircraft dynamics model and load kinematics model, and building a complete information pursuit and escape simulation environment containing a target intelligent agent with fixed rules, can realistically reproduce the dynamic confrontation process between the UAV and the target, providing a reliable data foundation and simulation support for subsequent temporal feature extraction and strategy learning, while ensuring the consistency between the training environment and the real confrontation scenario, and improving the generalization ability and physical credibility of the decision model.

[0060] Specifically, the aircraft dynamics model adopts a nonlinear six-degree-of-freedom rigid body dynamics equation set to fully describe the nonlinear response characteristics of the aircraft under the combined action of aerodynamic forces, thrust, and gravity. The load kinematics model is simplified to a variable-mass point model driven by the guidance law, and its dynamic characteristics are decomposed into two parts: kinematic updates in the inertial frame and dynamic forces in the trajectory coordinate system. When constructing the simulation environment, the above-mentioned aircraft dynamics model and load kinematics model are integrated, and a full-element adversarial simulation platform is built based on the JSBSim flight simulation engine. The agent to be trained is set as a UAV model, and the target agent is configured with a fixed rule strategy based on expert knowledge, equipped with a standard three-degree-of-freedom load and operating according to the expert knowledge base. At the same time, an environmental interaction interface is configured so that the UAV model and the target agent can read each other's position, velocity, attitude and other state information in real time through the interface, ensuring that both parties are in a state of complete information, thereby completing the construction of the UAV pursuit simulation environment and providing an interactive space that conforms to the laws of physics for subsequent adversarial simulations.

[0061] In this embodiment of the application, the aircraft dynamics model includes translational equations and rotational equations, wherein the translational equations are:

[0062] in, For aircraft quantity, and The three-axis velocity components of the body shaft system. For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the roll angular velocity, The pitch angular velocity, Yaw angular velocity, , and This represents the total external force components under the machine body's axis system; The equation of rotation is:

[0063] in, , and Principal moment of inertia, For inertial product, , and This represents the total external torque component under the machine body axis. For roll acceleration, For pitch acceleration, Yaw acceleration; The load kinematic equations of the load kinematic model in the inertial coordinate system are described as follows:

[0064] in, , and The three-axis velocity components of the load in the inertial coordinate system. For load speed, The pitch angle of the flight path. The yaw angle of the track; The load kinematic equations of the load kinematic model are described in the ballistic coordinate system as follows:

[0065] in, It is the acceleration due to gravity. For tangential overload, and To control overload laterally, Let be the tangential acceleration of the load. Yaw angular velocity, It represents the pitch angular velocity.

[0066] Specifically, the aircraft dynamics model is built based on the JSBSim flight simulation engine. The translational equations described above are Newton's equations describing the translation of the aircraft's center of mass, while the rotational equations are Euler's equations describing the aircraft's rotation about its center of mass, taking into account the effect of the inertial product. The combination of the two fully characterizes the six-degree-of-freedom motion characteristics of the aircraft, with each parameter closely matching the physical characteristics of a real aircraft. The components of the aircraft's rotational inertia tensor accurately reflect the coupling characteristics of the aircraft's rotational inertia; in the load kinematic model, tangential overload Depends on thrust With aerodynamic drag The combined force and aerodynamic drag are calculated using a dynamic cross-sectional area formula to simulate the drag characteristics during high angle-of-attack maneuvers. The guidance logic employs PN (Proportional Navigation) and calculates lateral command overload using the line-of-sight rotation rate. and In addition, load-related parameters also include: and These are the diameter and length of the load, respectively. , The angle difference between the load attitude and the velocity vector represents the value of the angle between the load attitude and the velocity vector. The drag coefficient can be taken as a constant of 0.1 in this embodiment. The atmospheric density decreases exponentially with altitude. These parameters together ensure the high fidelity of the load kinematics model and work in conjunction with the aircraft dynamics model to support the realism of the simulation environment.

[0067] In this embodiment of the application, extracting temporal features from the observation vector includes: inputting the observation vector into a policy network, outputting temporal features through the policy network, wherein the policy network includes a feature extraction network and a gated recurrent unit, extracting features from the observation vector through the feature extraction network, inputting the features from the observation vector into the gated recurrent unit, and extracting temporal features containing historical trajectory information during the pursuit process through the gated recurrent unit.

[0068] It is understood that by introducing a feature extraction network and a gated recurrent unit into the policy network, the embodiments of this application can fully explore the historical trajectory information and temporal dependencies in the pursuit process based on the effective feature encoding of the observation vector, thereby providing a more comprehensive situational awareness basis for the decision-making model, effectively improving the short-sightedness of decision-making caused by relying solely on single-step observation in related technical methods, and enhancing the rationality and foresight of decision-making in long-term confrontation scenarios.

[0069] Specifically, we first define the vector structure of the high-dimensional hybrid observation space, and the observation vector... There are a total of 77 dimensions, of which

[0070] For observations of the drone model itself, there are 9 dimensions in total, including flight altitude. Aircraft attitude angle roll , up and down The sine and cosine values, and the three-axis velocity components under the mechanical system. Real airspeed ; The observations of the target intelligent agent are in 6 dimensions, including relative approach velocity. Relative height difference Front view Target azimuth Relative distance and left and right relative positions (Left side is -1, right side is 1); UAV self-payload observation is

[0071]

[0072] A total of 62 dimensions, of which This represents the number of our effective payloads. For the target number of payloads, each payload has 5 states, including the first... Approach speed of each payload relative to the UAV model Front view Azimuth and relative distance and left and right relative positions The maximum number of payloads for both sides is 6. If the number is less than 6, zero-filling will be used to make up the difference.

[0073] Subsequently, a two-level processing network of "spatial feature encoding + temporal memory fusion" is constructed as the policy network. The spatial feature encoding part is the feature extraction network, which first processes the original observation vector. The input is a fully connected layer, which maps the physical space to a high-dimensional hidden layer space (the fully connected layer has 128 hidden neurons). Then, a ReLU activation function is used to introduce non-linear features. Finally, layer normalization is applied to eliminate numerical distribution differences caused by different physical units, generating a normalized instantaneous feature vector. The expression for the ReLU activation function is:

[0074] The specific method of layer normalization is to perform normalization and affine transformation for each feature. The specific calculation method is as follows:

[0075] in The mean of the input data. To determine the variance, the input data is first normalized to a distribution with a mean of 0 and a standard deviation of 1, and then a learnable parameter is used. and Affine transformation is performed to restore the expressive power of the input data. The temporal memory fusion part, namely the gated recurrent unit, takes the instantaneous feature vector as input. The data is passed to the GRU, which uses its internal update gate. and reset door Control the memory stream, including the hidden state from the previous time step. By fusing with the current input, the hidden state at the current time step is calculated. This state By aggregating historical trajectory information, it can effectively characterize the trend intention of the target intelligent agent's maneuver, that is, the temporal features containing historical trajectory information during the pursuit process, realize the accurate extraction of temporal features, solve the problem of "partial observability" under single-frame observation, and the GRU hidden layer dimension is 128.

[0076] In step S102, the temporal features are input into the autonomous pursuit and escape decision-making model, and the pursuit and escape strategy at the current moment is output through the autonomous pursuit and escape decision-making model. The pursuit and escape strategy includes at least one of the following: tracking instruction, attack instruction, and evasion instruction.

[0077] It is understood that the embodiments of this application, by inputting temporal features into the autonomous decision-making model for pursuit and escape and outputting pursuit and escape strategies including tracking, attacking, and evading, can make more forward-looking decisions based on historical trajectories and temporal dependencies. At the same time, by discretizing strategy instructions, the decision space is simplified, and the model's response speed and decision stability in complex adversarial scenarios are improved.

[0078] Specifically, the autonomous decision-making model for pursuit and escape adopts an Actor-Critic architecture. The Actor network acts as the upper-level decision-making hub, implementing a hierarchical decision-making strategy. This decouples the complex pursuit and escape process from upper-level strategy decision-making to lower-level flight control. The task of the upper-level strategy network is to process the temporal feature vector extracted in step S101. Mapped to a discretized strategy intent space; this strategy intent space is a set of mutually exclusive high-level strategy intents. These correspond to tracking, attack, and evasion commands, respectively. The decision logic for the tracking command is to adjust the UAV model's flight attitude to minimize the forward view, creating conditions for subsequent attacks. The decision logic for the attack command is to control the UAV model to release the payload when the payload constraint is met. The decision logic for the evasion command is to control the UAV model to abandon its current advantageous position and perform evasive maneuvers when the target agent's payload is close, increasing the distance from the target agent's payload to maximize the probability of survival. The policy network uses a fully connected linear layer as a decoder to process the temporal feature vectors. Mapped to action logic value Then, the Softmax activation function is applied to obtain the action probability distribution. Finally, the pursuit / escape strategy at the current moment is determined through a probability distribution sampling mechanism, and the strategy is output. The structure of the Critic network is basically the same as that of the Actor network, the only difference being the value function of the current state at the end of the output.

[0079] In this embodiment, the pursuit and escape strategy at the current moment is output through the autonomous pursuit and escape decision-making model, including: defining the output space of the policy network as a discretized policy intention through a hierarchical decision-making mechanism; constructing the autonomous pursuit and escape decision-making model based on the discretized policy intention; sampling the temporal features using the autonomous pursuit and escape decision-making model based on probability distribution; and determining the pursuit and escape strategy at the current moment based on the sampling results. The probability distribution sampling mechanism of the autonomous pursuit and escape decision-making model is as follows:

[0080] in, For time series feature vectors, and These are the weight matrix and bias vector of the top-level output layer of the policy network, respectively. This is the action logic value. and The first The and the first The exponent term of each action logic value is used to calculate the action probability. This represents the action probability distribution output by the policy network. The policy intent commands are sampled from the current policy distribution of the UAV model. It is a set of discrete action spaces.

[0081] Specifically, the core of the hierarchical decision-making mechanism is to discretize and semantically abstract the action space, transforming the continuous control surface problem into a high-level strategy intent selection problem. This greatly compresses the exploration space of reinforcement learning and avoids training oscillations and non-convergence caused by directly outputting continuous actions; the discretized set of strategy intents... That is, the set of discrete action spaces Number of elements These correspond to three strategic intent commands: tracking, attacking, and evading. The autonomous decision-making model for pursuit and escape is constructed based on this discrete set of strategic intents, with the temporal feature vector extracted in step S101 as the input. First, through the weight matrix of the output layer With bias vector The unnormalized action logic value is calculated. Then, the Softmax activation function was applied to... After normalization, the action probability distribution of each strategy intent instruction is obtained. This ensures that the sum of the probabilities of all actions is 1; during the sampling process, to maintain exploratory behavior during the training phase, the agent follows this probability distribution. Perform random sampling, i.e. In the testing or practical phase, the action with the highest probability is directly selected, i.e. Sampling This refers to the current pursuit and escape strategy instructions, which are then used to output the pursuit and escape strategy.

[0082] In this embodiment of the application, the expression for the reward function is:

[0083] in, The total reward value, Rewards for the final outcome of the game. Rewards are provided to guide users from a forward-looking perspective. As a reward for crash punishment, As a reward for attack launch, Payload threat penalty and reward; Endgame and security constraints rewards:

[0084]

[0085] in, The crash penalty coefficient is h, where h is the flight altitude. For the crash height threshold, For safety alarm height, The descent speed is the velocity of the ground. The descent speed threshold, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. Positional advantage and attack bonus:

[0086]

[0087] in, Forward perspective, As a reward for attack launch, Penalty for the number of launches, Rewards for advantageous attack positions; The payload threat is:

[0088] in, For the first The relative approach speed of each load, As the velocity normalization factor, This is a truncation function.

[0089] Specifically, the reward function is a composite reward function combining sparse and dense features, calculated at each simulation step, aiming to balance short-term advantages with long-term strategic goals. The specific implementation details of each sub-reward are as follows: 1. Endgame and security constraints rewards: The highest priority sparse reward is used to determine the outcome of a round. When the drone model completes the task, it is given a positive reward of 200. When the target agent completes the task, it is given a penalty of -200. The reward value is 0 at other times. The crash penalty reward is used to ensure the flight safety of the drone model. In this embodiment, the crash penalty coefficient is... Crash height threshold Safety alarm height descent speed threshold That is, when the flight altitude ,or exist Between and the downward speed At the same time, continuous penalties are imposed to constrain the drone model and prevent it from crashing; 2. Positional advantage and attack rewards: Mid-front perspective The reward value is negatively correlated with the forward viewpoint, which encourages the drone model to continuously adjust its nose direction so that the radar locks onto the target agent. Penalty based on the number of launches and attack position advantage rewards Composition, in which This is a piecewise function used to encourage drone models to hit the target on their first shot and suppress overfire, specifically:

[0090] This represents the cumulative number of payloads launched by the drone model in this round. The gain logic employs a distance-angle coupling, specifically expressed as follows:

[0091] Where 110 is the reference value (km) for the maximum effective range of the load. The angle representing the boundary of the load's effective range is only applicable when the relative distance between the UAV model and the target intelligent agent is... And front view When the drone model is in a certain position, it receives a positive incentive; if it exceeds the angle limit, it is severely penalized with a value of -10. 3. Payload threat penalty and reward: As a situational awareness-based passive defense mechanism, the velocity normalization factor in this embodiment... , For the first Relative approach speed of target intelligent agent payload By truncation function Limit the threat value to Within the range, when the payload approaches the drone model at high speed, the reward value becomes negative, forcing the drone model to perform a sharp turn to reduce the approach speed and maximize the probability of survival. The total reward value is calculated by linearly weighting each of the above-mentioned sub-rewards. This method transforms expert knowledge into mathematical reward signals, guiding drone models to actively seek the best attack position while avoiding entering the target intelligent agent's payload range, thus solving the cold start problem caused by sparse victory and defeat signals in long-term pursuit.

[0092] In step S103, the current pursuit strategy is used to control the drone model to perform pursuit actions, the environmental situation of the drone pursuit simulation environment is updated in the next moment, the reward function is used to guide the autonomous decision-making model to learn the pursuit strategy, the network parameters of the autonomous decision-making model are iteratively updated according to the data of each adversarial round, and the trained pursuit autonomous decision-making model is used to control the drone to perform pursuit actions.

[0093] It is understood that the embodiments of this application control the drone model to perform actions and update the environmental situation according to the current pursuit strategy, guide the model to learn by combining the reward function, and iteratively update the network parameters based on the data of each adversarial round. This enables the autonomous decision-making model to continuously optimize the pursuit strategy in continuous interaction, and ultimately achieve precise and stable control of the drone's pursuit actions, effectively improving the model's autonomous decision-making ability and game performance in complex adversarial scenarios.

[0094] Specifically, a low-level flight control system based on cascaded PID (Proportional-Integral-Derivative) is first constructed as a deterministic mapping channel from policy semantics to physical execution. The system adopts a three-level serial PID control structure, including an outer loop navigation / trajectory loop, a middle loop attitude loop, and an inner loop angular velocity / stabilization loop. The low-level flight control system receives the discrete pursuit and escape strategies (tracking, attacking, and evading commands) output in step S102 and first maps them to the expected flight state reference values ​​of the UAV model. The tracking command corresponds to setting the expected heading angle to align with the forward view and the expected altitude to match the target agent's altitude or occupy the energy advantage altitude. The evading command corresponds to setting the expected heading angle to fly in the opposite direction to the target agent's load direction and setting a dive command to increase the speed of the UAV model.

[0095] Subsequently, based on the deviation between the expected and measured values, the control quantity is calculated step by step by the PID controller. The longitudinal channel converts the altitude / velocity error into the expected pitch angle and the expected pitch rate, and finally calculates the elevator deflection. The lateral channel converts the heading error into the desired roll angle and desired roll rate, which are ultimately calculated as the aileron deflection. The directional channel will sideslip angle The error is converted into the desired yaw rate, which is then finally calculated as the rudder deflection. The thrust channel adjusts the throttle opening based on the deviation between the desired airspeed and the current airspeed. To form a four-dimensional aerodynamic control vector The aforementioned cascaded PID controller can also be replaced with other control structures such as LQR (Linear Quadratic Regulator) to adapt to control requirements in different scenarios.

[0096] The control vector is input into the six-DOF aircraft dynamics model to drive the UAV model to execute corresponding pursuit and escape actions, while simultaneously updating the next-moment situation of the UAV pursuit and escape simulation environment (position, velocity, attitude, and payload status of the UAV model and the target agent, etc.). In each simulation step, the total reward value is calculated through the aforementioned composite reward function to guide the autonomous decision-making model to learn the optimal pursuit and escape strategy. When the current adversarial round meets the termination criterion, the complete state, action, and reward trajectory data sequence for this round is collected. The data is stored in the experience replay buffer. When the amount of data in the buffer reaches the preset batch size, the PPO algorithm is used, combined with the algorithm framework of policy network and value network, to iteratively update the network parameters of the autonomous decision-making model. The above process is repeated until the model converges, and the trained autonomous decision-making model for pursuit and escape is obtained, which is used to control the UAV to perform pursuit and escape actions.

[0097] In this embodiment, the network parameters of the autonomous decision-making model are iteratively updated based on the data of each adversarial round, including: identifying the algorithm framework of the autonomous decision-making model, which includes a policy network and a value network; calculating the temporal difference error based on the evaluation value of the value network and recursively calculating the advantage function; calculating the policy loss of the policy network and the value loss of the value network based on the data of each adversarial round, and calculating the total loss function based on the policy loss, value loss, and entropy regularization term; and iteratively updating the network parameters of the policy network based on the advantage function and the total loss function.

[0098] It is understood that the embodiments of this application, by adopting an algorithmic framework that includes a policy network and a value network, combining temporal difference error and advantage function calculation, and constructing a total loss function based on policy loss, value loss and entropy regularization term, can improve policy exploration efficiency while ensuring training stability, achieve efficient iterative updates of network parameters, and thus enable the autonomous decision-making model to converge quickly and obtain better decision performance in complex chase-and-escape game scenarios.

[0099] Specifically, the algorithmic framework of the autonomous decision-making model is an Actor-Critic architecture, in which the policy network is used to output the action probability distribution and sample actions, and the value network is used to estimate the state value. The two work together to achieve iterative updates of network parameters, as follows: 1. Data Preparation: Using the current strategy, control the aircraft in a simulation environment to collect complete trajectory data sequences for each adversarial round. ,in The current state. The current pursuit and escape strategy (action). The composite reward value at the current moment. To prepare for the next state, trajectory data is stored in an experience replay buffer. When the buffer reaches a preset batch size, parameter updates are triggered. The simulation environment decision cycle is 2 seconds, but due to the large distance between the two aircraft, the decision cycle is longer. Before training, the observation vectors are normalized by dividing the observation in each dimension by a preset maximum amplitude, thereby normalizing the observations to the range [0,1], which facilitates neural network training.

[0100] 2. Calculation of temporal difference error and dominance function: based on value network , Estimate the current state value for the weight parameters of the value network. Value of the state at the next moment Combined with the current reward value Calculate the TD (Temporal Difference) residuals. ,in As a discount factor, a value of 0.99 can be used in this embodiment to weigh immediate rewards against long-term returns; subsequently, GAE (Generalized Advantage Estimation) is used to recursively calculate the advantage function. ,in The GAE smoothing factor, set to 0.95 in this embodiment, is used to balance the bias and variance of the advantage estimation. The advantage value... Representation of action The degree of superiority or inferiority relative to the average level.

[0101] 3. Loss function calculation: First, calculate the policy probability ratio. ,in These are the current weight parameters of the policy network. The old policy distribution before the update is given; then the PPO truncated policy loss is constructed. ,in The cutoff coefficient is set to 0.2 in this embodiment, limiting the range of variation in the probability ratio between the new and old strategies. In between, ensure the monotonicity of updates; calculate value loss. ,in For target value, usually , The value loss coefficient is set to 0.1 in this embodiment to ensure that the loss function of the value network is generally larger than that of the policy network, so that they are roughly on the same order of magnitude. The entropy regularization term is then calculated. = ,in Policy entropy measures the randomness of a policy and encourages agents to explore. is the entropy regularization coefficient. Since policy entropy is a regularization term, its weight should not be too high; therefore, in this embodiment, it is set to 0.001. Finally, the total loss function is constructed. .

[0102] 4. Parameter Update: By maximizing the total loss function Iteratively update the weight parameters of the policy network. Weight parameters of the value network The PPO algorithm's truncation mechanism is used to limit the policy update step size, preventing performance collapse due to policy mutations in complex six-degree-of-freedom dynamic environments. The data preparation, error calculation, loss calculation, and parameter update processes are repeated. Through continuous iteration with a fixed-rule target agent policy, the network parameters of the autonomous decision-making model converge, resulting in the optimal pursuit-escape decision model. The hyperparameters of the agent model during training are shown in Table 1. Table 1

[0103] The air combat intelligent decision-making method based on temporal reinforcement learning proposed in this application can effectively capture the dynamic dependencies in the pursuit process by extracting the temporal features of the observation vectors in the simulation environment and inputting them into the pursuit and escape autonomous decision-making model. This avoids the short-sightedness of decision-making caused by relying solely on information at the current moment, thereby improving the rationality and adaptability of decision-making under long-term confrontation. By outputting the strategic intents of tracking, attacking, and evading and combining them with reward functions to guide learning, the UAV can autonomously optimize its decision-making behavior in complex dynamic environments, enabling it to have stronger autonomous decision-making capabilities and adversarial advantages in beyond-visual-range pursuit and escape games.

[0104] The following will elaborate on the proposed air combat intelligent decision-making method based on temporal reinforcement learning through a specific embodiment, such as... Figure 2 As shown, the specific steps are as follows: In step one, the high-fidelity pursuit environment is constructed and initialized.

[0105] A high-fidelity pursuit simulation environment based on the JSBSim flight simulation engine was constructed, integrating a six-DOF aircraft dynamics model and a three-DOF payload kinematics model. The drone model to be trained was designated as the friendly agent, while the target agent, following an expert knowledge base and carrying standard payloads, was designated as the adversary with fixed rules. The positions, velocities, attitudes, and other states of both parties were initialized to provide a physically-compliant interactive foundation for subsequent adversarial training.

[0106] In step two, multi-source situational observation and temporal feature extraction are performed.

[0107] At each simulation decision step, multi-source observation vectors are acquired and assembled in real time via the environment interface. These vectors include the UAV's own flight state, the target's relative motion parameters, and the target payload's state information. This vector is then input into a policy network containing a feature extraction network and gated recurrent units, such as... Figure 3 As shown, feature encoding is first completed through a fully connected layer, ReLU activation and layer normalization, and then GRU is used to fuse historical trajectory information to extract dynamic features containing temporal dependencies, thus solving the "partial observability" problem of single-frame observation.

[0108] In step three, policy intent decisions are made based on a hierarchical architecture.

[0109] A hierarchical decision-making mechanism is adopted, defining the output space of the upper-level policy network as discrete policy intentions (tracking, attacking, evading), such as... Figure 3 As shown, the policy network maps the temporal features extracted in step two into an action probability distribution, and outputs the optimal policy intent at the current moment through Softmax sampling. This transforms the complex continuous control problem into a high-level intent selection, significantly compressing the exploration space and avoiding training oscillations.

[0110] In step four, the underlying control calculations and the physics engine interact.

[0111] Construct a low-level flight control system based on cascaded PID (or LQR), such as Figure 3 As shown, the discrete strategy intent output from the upper layer is mapped to specific aileron, elevator, rudder deflection, and throttle opening. These control quantities are input into the six-DOF aircraft dynamics model to drive the UAV to perform maneuvers and simultaneously update the next-moment situation of the simulation environment.

[0112] In step five, the composite reward is calculated.

[0113] The algorithm calculates a composite reward value that includes endgame outcome, forward-view guidance, crash penalty, attack incentive, and payload threat. By combining sparse rewards (outcome, crash) and dense rewards (forward-view, attack, threat), the algorithm guides the agent to actively seek the optimal attack position while avoiding entering the target payload range, thus solving the cold start problem in long-term games.

[0114] In step six, the policy network is iteratively updated.

[0115] If the termination criterion is not met in the current adversarial round, the state, action, and reward trajectory data are stored in the experience replay buffer, triggering the proximal policy optimization algorithm to update the network parameters. The advantage function is calculated using generalized advantage estimation, and a truncation mechanism is used to limit the policy update step size, improving policy performance while ensuring training stability. The updated policy network is then returned to step two to begin the next iteration. Figure 4 As shown, during training, the agent's average round reward exhibits a continuously increasing and gradually converging trend, reflecting the continuous optimization of the strategy; as Figure 5 As shown, with the increase in training steps, the win rate of the agent against the fixed expert policy continuously improves while the loss rate significantly decreases, verifying the effectiveness of the proposed method. Table 2 shows the comparison of the agent model's performance against the expert policy on the test set scenarios: Table 2

[0116] In tests conducted on 50 randomly generated scenarios, the agent trained using reinforcement learning significantly outperformed the strategies of expert systems, achieving superior training results. Because the opponent's strategy was stronger, it performed well in most scenarios, while expert strategies, with their fixed rules, were difficult to adapt to stronger opponents, resulting in poor performance. Reinforcement learning, on the other hand, easily generated corresponding counter-strategies to counter the opponent's strategy, thus achieving better results. Table 3 shows the win rate comparison of the agent model on different test examples. Table 3

[0117] For the agent trained by reinforcement learning, three sets of test scenarios were randomly generated under different random number seeds, with 50 scenarios in each set, and the test was conducted. The results show that the win rate and other values ​​remain high in different scenarios, and it has an overwhelming advantage against the opponent's strategy, and has strong generalization ability.

[0118] When the termination criterion is met in a round, training is complete, and the optimal air combat agent model is output, resulting in an autonomous decision-making model for UAV pursuit and escape that can be directly used in actual combat.

[0119] In summary, the embodiments of this application have at least the following beneficial effects: (1) Significantly enhance situational awareness and game decision-making ability under incomplete information: By introducing a gated loop unit to construct a temporal memory mechanism, it is possible to extract potential strategic intentions and energy state information from historical trajectories, effectively solving the "non-Markov" and "partially observable" problems that are widespread in the pursuit and escape game, enabling the agent to make strategic actions with long-term planning rather than short-sighted stress responses.

[0120] (2) Effectively overcome the problem of exploration and convergence in high-dimensional continuous action space: adopting a hierarchical decision-making architecture with discrete policy intent and continuous bottom-level control, decoupling the complex six-degree-of-freedom manipulation problem, greatly compressing the search space of reinforcement learning, avoiding the model non-convergence phenomenon caused by high-frequency oscillation of policy in the early stage of training, and ensuring that the generated flight trajectory is dynamically smooth, stable and executable.

[0121] (3) Ensure the physical feasibility of intelligent strategies in complex mechanical environments: The rotational inertia coupling and nonlinear aerodynamic characteristics of a six-degree-of-freedom aircraft are fully introduced into the training environment. The trained strategies fully consider energy loss, attitude response delay and overload limitation during maneuvering, effectively eliminating the gap between simulation and actual combat, and making the actions generated by the intelligent agent have high engineering reference value and practical potential.

[0122] (4) Achieve efficient strategy restraint and cold start acceleration against expert strategies: By introducing a composite reward function, the problem of extremely sparse win-loss rewards in long-cycle pursuit is solved. It can guide the agent to quickly master energy management and position grabbing skills in the early stage of training. It can evolve an effective restraint strategy against fixed expert rules within a short number of training iterations, significantly reducing the time cost and computing power consumption of training.

[0123] Next, referring to the accompanying drawings, we describe the air combat intelligent decision-making device based on temporal reinforcement learning proposed according to the embodiments of this application.

[0124] Figure 6 This is a block diagram of an air combat intelligent decision-making device based on temporal reinforcement learning, according to an embodiment of this application.

[0125] like Figure 6 As shown, the air combat intelligent decision-making device 60 based on temporal reinforcement learning includes: an acquisition module 601, an output module 602, and a control module 603.

[0126] The acquisition module 601 is used to perform adversarial simulation on the UAV model in the UAV pursuit simulation environment, acquire the observation vector of the UAV model at each simulation decision step, and extract temporal features from the observation vector; the output module 602 is used to input the temporal features into the pursuit autonomous decision model, and output the pursuit strategy at the current moment through the pursuit autonomous decision model, the pursuit strategy includes at least one of the following: tracking command, attack command, and evasion command; the control module 603 is used to control the UAV model to execute pursuit actions using the pursuit strategy at the current moment, update the environmental situation of the UAV pursuit simulation environment at the next moment, guide the autonomous decision model to learn the pursuit strategy using the reward function, iteratively update the network parameters of the autonomous decision model according to the data of each adversarial round, and control the UAV to execute pursuit actions using the trained pursuit autonomous decision model.

[0127] In this embodiment of the application, the acquisition module 601 is further used to: construct the aircraft dynamics model and the load kinematics model of the UAV; construct a UAV pursuit and escape simulation environment based on the aircraft dynamics model and the load kinematics model, the UAV pursuit and escape simulation environment includes the UAV model and the target intelligent agent based on fixed rule strategy, and both the UAV model and the target intelligent agent acquire each other's position, speed and attitude information in real time.

[0128] In this embodiment of the application, the aircraft dynamics model includes translational equations and rotational equations, wherein the translational equations are:

[0129] in, For aircraft quantity, and The three-axis velocity components of the body shaft system. For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the roll angular velocity, The pitch angular velocity, Yaw angular velocity, , and This represents the total external force components under the machine body's axis system; The equation of rotation is:

[0130] in, , and Principal moment of inertia, For inertial product, , and This represents the total external torque component under the machine body axis. For roll acceleration, For pitch acceleration, Yaw acceleration; The load kinematic equations of the load kinematic model in the inertial coordinate system are described as follows:

[0131] in, , and The three-axis velocity components of the load in the inertial coordinate system. For load speed, The pitch angle of the flight path. The yaw angle of the track; The load kinematic equations of the load kinematic model are described in the ballistic coordinate system as follows:

[0132] in, It is the acceleration due to gravity. For tangential overload, and To control overload laterally, Let be the tangential acceleration of the load. Yaw angular velocity, It represents the pitch angular velocity.

[0133] In this embodiment of the application, the acquisition module 601 is further configured to: input the observation vector into the policy network, and output the temporal features through the policy network, wherein the policy network includes a feature extraction network and a gated recurrent unit, the feature extraction network extracts the features of the observation vector, the features of the observation vector are input into the gated recurrent unit, and the gated recurrent unit extracts the temporal features containing historical trajectory information during the pursuit process.

[0134] In this embodiment, the output module 602 is further configured to: define the output space of the policy network as a discretized policy intent through a hierarchical decision-making mechanism; construct an autonomous pursuit decision-making model based on the discretized policy intent; sample the temporal features using the autonomous pursuit decision-making model; and determine the pursuit strategy at the current moment based on the sampling results. The probability distribution sampling mechanism of the autonomous pursuit decision-making model is as follows:

[0135] in, For time series feature vectors, , These are the weight matrix and bias vector of the output layer, respectively. This is the action logic value. and The first The and the first The exponent term of each action logic value is used to calculate the action probability. This represents the action probability distribution output by the policy network. The policy intent instruction is obtained by the agent based on the current policy distribution. It is a set of discrete action spaces.

[0136] In this embodiment of the application, the expression for the reward function is:

[0137] in, The total reward value, Rewards for the final outcome of the game. Rewards are provided to guide users from a forward-looking perspective. As a reward for crash punishment, As a reward for attack launch, Payload threat penalty and reward; Endgame and security constraints rewards:

[0138]

[0139] in, The crash penalty coefficient is h, where h is the flight altitude. For the crash height threshold, For safety alarm height, The descent speed is the velocity of the ground. The descent speed threshold, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. Positional advantage and attack bonus:

[0140]

[0141] in, Forward perspective, As a reward for attack launch, Penalty for the number of launches, Rewards for advantageous attack positions; The payload threat is:

[0142] in, For the first The relative approach speed of each load, As the velocity normalization factor, This is a truncation function.

[0143] In this embodiment, the control module 603 is further configured to: identify the algorithm framework of the autonomous decision-making model, the algorithm framework including a policy network and a value network; calculate the temporal difference error based on the evaluation value of the value network, and recursively calculate the advantage function; calculate the policy loss of the policy network and the value loss of the value network based on the data of each adversarial round, and calculate the total loss function based on the policy loss, value loss and entropy regularization term; and iteratively update the network parameters of the policy network based on the advantage function and the total loss function.

[0144] It should be noted that the foregoing explanation of the embodiment of the air combat intelligent decision-making method based on temporal reinforcement learning also applies to the air combat intelligent decision-making device based on temporal reinforcement learning in this embodiment, and will not be repeated here.

[0145] The air combat intelligent decision-making device based on temporal reinforcement learning proposed in this application can effectively capture the dynamic dependencies in the pursuit process by extracting the temporal features of the observation vectors in the simulation environment and inputting them into the pursuit and escape autonomous decision-making model. This avoids the short-sightedness of decision-making caused by relying solely on information at the current moment, thereby improving the rationality and adaptability of decision-making under long-term confrontation. By outputting the strategic intents of tracking, attacking, and evading and combining them with reward functions to guide learning, it can autonomously optimize decision-making behavior in complex dynamic environments, enabling UAVs to have stronger autonomous decision-making capabilities and adversarial advantages in beyond-visual-range pursuit and escape games.

[0146] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 701, the processor 702, and the computer program stored on the memory 701 and executable on the processor 702.

[0147] When the processor 702 executes the program, it implements the air combat intelligent decision-making method based on temporal reinforcement learning provided in the above embodiments.

[0148] Furthermore, electronic devices also include: Communication interface 703 is used for communication between memory 701 and processor 702.

[0149] The memory 701 is used to store computer programs that can run on the processor 702.

[0150] The memory 701 may include high-speed RAM (Random Access Memory) memory, and may also include non-volatile memory, such as at least one disk storage.

[0151] If the memory 701, processor 702, and communication interface 703 are implemented independently, then the communication interface 703, memory 701, and processor 702 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0152] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, then the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0153] The processor 702 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of this application.

[0154] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described air combat intelligent decision-making method based on temporal reinforcement learning.

[0155] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0156] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0157] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0158] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0159] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0160] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A smart air combat decision-making method based on temporal reinforcement learning, characterized in that, Includes the following steps: In a drone pursuit simulation environment, an adversarial simulation is performed on a drone model to obtain the observation vector of the drone model at each simulation decision step, and temporal features are extracted from the observation vector. The time-series features are input into the autonomous pursuit and escape decision-making model, and the autonomous pursuit and escape decision-making model outputs the pursuit and escape strategy at the current moment. The pursuit and escape strategy includes at least one of the following: tracking command, attack command, and evasion command. The drone model is controlled to perform pursuit actions using the pursuit strategy at the current moment, the environmental situation of the drone pursuit simulation environment is updated in the next moment, the autonomous decision-making model is guided to learn the pursuit strategy using the reward function, the network parameters of the autonomous decision-making model are iteratively updated based on the data of each adversarial round, and the trained pursuit autonomous decision-making model is used to control the drone to perform pursuit actions.

2. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 1, characterized in that, In a drone pursuit simulation environment, adversarial simulation of drone models is conducted, including: Construct the aircraft dynamics model and load kinematics model of the UAV; A drone pursuit simulation environment is constructed based on the aircraft dynamics model and the load kinematics model. The drone pursuit simulation environment includes the drone model and the target intelligent agent based on a fixed rule strategy. Both the drone model and the target intelligent agent acquire the other's position, speed and attitude information in real time.

3. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 2, characterized in that, The aircraft dynamics model includes translational and rotational equations, wherein... The translation equation is: in, For the mass of the aircraft, and The three-axis velocity components of the body shaft system. For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the body axis Directional velocity components rate of change over time For the roll angular velocity, The pitch angular velocity, Yaw angular velocity, , and This represents the total external force components under the machine body's axis system; The equation of rotation is: in, , and Principal moment of inertia, For inertial product, , and This represents the total external torque component under the machine body axis. For roll acceleration, For pitch acceleration, Yaw acceleration; The load kinematic equations of the load kinematic model are described in the inertial coordinate system as follows: in, , and The three-axis velocity components of the load in the inertial coordinate system. For load speed, For the pitch angle of the flight path, The yaw angle of the track; The load kinematic equations of the load kinematic model are described in the ballistic coordinate system as follows: in, It is the acceleration due to gravity. For tangential overload, and To control overload laterally, Let be the tangential acceleration of the load. Yaw angular velocity, It represents the pitch angular velocity.

4. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 1, characterized in that, The extraction of temporal features from the observation vector includes: The observation vector is input into a policy network, and the policy network outputs the temporal features. The policy network includes a feature extraction network and a gated recurrent unit. The feature extraction network extracts features from the observation vector, and the features from the observation vector are input into the gated recurrent unit. The gated recurrent unit extracts temporal features containing historical trajectory information during the pursuit process.

5. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 4, characterized in that, The process of outputting the current pursuit strategy through the pursuit autonomous decision-making model includes: The output space of the policy network is defined as a discretized policy intent through a hierarchical decision-making mechanism. The autonomous pursuit decision-making model is constructed based on the discretized strategy intent. The model then performs probability distribution sampling on the temporal features, and determines the pursuit strategy for the current moment based on the sampling results. The probability distribution sampling mechanism of the autonomous pursuit decision-making model is as follows: in, For time series feature vectors, and These are the weight matrix and bias vector of the top-level output layer of the policy network, respectively. This is the action logic value. and The first The and the first The exponential term of the action logic value, This represents the action probability distribution output by the policy network. The policy intent commands are sampled from the current policy distribution of the UAV model. It is a set of discrete action spaces.

6. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 1, characterized in that, The expression for the reward function is: in, The total reward value, Rewards for the final outcome of the game. Rewards are provided to guide users from a forward-looking perspective. As a reward for crash punishment, As a reward for attack launch, Payload threat penalty and reward; The endgame and security constraint rewards: in, The crash penalty coefficient is h, where h is the flight altitude. For the crash height threshold, For safety alarm height, The descent speed is the velocity of the ground. The descent speed threshold, This is an indicator function that takes the value 1 when the condition is met, and 0 otherwise. Positional advantage and attack bonus: in, Forward perspective, As a reward for attack launch, Penalty for the number of launches, Rewards for advantageous attack positions; The load threat is: in, For the first The relative approach speed of each load, As the velocity normalization factor, This is a truncation function.

7. The air combat intelligent decision-making method based on temporal reinforcement learning according to claim 1, characterized in that, The step of iteratively updating the network parameters of the autonomous decision-making model based on the data from each adversarial round includes: An algorithmic framework for identifying the autonomous decision-making model, comprising a policy network and a value network; The time-series difference error is calculated based on the evaluation value of the value network, and the dominance function is calculated recursively. The policy loss of the policy network and the value loss of the value network are calculated based on the data of each adversarial round. The total loss function is then calculated based on the policy loss, the value loss, and the entropy regularization term. The network parameters of the policy network are iteratively updated based on the advantage function and the total loss function.

8. An intelligent air combat decision-making device based on temporal reinforcement learning, characterized in that, include: The acquisition module is used to perform adversarial simulation on the UAV model in a UAV pursuit simulation environment, acquire the observation vector of the UAV model at each simulation decision step, and extract temporal features from the observation vector; The output module is used to input the time-series features into the autonomous pursuit and escape decision-making model, and output the pursuit and escape strategy at the current moment through the autonomous pursuit and escape decision-making model. The pursuit and escape strategy includes at least one of the following: tracking instruction, attack instruction, and evasion instruction. The control module is used to control the drone model to perform pursuit and escape actions using the pursuit and escape strategy at the current moment, update the environmental situation of the drone pursuit and escape simulation environment at the next moment, guide the autonomous decision-making model to learn the pursuit and escape strategy using the reward function, iteratively update the network parameters of the autonomous decision-making model based on the data of each adversarial round, and control the drone to perform pursuit and escape actions using the trained pursuit and escape autonomous decision-making model.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the air combat intelligent decision-making method based on temporal reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they implement the air combat intelligent decision-making method based on temporal reinforcement learning as described in any one of claims 1-7.

Images