A beyond-visual-range air combat double-loop coupling autonomous maneuver decision-making method, device, medium and product based on situation driving

Abstract

The application discloses a beyond-visual-range air combat double-loop coupling autonomous maneuver decision-making method and device based on situation driving, a medium and a product, and relates to the technical field of aerospace. The method comprises the following steps: using a trained LSTM model, respectively according to state control information of an enemy target and a local machine, performing recursive prediction to obtain multi-step track prediction information of the enemy target and the local machine; calculating the situation change gradient of both sides; inputting the track information of the local machine and the state control information of the enemy target into a trained first reinforcement learning model to generate a main action instruction; inputting the track information of the local machine, the state control information of the enemy target and the situation change gradient of both sides into a trained second reinforcement learning model to generate a preloaded action instruction; and using a null space behavior method to fuse the main action instruction and the preloaded action instruction to obtain an autonomous maneuver decision-making instruction. The application can reduce the decision-making risk under incomplete information and realize smooth tactical conversion of a combat aircraft.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aerospace technology, in particular to a situational driving-based beyond-visual-range air combat double-loop coupled autonomous maneuver decision-making method, device, medium and product. BACKGROUND

[0002] With the continuous development of air combat technology, the performance of aircraft is getting stronger, the range of air-to-air missiles is getting farther, and the range of air combat engagement area is also getting larger, and beyond-visual-range air combat occupies a dominant position in future air combat. In modern air combat, multi-aircraft cooperative air combat has higher combat capability and combat efficiency than single-aircraft air combat. With the maturation of unmanned aerial vehicle formation combat technology, the battlefield pattern has evolved from single-platform confrontation to multi-agent cooperative battlefield. However, due to the limitations of sensor performance and the implementation of interference and deception by the enemy in the actual battlefield environment, incomplete information sampling often occurs. Since the information obtained by our side is the historical information of the enemy, the sudden switching of the tactical state of the enemy aircraft will cause the delay of our decision-making, and the state instability in the process of emergency switching. Facing the dynamic confrontation game scene of beyond-visual-range air combat, in view of the problem of sudden state switching of warplanes caused by incomplete information, how to make maneuver decision to realize the smooth tactical conversion of warplanes has important research significance. SUMMARY

[0003] The purpose of the present application is to provide a situational driving-based beyond-visual-range air combat double-loop coupled autonomous maneuver decision-making method, device, medium and product, which can reduce the decision-making risk under incomplete information and realize the smooth tactical conversion of warplanes.

[0004] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0005] In a first aspect, the present application provides a situational driving-based beyond-visual-range air combat double-loop coupled autonomous maneuver decision-making method, comprising:

[0006] obtaining state control information of an enemy target and a host aircraft; the state control information includes state information and control quantity information;

[0007] using a trained LSTM model to perform recursive prediction according to the state control information of the enemy target, to obtain multi-step track prediction information of the enemy target; the multi-step track prediction information includes predicted state control information at a plurality of time steps;

[0008] using a trained LSTM model to perform recursive prediction according to the state control information of the host aircraft, to obtain multi-step track prediction information of the host aircraft;

[0009] calculating the situational change gradient of both sides according to the multi-step track prediction information of the enemy target and the multi-step track prediction information of the host aircraft;

[0010] The system inputs the local trajectory information and the state control information of the enemy target into the trained first reinforcement learning model, and uses a deep deterministic policy gradient algorithm suitable for continuous action space to generate the main action command; the local trajectory information includes: state information, control information, sensor information and weapon information;

[0011] The machine's trajectory information, the enemy target's state control information, and the situation change gradients of both sides are input into the trained second reinforcement learning model, and a deep deterministic policy gradient algorithm suitable for continuous action space is used to generate preloaded action instructions.

[0012] By using the zero-space behavior method, the active action command and the preloaded action command are fused together to obtain the autonomous maneuver decision command.

[0013] Optionally, the step of calculating the situational change gradient of both sides based on the multi-step trajectory prediction information of the enemy target and the multi-step trajectory prediction information of the local machine specifically includes:

[0014] Based on the enemy target and the predicted state control information of the machine at each time step, the situational value for the corresponding time step is calculated, resulting in situational values ​​for several time steps. The formula is as follows: Where S represents the current situation function, This represents a function that describes basic characteristics such as relative distance, angle, and velocity. The total number of basic characteristics, that is, the number of fundamental physical quantities used to describe the situation. Representing the One "basic feature" Representative description of the first Functions with fundamental characteristics Indicates the first The weighting coefficients of the basic features; This indicates the tactical characteristics of the current tactic. Weighting coefficients representing tactical characteristics; This represents the dynamic adversarial characteristics in the current state. Weight coefficients representing dynamic adversarial features;

[0015] The gradient of the situational changes for both sides is calculated based on the situational values ​​at several time steps, using the following formula: , where t represents time.

[0016] Optionally, the step of using the zero-space behavior method to fuse the active action command with the pre-loaded action command to obtain the autonomous maneuver decision command specifically includes:

[0017] The main task Jacobian matrix is ​​calculated based on the main action instruction to obtain the main task null projection matrix;

[0018] projecting the pre-loaded action instruction into the main task null space according to the main task null space projection matrix to obtain a projection result;

[0019] fusing the projection result and the main action instruction to obtain a self-motion decision instruction.

[0020] Optionally, a main task Jacobian matrix is calculated according to the main action instruction, and a calculation formula of the main task null space projection matrix is:

[0021]

[0022]

[0023] wherein, the main task null space projection matrix, the control variable vector, the main task function, the unit matrix, the Jacobian matrix of the main task function, the pseudo-inverse of the Jacobian matrix.

[0024] Optionally, a formula of the projecting the pre-loaded action instruction into the main task null space according to the main task null space projection matrix to obtain a projection result is:

[0025]

[0026] wherein, the projection result, the main task null space projection matrix, u 2 the pre-loaded action instruction.

[0027] Optionally, the state control information of the enemy target and the host are obtained; specifically including:

[0028] obtaining state information of the enemy target;

[0029] calculating control variable information of the enemy target according to the state information of the enemy target by a control variable inverse solution method;

[0030] obtaining state control information of the host.

[0031] Optionally, the training process of the LSTM model includes:

[0032] obtaining historical state control information; the historical state control information includes historical state control information at a previous time and historical state control information at a next time; ​​​​

[0033] training the LSTM network by using the historical state control information to obtain an LSTM model;

[0034] The training process of the first reinforcement learning model comprises:

[0035] obtaining first historical learning information; the first historical learning information comprises historical native track information, historical state control information of an enemy target and first historical action instruction information;

[0036] training the first reinforcement learning model by using the first historical learning information and a deep deterministic policy gradient algorithm to maximize cumulative rewards; the first reinforcement learning model is used to generate a main action instruction;

[0037] The training process of the second reinforcement learning model comprises:

[0038] obtaining second historical learning information; the second historical learning information comprises historical native track information, historical state control information of an enemy target, historical situation change gradient values of both sides and second historical action instruction information;

[0039] training the second reinforcement learning model by using the second historical learning information and a deep deterministic policy gradient algorithm to maximize cumulative rewards; the second reinforcement learning model is used to generate a preloaded action instruction.

[0040] In a second aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the situation-driven bistable loop coupled autonomous maneuver decision-making method for beyond-visual-range air combat.

[0041] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the situation-driven bistable loop coupled autonomous maneuver decision-making method for beyond-visual-range air combat.

[0042] In a fourth aspect, the present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the situation-driven bistable loop coupled autonomous maneuver decision-making method for beyond-visual-range air combat.

[0043] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0044] This application provides a situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method, device, medium, and product. The method includes: acquiring state control information of the enemy target and the machine itself; the state control information includes state information and control variable information; using a trained LSTM model, recursively predicting based on the enemy target's state control information to obtain multi-step trajectory prediction information of the enemy target; the multi-step trajectory prediction information includes predicted state control information for several time steps; using a trained LSTM model, recursively predicting based on the machine's state control information to obtain the machine's multi-step trajectory prediction information; and based on the enemy target's multi-step trajectory prediction information and the machine's multi-step trajectory prediction information... The system measures information and calculates the gradient of situational changes for both sides. It inputs the pilot's flight path information and the enemy target's state control information into a trained first reinforcement learning model, and uses a deep deterministic policy gradient algorithm suitable for continuous action space to generate the main action command. The pilot's flight path information includes state information, control information, sensor information, and weapon information. It then inputs the pilot's flight path information, the enemy target's state control information, and the gradient of situational changes for both sides into a trained second reinforcement learning model, and uses a deep deterministic policy gradient algorithm suitable for continuous action space to generate pre-loaded action commands. Using the null space behavior method, the main action command and the pre-loaded action command are fused to obtain the autonomous maneuver decision command. This application proactively addresses information gaps by predicting situational change gradients and designing a pre-trigger mechanism; it ensures smooth action through a dual-loop coupling architecture and command fusion. Before an emergency actually occurs, the system completes partial response preparation through an auxiliary decision loop, thereby transforming the passive behavior of "emergency state switching" into "prediction-based proactive transition," which can reduce the decision-making risk under incomplete information and improve the overall stability and survivability of the aircraft in dynamic air combat. Therefore, this invention addresses the problem of emergency switching of fighter jets due to sudden situations caused by incomplete information, and realizes a decision-making shift from "emergency response" to "pre-response". Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an application environment diagram of a situation-driven beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method according to an embodiment of this application.

[0047] Figure 2 This is a flowchart illustrating a situation-driven, beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method provided in one embodiment of this application.

[0048] Figure 3 This is a schematic diagram of an LSTM network structure provided in an embodiment of this application.

[0049] Figure 4 This is a schematic diagram of the LSTM trajectory prediction process provided in an embodiment of this application.

[0050] Figure 5 This is a schematic diagram of the basic framework of reinforcement learning provided in an embodiment of this application.

[0051] Figure 6 This is a schematic diagram of the NSB provided in one embodiment of this application.

[0052] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] This application provides a situation-driven, beyond-visual-range (BVR) air combat dual-loop coupled autonomous maneuver decision-making method. First, a dual-loop coupled decision architecture for BVR air combat is designed: the main decision loop uses reinforcement learning to generate active action commands based on the current state, ensuring the stability of current tactical execution; the auxiliary decision loop, when the situation-driven tactical pre-triggering conditions are met, uses reinforcement learning to generate pre-loaded action commands, achieving advantage accumulation; finally, the null space behavior method is combined to fuse the active action commands and the pre-loaded action commands. Second, a situation-driven tactical pre-triggering mechanism is constructed: a situation quantification model is established, and an LSTM network is used to predict enemy and friendly multi-step trajectories, calculating the gradient of predicted situational changes; based on this gradient, pre-triggering logic is designed to activate the auxiliary decision loop before the situation begins to change. For dynamic adversarial game scenarios in BVR air combat, this method can effectively predict the trend of situational evolution, significantly reduce the impact of sudden states, and achieve smooth and coherent tactical transitions in typical scenarios with incomplete information such as limited sensors and unclear opponent intentions. This algorithm effectively solves the problem of emergency state switching caused by sudden situations under incomplete information, and improves the survivability of autonomous combat platforms in dynamic and uncertain air combat environments.

[0055] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The situation-driven, beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server.

[0057] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method is provided. This method is executed by computer equipment, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S1 to S7. Wherein:

[0059] S1. Obtain the status control information of the enemy target and the local machine; the status control information includes: status information and control quantity information.

[0060] The process involves first obtaining the enemy target's status information, and then calculating the enemy target's control quantity information using the control quantity inverse solution method based on the enemy target's status information.

[0061] In this embodiment, the dynamics and kinematics models of UAVs and missiles, the quantitative model of beyond-visual-range air combat situation, and the tactical pre-trigger conditions are first constructed.

[0062] First, three-degree-of-freedom dynamics and kinematics modeling is performed for the UAV and missile. When studying air combat maneuvering decisions, to accurately describe the aircraft's trajectory and maneuvering characteristics, a three-degree-of-freedom point-mass kinematics model of the aircraft is performed in three-dimensional space. To focus the research on air combat maneuvering decisions, this embodiment establishes the model based on the following conditions:

[0063] (1) During the flight of the aircraft, the influence of airflow speed on the flight of the aircraft is not considered.

[0064] (2) The gravitational acceleration g remains constant, and the influence of environmental factors such as altitude and air density on acceleration is not considered.

[0065] (3) The size of the aircraft has negligible effect on the motion, and the aircraft is regarded as a movable point mass.

[0066] (4) The aircraft does not experience sideslip during its movement.

[0067] (5) During the flight of the aircraft, the ground reference frame remains stationary and the Earth’s rotation is not considered.

[0068] Based on the above conditions, the dynamic equations of a three-degree-of-freedom particle in three-dimensional space can be expressed as:

[0069] ;

[0070] In the formula, This indicates the position of the aircraft in the inertial coordinate system; Indicates the speed of the aircraft Components on the three coordinate axes; These represent the track inclination angle, track deviation angle, and roll angle, respectively. These are expressed as the first derivatives of velocity, track inclination, and track deviation with respect to time, respectively. and These represent the tangential overload along the velocity direction and the normal overload perpendicular to the velocity direction, respectively; in the formula, the first three terms are the kinematic model of the aircraft mass, and the last three terms are the dynamic model of the aircraft; the state variables include and Control quantities include and .

[0071] From the three-degree-of-freedom particle dynamics equations of an aircraft in three-dimensional space, it can be seen that the aircraft's maneuvers can be derived from... n x , and Three variables are used to control the aircraft's maneuvers. To ensure the effectiveness of the research and to make the maneuver decisions made by reinforcement learning conform to the characteristics of a real maneuver model, factors such as the strength of the aircraft's internal structure and the normal operating conditions of various sensors need to be considered during the design phase. Constraints need to be placed on the control variables, as shown in the following formula:

[0072] ;

[0073] In the formula, , These represent the minimum usable values ​​for tangential overload and normal overload, respectively. , These represent the limit values ​​for tangential and normal overloads, respectively. , These represent the maximum roll angle and the minimum roll angle, respectively. The missile uses proportional guidance for guidance.

[0074] Considering the physical limits that an aircraft can withstand, , , , .

[0075] The beyond-visual-range air combat situational quantification model includes basic characteristics, tactical characteristics, and dynamic adversarial characteristics, as shown in the following formula:

[0076] ;

[0077] Where S represents the current situation function, This represents a function that describes basic characteristics such as relative distance, angle, and velocity. The total number of basic characteristics, that is, the number of fundamental physical quantities used to describe the situation. Representing the One "basic feature" Representative description of the first Functions with fundamental characteristics Indicates the first The weighting coefficients of the basic features; This indicates the tactical characteristics of the current tactic. Weighting coefficients representing tactical characteristics; This represents the dynamic adversarial characteristics in the current state. These represent the weighting coefficients for dynamic adversarial characteristics. The tactical characteristics of the current tactic and the dynamic adversarial characteristics under the current state are not included in the state information and control variable information; they are only used to calculate the situation gradient value.

[0078] Tactical pre-triggering conditions are characterized by the situation change gradient. If the predicted situation change gradient shows a certain change, the tactic is pre-triggered. The calculation of the situation change gradient is as follows:

[0079] .

[0080] S2. Using the trained LSTM model, recursively predict the enemy target's state control information to obtain multi-step trajectory prediction information; the multi-step trajectory prediction information includes predicted state control information for several time steps.

[0081] S3. Using the trained LSTM model, recursively predict based on the local state control information to obtain the local multi-step trajectory prediction information.

[0082] In this embodiment, LSTM is used to predict the multi-step flight paths of both sides, and the gradient of beyond-visual-range air combat situation changes is calculated based on the prediction results.

[0083] The structure of an LSTM network (Long Short-Term Memory Artificial Neural Network (Temporal Recurrent Neural Network)) is as follows: Figure 3 As shown, the prediction process is as follows: Figure 4 As shown, in At any given moment, a single layer of a network consists of two information flows, one above the other. From... arrive The information flow represents the transmission of cell states, and the entire line interacts linearly with the information flow below through three gating structures. The gating structures allow information to selectively pass through, flowing upwards... arrive The information flow deletes or adds information about the cell state. In the gating structure... Activation function layer and The activation function layer transforms the input into the ranges (0, 1) and (-1, 1) respectively, generating weights for the input data and thus filtering it. Each LSTM network layer has three gate structures to control the cell state.

[0084] (1) Gate of Oblivion:

[0085] The expression for the forget gate is as follows:

[0086] ;

[0087] in, Here is the weight matrix for the forget gate. Let be the bias vector of the forget gate. The layer uses the hidden state from the previous time step. and Input at any time A value between 0 and 1 is obtained as the probability that the cell state of the previous layer is forgotten, and is considered as... The decay coefficient of memory, denoted as .

[0088] (2) Input gate:

[0089] Input gate part will Hidden state from the previous moment After linear combination, through Layer activation is obtained This part determines which information needs to be updated; this information is the part selected to be forgotten through the forgetting gate. The other part will... and Through a The layer generates a vector This refers to the alternative content to be updated. Then, the two parts will be combined to update the state. Updated to The formula for the input gate update process is:

[0090] ;

[0091] ;

[0092] ;

[0093] In the formula: Here is the weight matrix of the input gate. This is the bias vector for the input gate. This is the weight matrix for cell states. This is the bias vector for the cell state. For the input gate at time The output vector, This represents convolution.

[0094] (3) Output gate:

[0095] ;

[0096] ;

[0097] In the formula: For the output gate at time The output vector, This is the weight matrix of the output gate. This is the bias vector for the cell state. For cells at any time The hidden state. Because each loop uses information from the previous loop. and Each output state is influenced by the previous state, so LSTM has the ability to remember long-term historical information.

[0098] Assuming the plane is currently in At all times, during the training phase, for and The model is trained using historical state information from the previous time step, with the previous time step's state information as input. and control quantity information , These respectively represent the aircraft in Location at any given moment Indicates that the aircraft is in At any given moment, the inclination angle, the deviation angle, and the speed. These respectively represent the aircraft in The model outputs the state information for the next time step, including the tangential overload along the velocity direction, the normal overload perpendicular to the velocity direction, and the roll angle. and control quantity information , These respectively represent the aircraft in Location at any given moment Indicates that the aircraft is in At any given moment, the inclination angle, the deviation angle, and the speed. These respectively represent the aircraft in The system continuously monitors tangential overload along the velocity direction, normal overload perpendicular to the velocity direction, and roll angle. During the prediction phase, LSTM is used to... Status information and control quantity information Prediction is obtained and .Will As a data tuple, then the predicted values As input, recursively predict subsequent information. This is to achieve the goal of multi-step trajectory prediction.

[0099] The control parameters of the enemy target input to the model can be calculated from the state information. Firstly, the inertial frame acceleration can be obtained through state estimation:

[0100] ;

[0101] ;

[0102] in, These are the velocity components of the aircraft in the inertial coordinate system. They are respectively The first derivative with respect to time, These represent the acceleration components of the aircraft in the inertial coordinate system. If the acceleration vector is given, then the non-gravitational acceleration is:

[0103] ;

[0104] in, It is the acceleration due to gravity. The vector is a non-gravitational acceleration vector. Since the basis vectors of the track coordinate system are:

[0105] ;

[0106] in, Let them be the tangential basis vectors. Normal basis vectors, Given the horizontal basis vectors, the output control quantity can be obtained as follows:

[0107] ;

[0108] Therefore, the tangential overload can be solved. Normal overload and roll angle Three control variables.

[0109] S4. Calculate the gradient of situational change for both sides based on the multi-step trajectory prediction information of the enemy target and the multi-step trajectory prediction information of the machine.

[0110] After obtaining the predicted multi-step trajectory, the corresponding situational gradient value is calculated. The beyond-visual-range air combat situational quantification model includes basic characteristics, tactical characteristics, and dynamic adversarial characteristics, as shown in the following formula:

[0111] ;

[0112] Where S represents the current situation function, This represents a function that describes basic characteristics such as relative distance, angle, and velocity. The total number of basic characteristics, that is, the number of fundamental physical quantities used to describe the situation. Representing the One "basic feature" Representative description of the first Functions with fundamental characteristics Indicates the first The weighting coefficients of the basic features; This indicates the tactical characteristics of the current tactic. Weighting coefficients representing tactical characteristics; This represents the dynamic adversarial characteristics in the current state. The weight coefficients represent the dynamic adversarial features.

[0113] Tactical pre-triggering conditions are characterized by the situation change gradient. If the predicted situation change gradient shows a certain change, the tactic is pre-triggered. The calculation of the situation change gradient is as follows:

[0114] .

[0115] S5. Input the local trajectory information and the state control information of the enemy target into the trained first reinforcement learning model, and use the deep deterministic policy gradient algorithm suitable for continuous action space to generate the main action command; the local trajectory information includes: state information, control quantity information, sensor information and weapon information.

[0116] S6. Input the local trajectory information, the enemy target's state control information, and the situational change gradients of both sides into the trained second reinforcement learning model, and use a deep deterministic policy gradient algorithm suitable for continuous action space to generate preloaded action instructions.

[0117] This embodiment uses a reinforcement learning algorithm to generate main action / preload action instructions based on the current state information.

[0118] The core issue addressed by reinforcement learning is how an agent can maximize its rewards in a complex and uncertain environment. For example... Figure 5 The diagram shown is a basic framework diagram of reinforcement learning. The two basic components of reinforcement learning are: agent and environment.

[0119] In addition to the two basic components of an agent and an environment, reinforcement learning generally includes four elements: policy, reward signal, value function, and model of the environment. At any given moment, the state information that the agent obtains from the environment. With reward information and perform actions When an action is applied to the environment, the state of the environment changes, transitioning to the next state. and provide feedback. .

[0120] The goal of reinforcement learning is to find a strategy that maximizes the long-term cumulative reward of the agent, i.e.:

[0121] ;

[0122] in, Regarding strategy Expectations For time steps, At time step Instant rewards received This is a discount factor used to calculate cumulative returns. In fact, it is a confidence allocation of rewards at different times. Generally, the impact of rewards on the optimal strategy decreases as time goes on. For strategy The expected value function in reinforcement learning includes the state-value function. and behavioral value functions The state value function is:

[0123] ;

[0124] in, For status information, Given the initial state information, the behavior-value function is:

[0125] ;

[0126] in, As an action variable, In the initial state The agent selects an action, and the optimal policy is then determined. The process is essentially the optimization of the aforementioned value function. Based on the above, the main action / preload action instructions are generated.

[0127] The above text explains how to maximize the agent's long-term accumulated reward. The specific instant reward function used to calculate the accumulated reward is shown below:

[0128] The determination of victory in air combat is mainly divided into two situations: ① Our missiles destroy the enemy aircraft, and we win the air combat; ② Our aircraft are destroyed by the enemy missiles, and we lose the air combat.

[0129] The reward function for wins and losses is designed as follows:

[0130] ;

[0131] Beyond-visual-range (BVR) air combat capability depends primarily on the performance of the fire control radar and medium-range air-to-air missiles, in addition to factors such as aircraft performance (e.g., rapid nose-pointing capability). Key indicators determining fire control radar performance include: radar search azimuth angle. Maximum search distance Detection probability, etc. The main indicators determining the performance of medium-range air-to-air missiles include: maximum off-axis launch angle, etc. Maximum and minimum attack range , Maximum and minimum no-escape zone distance , Non-escape cone angle In summary, the reward function is designed as follows:

[0132] 1) Distance dominance function

[0133] The probability of a missile killing a target is higher at intermediate launch distances, while it is lower and changes rapidly near the closest and furthest launch zones. Launching a missile within its inescapable zone ensures that the target aircraft cannot escape the missile attack, regardless of its evasive maneuvers. Constructing a range advantage function... as follows:

[0134] ;

[0135] in, The distance between the enemy and our side.

[0136] 2) Angle dominance function

[0137] Target azimuth Impact on the dominance function:

[0138] ① When the target enters the radar search area but not the missile attack area, the advantage index is relatively small.

[0139] ② If a missile enters the missile attack zone but not the inescapable zone, its advantage index is in the middle.

[0140] ③ If the target enters the inescapable zone and the distance is also less than a certain value, then no matter what maneuver the target makes to evade, it will not be able to escape the missile attack.

[0141] Missiles launched at this time will all kill the target with a probability of at least a certain value. Construct the azimuth advantage function. :

[0142] ;

[0143] If a missile is in the air before final guidance, the advantage function of the azimuth angle can be defined as follows:

[0144] ;

[0145] Constructing the entry angle Advantage function :

[0146] ;

[0147] The overall angular advantage is taken as the product of the two:

[0148] ;

[0149] in, These are the weighting coefficients, and This is used to adjust the ratio of the two in the product.

[0150] 3) Energy dominance function

[0151] When conducting missile attacks, it is necessary to maintain the aircraft's energy advantage over the target. The main objective is to ensure that the aircraft's missiles have a superior launch range and approach speed compared to the enemy's missiles. This requires careful consideration of altitude before entering the launch zone. and speed To occupy a tactically advantageous position. When the speeds of the aircraft and the enemy aircraft are equal, the qualitative relationship of the aircraft's missile advantage over the enemy aircraft's missile mainly includes the following three situations: 1) The aircraft's missile advantage zone, where missiles launched from this zone hit the enemy aircraft first; 2) The enemy aircraft's missile advantage zone, where missiles launched from this zone hit the aircraft first; 3) The equal opportunity zone, where missiles launched from this zone hit the target "simultaneously".

[0152] To achieve a high initial kinetic energy at launch and extend the missile's launch range, the carrier aircraft needs to rapidly increase its speed and kinetic energy before launch. Therefore, to achieve a greater advantage, the carrier aircraft generally needs to reach the speed of sound at launch. Thus, an energy index can be constructed:

[0153] ;

[0154] in: , The altitude, speed, and energy of the carrier aircraft at the time of missile launch. The advantage function can be defined as:

[0155] ;

[0156] in: and These are the energy indicators for the carrier aircraft and the target, respectively.

[0157] In summary, the comprehensive situation assessment function is constructed as the immediate reward during the reinforcement learning training process of the main decision loop and the auxiliary decision loop, as follows:

[0158] ;

[0159] in, These are the weighting coefficients, and This is used to adjust the proportion of each factor in the overall situation function.

[0160] For the main decision-making loop, real-time enemy and friendly status information, control information, friendly aircraft sensor information, and weapon information are normalized into vectors. Unlike the main decision loop, the input to the auxiliary decision loop, used as input for reinforcement learning, includes the predicted situation gradient. As the output of reinforcement learning, a deep deterministic policy gradient algorithm suitable for continuous action spaces is used to generate main action / preloaded action instructions.

[0161] The training process of the first reinforcement learning model includes:

[0162] Acquire first historical learning information; the first historical learning information includes: historical local flight track information, historical enemy target status control information, and first historical action command information;

[0163] Using the first historical learning information, a first reinforcement learning model is trained by employing a deep deterministic policy gradient algorithm with the goal of maximizing cumulative reward, and is used to generate main action instructions.

[0164] The training process of the second reinforcement learning model includes:

[0165] Acquire second historical learning information; the second historical learning information includes: historical local flight track information, historical enemy target status control information, historical situation change gradient values ​​of both sides, and second historical action command information;

[0166] Using the second historical learning information, and employing the DDPG algorithm to maximize cumulative reward, a second reinforcement learning model is trained to generate preloaded action instructions.

[0167] S7. Using the zero-space behavior method, the active action command and the preloaded action command are fused to obtain the autonomous maneuver decision command (the final control command of the UAV).

[0168] Specifically: the main task Jacobian matrix is ​​calculated based on the main action command to obtain the main task null space projection matrix; the preloaded action command is projected onto the main task null space based on the main task null space projection matrix to obtain the projection result; the projection result is fused with the main action command to obtain the autonomous maneuver decision command.

[0169] The NSB method combines the null space behavior approach to integrate active action instructions with preloaded action instructions. The core idea of ​​the NSB method is to project low-priority behaviors onto the null space of high-priority behaviors and output a single instruction. This ensures the smooth completion of high-priority behaviors while also taking into account low-priority behaviors, thus avoiding potential conflicts between different behaviors.

[0170] Assumption These are coordinates in an inertial coordinate system, defining the position of the agent as... The corresponding linear velocity is . For controllable task variables, the corresponding function is:

[0171] ;

[0172] ;

[0173] but:

[0174] ;

[0175] in: It is a Jacobian matrix. For task functions, For the speed of the intelligent agent, Let be the derivative of position with respect to time. The above equation can be solved using the Lagrange multiplier method, therefore the above equation can be written as:

[0176] ;

[0177] in, For Lagrange multipliers, For task variables The first derivative with respect to time, for the above equation and Find the partial derivative; the point where the derivative is 0 is the extreme point.

[0178] ;

[0179] Solving for the given information yields:

[0180] ;

[0181] In the formula: For Jacobian matrices, This is the transpose of the Jacobian matrix. Let be the pseudo-inverse of the Jacobian matrix. If there are multiple actions, the priority sequence of the actions is: NSB is Level of behavior Level behavior projection, the obtained behavior vector value and Level behavior and again Projecting the level behavior onto the null space yields the new task output. Following this pattern, the recursive formula is:

[0182] ;

[0183] In the formula For behavior The null space. Taking three sub-behaviors as an example, the principle of NSB is as follows: Figure 6 As shown.

[0184] In this embodiment, the main action instruction (main task control variable) and the preloaded action instruction (auxiliary task control variable) are known. The task space of the main task instruction must be strictly satisfied, and the preloaded action instruction is superimposed on its null space projection to avoid conflicts. The steps are as follows: First, construct the null space projection matrix of the main action instruction. Secondly, the preloaded action instructions are projected onto the null space of the main action instructions, and finally the final control quantity is fused and output.

[0185] Control quantity of active action command The control quantity of the preloaded action instruction is Now calculate the Jacobian matrix of the active action instruction:

[0186] ;

[0187] ;

[0188] in, This represents the null projection matrix of the main task. For control vectors, The main task function, It is the identity matrix. The Jacobian matrix of the main task function. Jacobian matrix The false rebellion.

[0189] Project the preloaded action instructions into the null space of the main action instructions:

[0190] ;

[0191] The final control output (autonomous maneuver decision command) is as follows:

[0192] ;

[0193] After obtaining the final control input, the UAV's state for the next moment is updated by solving the UAV's kinematic and dynamic equations, thereby enabling the UAV's dynamic maneuvers. The updated state information is then used as input for the state information at the next moment, and this process is repeated to complete the UAV's maneuver decision-making process.

[0194] Compared with existing technologies, this embodiment has the following beneficial effects: This embodiment provides a situation-driven, beyond-visual-range (BVR) air combat dual-loop coupled autonomous maneuver decision-making method. The BVR air combat dual-loop coupled decision architecture designed in this embodiment significantly reduces the impact of sudden situations, avoids control instability and instantaneous situational deterioration caused by "cliff-like" decision switching, and improves survivability and continuous combat capability in dynamic and uncertain environments. The situation-driven tactical pre-triggering mechanism established in this embodiment utilizes multi-step prediction and situational change gradients to endow the decision system with strong foresight, realizing a leap from passive reaction to proactive planning. Facing dynamic adversarial game scenarios in BVR air combat, this algorithm can effectively solve the problem of emergency state switching caused by sudden situations under incomplete information. Compared with existing single-loop reinforcement learning and reactive decision-making methods, this embodiment shows significant advantages in tactical switching smoothness, situational advantage maintenance rate, and combat effectiveness under incomplete information conditions.

[0195] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method.

[0196] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0197] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0198] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0199] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0201] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0202] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0203] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0204] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method, characterized in that, include: Acquire status control information of enemy targets and the local unit; The status control information includes: status information and control quantity information; Using a trained LSTM model, recursive prediction is performed based on the enemy target's state control information to obtain multi-step trajectory prediction information of the enemy target; the multi-step trajectory prediction information includes predicted state control information for several time steps. Using the trained LSTM model, recursive prediction is performed based on the local state control information to obtain the local multi-step trajectory prediction information; Based on the multi-step trajectory prediction information of the enemy target and the multi-step trajectory prediction information of the machine, calculate the gradient of the situational change of both sides; The system inputs the local trajectory information and the state control information of the enemy target into the trained first reinforcement learning model, and uses a deep deterministic policy gradient algorithm suitable for continuous action space to generate the main action command; the local trajectory information includes: state information, control information, sensor information and weapon information; The machine's trajectory information, the enemy target's state control information, and the situation change gradients of both sides are input into the trained second reinforcement learning model, and a deep deterministic policy gradient algorithm suitable for continuous action space is used to generate preloaded action instructions. By using the zero-space behavior method, the active action command and the preloaded action command are fused together to obtain the autonomous maneuver decision command; The step of calculating the situational change gradient of both sides based on the multi-step trajectory prediction information of the enemy target and the multi-step trajectory prediction information of the local machine specifically includes: Based on the enemy target and the predicted state control information of the machine at each time step, the situational value for the corresponding time step is calculated, resulting in situational values ​​for several time steps. The formula is as follows: Where S represents the current situation function, This represents a function that describes basic characteristics such as relative distance, angle, and velocity. This represents the total number of basic features. Representing the One "basic feature" Representative description of the first Functions with fundamental characteristics Indicates the first The weighting coefficients of the basic features; This indicates the tactical characteristics of the current tactic. Weighting coefficients representing tactical characteristics; This represents the dynamic adversarial characteristics in the current state. Weight coefficients representing dynamic adversarial features; The gradient of the situational changes for both sides is calculated based on the situational values ​​at several time steps, using the following formula: , where t represents time; The method of using zero-space behavior to fuse the active action command with the pre-loaded action command to obtain autonomous maneuver decision commands specifically includes: The main task Jacobian matrix is ​​calculated based on the main action instruction to obtain the main task null projection matrix; Based on the main task null space projection matrix, the preloaded action instructions are projected onto the main task null space to obtain the projection result; The projection results are fused with the main action command to obtain the autonomous maneuver decision command; The formula for calculating the main task Jacobian matrix based on the main action instruction is as follows: ； ； in, This represents the null projection matrix of the main task. For control vectors, The main task function, It is the identity matrix. The Jacobian matrix of the main task function. Jacobian matrix The false rebellion.

2. The situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method according to claim 1, characterized in that, The formula for projecting the preloaded action instructions into the main task null space based on the main task null space projection matrix to obtain the projection result is as follows: ； in, Indicates the projection result. This represents the null projection matrix of the main task. u 2 indicates a preload action instruction.

3. The situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method according to claim 1, characterized in that, The acquisition of the status control information of the enemy target and the local machine specifically includes: Obtain the status information of enemy targets; Based on the enemy target's state information, the control quantity information of the enemy target is calculated using the inverse control method. Obtain the status control information of the local machine.

4. The situation-driven, beyond-visual-range air combat dual-loop coupled autonomous maneuver decision-making method according to claim 1, characterized in that, The training process of the LSTM model includes: Acquire historical state control information; the historical state control information includes: the state control information of the previous historical moment and the state control information of the next historical moment; The historical state control information is used to train an LSTM network to obtain an LSTM model; The training process of the first reinforcement learning model includes: Acquire first historical learning information; the first historical learning information includes: historical local flight track information, historical enemy target status control information, and first historical action command information; Using the first historical learning information, a first reinforcement learning model is trained by employing a deep deterministic policy gradient algorithm with the goal of maximizing cumulative reward; the first reinforcement learning model is used to generate active action commands. The training process of the second reinforcement learning model includes: Acquire second historical learning information; the second historical learning information includes: historical local flight track information, historical enemy target status control information, historical situation change gradient values ​​of both sides, and second historical action command information; Using the second historical learning information, a second reinforcement learning model is trained by employing a deep deterministic policy gradient algorithm with the goal of maximizing cumulative reward; the second reinforcement learning model is used to generate preloaded action instructions.

5. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the situation-driven beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the situation-driven, beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method as described in any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the situation-driven, beyond-visual-range air combat dual-ring coupled autonomous maneuver decision-making method as described in any one of claims 1-4.

Images