A reinforcement learning-based intelligent cooperative adversarial decision-making method and system for aircraft

Abstract

This invention discloses a reinforcement learning-based intelligent cooperative adversarial decision-making method and system for aircraft. The method includes: observation design: simulation modeling of the flight dynamics of different aircraft types, as well as weapons and radar; action space design: designing the action space for each aircraft, including the target aircraft number and command values ​​issued in four headings, where the command action values ​​include angle of attack, roll angle, and throttle; reward function design: designing inventory rewards, distance rewards / penalties, and radar lock-on terms; reinforcement learning environment design: using training and application modes for dynamic control of the aircraft and the adversary, and implementing data interface functions for state, action, and reward values. This invention customizes an intelligent cooperative adversarial decision-making system for aircraft, with a reasonable objective function. After a certain period of training, it exhibits significant effects, ensuring the effectiveness of the model and algorithm, and can be used to formulate appropriate adversarial strategies for aircraft.

Description

Technical Field

[0001] This invention relates to the field of intelligent cooperative combat technology for aircraft, and in particular to a decision-making method and system for intelligent cooperative combat based on reinforcement learning. Background Technology

[0002] With the continuous advancement of military technology, low-altitude defense systems are gradually developing towards integrated defense encompassing air, sea, submarine, and land routes. This means that future air combat will no longer be limited to single-aircraft operations, but will gradually transform into confrontations between systems and frameworks. Against this backdrop, the concepts and tactics of multi-aircraft cooperative operations are constantly evolving, and cooperative task allocation technology has become an increasingly important research topic. Task allocation refers to establishing an allocation scheme that maximizes overall operational effectiveness based on mission requirements, battlefield environment, target configuration, and other information, while meeting certain constraints. A suitable task allocation scheme plays a crucial role in multi-aircraft cooperative operations.

[0003] This system enables customized combat scenarios involving clusters of up to four aircraft on each side, with the specific aircraft types allowed for specification. To achieve realistic simulation results in a distributed real-time simulation system, it's often necessary not only to perform real-time calculations on various data models within the system but also to utilize a deterministic network with extremely low latency to transmit data between subsystems, ensuring coordinated operation. Traditionally, distributed environment simulation uses a "high-speed Ethernet + upper and lower level computer" solution to meet these requirements. However, due to limitations of the TCP / IP protocol, traditional Ethernet cannot meet the demands for real-time, deterministic data transmission between real-time simulation subsystems. While measures such as increasing network speed and reducing network load can reduce latency, it's difficult to fundamentally overcome the inherent limitations of Ethernet in terms of real-time performance and determinism, and doing so also incurs additional costs.

[0004] The JSBSim model is a general-purpose flight dynamics model developed abroad. It can simulate various aircraft types, offers high real-time performance, and meets the flight dynamics model requirements of this project. JSBSim is an open-source, cross-platform, six-degree-of-freedom nonlinear flight dynamics model. It is written in object-oriented C++ and supports dynamic modeling of different types of aerospace vehicles. The dynamic characteristics of the aircraft in this model are expressed using Extensible Markup Language, allowing users to build their own six-degree-of-freedom aircraft models and perform simulations without compiling and linking the code.

[0005] The MAPPO algorithm is a variant of the PPO algorithm applied to multi-agent tasks, and it also employs the ActorCritic architecture. The difference lies in the Actor part, where a generalized advantage estimation function is used instead to further reduce the variance of the advantage function. By estimating the advantage function using a method similar to TD (Targeting and Detecting), its variance and bias are balanced, significantly reducing the evaluation variance even with a certain degree of bias. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a reinforcement learning-based intelligent cooperative adversarial decision-making method and system for aircraft, ensuring the effectiveness of the model and algorithm, and enabling aircraft to formulate appropriate adversarial strategies. The specific solution is as follows:

[0007] A reinforcement learning-based intelligent cooperative adversarial decision-making method for aircraft includes the following steps:

[0008] Step 1: Observation design: Simulation modeling of different aircraft types, their weapons, and radars based on flight dynamics; the information acquired by the aircraft includes: its own position information, the relative position relationship between itself and the enemy aircraft, the speed of its own aircraft, and the speed difference between its own aircraft and the enemy aircraft.

[0009] Step 2: Maneuver Space Design: Design the maneuver space for each aircraft, including the target aircraft number and the command values ​​for the four headings. The command values ​​include angle of attack, roll angle, and throttle, written in the following form:

[0010]

[0011] in, This indicates the number of the target aircraft selected by the aircraft. These represent the command values ​​made by the agent in the four track dimensions, respectively;

[0012] Step 3: Reward Function Design: Design survival reward, distance reward / penalty, and radar lock-on parameters. The reward function is written in the following form:

[0013]

[0014] in, For the survival reward portion, For distance-based rewards / penalties, For radar lock-on; , and The proportions of each part;

[0015] Step 4: Reinforcement learning environment design: Employ training and application modes for dynamic control of the aircraft and the adversary, and implement data interface functions for status, actions, and reward values;

[0016] Training rounds consist of training steps, and each training round contains a finite number of training steps.

[0017] The agent uses state information as input to a deep neural network, and generates actions after processing.

[0018] After the actions are formatted, they are converted into executable commands for the aircraft and sent to the environment.

[0019] Furthermore, radar simulation modeling is used to simulate air interception and air combat within the air / air functionality, specifically including:

[0020] Step 1.1: Radar Data Processing Modeling

[0021] Step a: Perform data preprocessing on the target detection point information;

[0022] Step b: Enter the track management module, determine if the track information is a real new target, and then open a new track; if the track can be associated with existing track information at this time, it becomes a stable moving track.

[0023] Step c: Convert the range, azimuth, and elevation angle information of the target in the spherical coordinate system to the three-dimensional position coordinates in the rectangular coordinate system, thereby performing filtering and prediction, and sending the filtering results to the output interface;

[0024] Step d: If the target is lost for a period of time, the flight path is considered terminated and the interface is cleared;

[0025] Step 1.2: Data Preprocessing

[0026] The transmitter module, receiver module, and target processing module in the overall design of the radar system are combined and simplified. For beam transmission and reception, the theoretical maximum detectable range is determined by the radar system's scanning range, transmission power, target distance, and the target's radar cross-section, under the conditions of no electronic interference, clear weather, and the target falling in a clutter-free area.

[0027] Furthermore, the track management module performs track association in the following specific ways:

[0028] Calculate whether the currently detected target point falls within the set range centered on the point previously predicted to be at this moment on the existing track:

[0029] 1) If the target detection module detects a target point, and the detected target point fails to be associated with an established track, it is identified as a new target. If the radar can successfully associate the target point twice in a row, the track will start.

[0030] 2) If the target detection module detects target information, but the quasi-target information fails to be associated with the established track, and in the following period of time, the target points detected by the airborne radar fail to be associated with the previously established track, then it is judged as a false alarm and the track is terminated.

[0031] 3) If the target detection module does not detect the target point, the track association fails, and a small search event is scheduled to be performed at the next moment. If the radar does not detect the lost target during the small search, the track is terminated.

[0032] 4) If the target detection module detects a target point and the detected target point is successfully associated with the established track in terms of distance, azimuth, and pitch in three dimensions, then the target point is determined to be a new observation point for the track, and the track needs to be maintained.

[0033] A reinforcement learning-based intelligent cooperative adversarial decision-making system for aircraft, comprising an upper-layer architecture and a lower-layer architecture;

[0034] The upper-level architecture includes a command and control simulation node, a tactical command simulation node, a battlefield environment management node, a tactical simulation node, and a tactical simulator. The simulation nodes communicate and interact with each other through the DIS network.

[0035] The lower-level architecture is located within a single tactical simulator, and connects the fire control calculation module, flight control calculation module, visual scene calculation module, visual scene display module, instrument calculation module, multi-functional display module, and equipment control and acquisition module within the simulator through a hybrid real-time communication network.

[0036] Furthermore,

[0037] 1) The aforementioned control simulation node is the management and monitoring center of the entire system, used to coordinate and control the operation of the entire system, monitor the system status, and record and replay data for evaluation; it interacts with other simulation nodes through the DIS network, including issuing commands, querying status, and transmitting data.

[0038] 2) The tactical command simulation node is responsible for the command and coordination of the aircraft, and is used to realize communication and cooperation between the aircraft, ensure team cooperation and achieve the designated mission objectives; it receives instructions from the guidance and control simulation node through the DIS network, sends instructions to the battlefield environment management node and the tactical simulation node, and receives data from the aircraft to update the status.

[0039] 3) The battlefield environment management node builds a simulated air combat environment by establishing a JSBSim dynamic model. It is responsible for managing and monitoring the entire battlefield environment, which is used to realize environmental modeling and simulation, as well as to locate and track the position of aircraft on the battlefield. It receives instructions from the guidance and control simulation node through the DIS network, updates the battlefield environment information, and sends it to the tactical simulation node.

[0040] 4) The tactical simulation node is responsible for tactical simulation and planning. It is used to collect and analyze information from other nodes, formulate tactical strategies and plan routes. It receives information from battlefield environment management nodes and tactical command simulation nodes through the DIS network, analyzes this information and generates corresponding action plans.

[0041] 5) The tactical simulator is responsible for simulating the behavior of the aircraft, and is used to predict the behavior and performance of the aircraft through simulation in order to guide the aircraft's actions; it receives information from the battlefield environment management node and the tactical simulation node through the DIS network, and simulates the behavior of the aircraft based on this information.

[0042] Furthermore,

[0043] 1) The fire control calculation module is responsible for calculating the fire control data of the aircraft, including the missile launch azimuth and elevation angles, target distance, and trajectory correction; this module receives data from the flight control calculation module and the visual scene calculation module, generates corresponding fire control data through calculation, and sends it to the flight control calculation module;

[0044] 2) The flight control calculation module is responsible for calculating the flight control data of the aircraft, including flight speed, altitude and attitude; this module receives data from the fire control calculation module, the visual calculation module and the instrument calculation module, generates the corresponding flight control data through calculation, and sends it to the equipment control and acquisition module;

[0045] 3) The visual scene calculation module is responsible for calculating the scene rendering of the aircraft. This module receives data from the fire control calculation module, the flight control calculation module and the equipment control and acquisition module, generates corresponding image data through calculation, and sends it to the visual scene display module.

[0046] 4) The visual display module is responsible for displaying the visual data generated by the visual resolution module in the form of images. This module receives data from the visual resolution module and renders it into a visualized image.

[0047] 5) The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, altitude and attitude; this module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to the multi-function display module;

[0048] 6) The multi-functional display module is responsible for displaying the instrument data generated by the instrument calculation module, as well as other aircraft-related data, including fire control data, mission information, and battery status; this module receives data from the instrument calculation module and the equipment control and acquisition module and renders it into visualized information;

[0049] 7) The device control and acquisition module is responsible for communicating with and acquiring data from various devices on the aircraft.

[0050] Furthermore, within the aforementioned upper and lower architectures, the following four technologies are organically integrated to form a simulation architecture based on HLA and hybrid real-time networks:

[0051] 1) Utilize DIS distributed management, time-based advancement mechanism, and load balancing control technology

[0052] In the upper-level architecture of the system, distributed management and data communication are realized through the DIS network, enabling efficient collaboration among simulation nodes. At the same time, the system uses a time-progression mechanism to ensure the accuracy and synchronization of simulation results, and uses load balancing control technology to ensure the stability and reliability of the system.

[0053] 2) Utilizing the high real-time performance and deterministic latency of reflective memory networks

[0054] In the lower-level architecture of the system, the reflective memory network technology is used to achieve high real-time performance and deterministic latency, enabling the modules to quickly and accurately interact and collaborate.

[0055] 3) Utilizing RTX's precise clock and preemptive task scheduling mechanism

[0056] In the lower-level architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system enable the system to perform fine-grained control and scheduling of tasks, thereby ensuring the efficiency and accuracy of data interaction and collaborative work between modules.

[0057] 4) Data communication mechanism using CAN bus

[0058] In the lower-level architecture of the system, the data communication mechanism of the CAN bus is used to realize efficient data transmission and communication between modules, thereby ensuring the stability and reliability of the system.

[0059] Furthermore, the basic features of the JSBSim dynamic model include: wingspan, chord length, wing area, pilot's eye position, aerodynamic reference point, center of gravity position, moment of inertia, product of inertia, front and main landing gear touchdown points, engine thrust lines, and landing gear model.

[0060] This invention is based on reinforcement learning and customizes an intelligent cooperative adversarial decision-making system for aircraft. The objective function is reasonable, and it has significant effects after a certain amount of training. It can ensure the effectiveness of the model and algorithm and can be used to formulate appropriate adversarial strategies for aircraft. Attached Figure Description

[0061] Figure 1 This is the basic architecture of the aircraft tactical confrontation simulation system in this invention. Detailed Implementation

[0062] To explain the features and technical content of the present invention in more detail, the present invention will be described below with reference to the accompanying drawings. The features and technical content described herein are for illustrative and explanatory purposes only and are not intended to limit the present invention. Those skilled in the art can modify the technical solutions of the foregoing examples according to their applications, but such modifications do not prevent the essence of the technical solution from remaining within the scope of the present disclosure.

[0063] like Figure 1 As shown, this invention provides an intelligent cooperative adversarial decision-making system for aircraft based on reinforcement learning, comprising:

[0064] 1. Upper-level architecture: including command and control simulation nodes, tactical command simulation nodes, battlefield environment management nodes, tactical simulation nodes, tactical simulators, etc. These simulation nodes communicate and interact with each other through the DIS network.

[0065] (1) Command and Control Simulation Node: The command and control simulation node is the management and monitoring center of the entire system. Its main function is to coordinate and control the operation of the entire system, monitor the system status, and record and replay data for evaluation. This node interacts with other simulation nodes through the DIS network, including issuing commands, querying status, and transmitting data.

[0066] (2) Tactical Command Simulation Node: The tactical command simulation node is responsible for the command and coordination of the aircraft. Its main function is to realize communication and cooperation between aircraft, ensure teamwork and achieve the designated mission objectives. This node receives instructions from the guidance and control simulation node through the DIS network, sends instructions to the battlefield environment management node and the tactical simulation node, and receives data from the aircraft to update the status.

[0067] (3) Battlefield Environment Management Node: Establishes a JSBSim dynamic model to build a simulated air combat environment, responsible for managing and monitoring the entire battlefield environment. Its main functions are to realize environmental modeling and simulation, as well as to locate and track the position of aircraft on the battlefield. This node receives instructions from the command and control simulation node through the DIS network, updates the battlefield environment information, and sends it to the tactical simulation node.

[0068] (4) Tactical Simulation Node: The tactical simulation node is responsible for tactical simulation and planning. Its main function is to collect and analyze information from other nodes, formulate tactical strategies, and plan flight paths. This node receives information from the battlefield environment management node and the tactical command simulation node through the DIS network, analyzes this information, and generates corresponding action plans.

[0069] (5) Tactical Simulator: The tactical simulator node is responsible for simulating the behavior of aircraft. Its main function is to predict the behavior and performance of aircraft through simulation in order to better guide the actions of aircraft. This node receives information from the battlefield environment management node and the tactical simulation node through the DIS network and simulates the behavior of aircraft based on this information.

[0070] 2. Lower-level architecture: Located within a single tactical simulator, the fire control calculation module, flight control calculation module, visual scene calculation module, visual scene display module, and instrument calculation module within the simulator are connected using the hybrid real-time communication network proposed in this paper.

[0071] (1) Fire control calculation module: The fire control calculation module is responsible for calculating the fire control data of the aircraft, including missile launch azimuth and elevation angles, target distance, trajectory correction, etc. This module receives data from the flight control calculation module and the visual scene calculation module, generates the corresponding fire control data through calculation, and sends it to the flight control calculation module.

[0072] (2) Flight control calculation module: The flight control calculation module is responsible for calculating the flight control data of the aircraft, including flight speed, altitude, attitude, etc. This module receives data from the fire control calculation module, the visual scene calculation module and the instrument calculation module, generates the corresponding flight control data through calculation, and sends it to the equipment control and acquisition module.

[0073] (3) Visual Scene Calculation Module: The visual scene calculation module is responsible for calculating the scene rendering of the aircraft. This module receives data from the fire control calculation module, the flight control calculation module, and the equipment control and acquisition module, generates corresponding image data through calculation, and sends it to the visual scene display module.

[0074] (4) Visual Display Module: The visual display module is responsible for displaying the visual data generated by the visual resolution module in the form of images. This module receives data from the visual resolution module and renders it into a visualized image.

[0075] (5) Instrument Calculation Module: The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, altitude, attitude, etc. This module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to the multi-function display module.

[0076] (6) Multifunctional Display Module: The multifunctional display module is responsible for displaying instrument data generated by the instrument calculation module, as well as other aircraft-related data, including fire control data, mission information, battery status, etc. This module receives data from the instrument calculation module and the equipment control and acquisition module and renders it into visualized information.

[0077] (7) Equipment control and data acquisition module: The equipment control and data acquisition module is responsible for communicating with and acquiring data from various devices on the aircraft.

[0078] 3. Construction of the simulated air combat environment: JSBSim, an open-source, cross-platform six-degree-of-freedom nonlinear flight dynamics model, is mainly used.

[0079] The JSBSim dynamic model is established, and the main contents are as follows:

[0080] Basic characteristics include: wingspan, chord length, wing area, pilot's eye position, aerodynamic reference point, center of gravity position, moment of inertia, product of inertia, front and main landing gear touchdown points, engine thrust lines, and landing gear model.

[0081] Flight control scheme: Since the aircraft control surface state is directly operated end-to-end by the intelligent agent in this invention, no stability augmentation design is adopted.

[0082] This invention relates to a reinforcement learning-based intelligent cooperative adversarial decision-making method for aircraft, as detailed below:

[0083] Step 1: Observation design: Simulation modeling of different aircraft types, their weapons, and radars based on flight dynamics; the information acquired by the aircraft includes: its own position information, the relative position relationship between itself and the enemy aircraft, the speed of its own aircraft, and the speed difference between its own aircraft and the enemy aircraft.

[0084] Step 2: Maneuver Space Design: Design the maneuver space for each aircraft, including the target aircraft number and the command values ​​for the four headings. The command values ​​include angle of attack, roll angle, and throttle.

[0085] It can be written in the following form:

[0086]

[0087] in, This indicates the number of the target aircraft selected by the aircraft. These represent the command values ​​made by the agent in the four track dimensions.

[0088] Step 3: Reward Function Design: Design survival rewards, distance rewards / penalties, and radar lock-on parameters;

[0089] The reward function can be written in the following form:

[0090]

[0091] in, For the survival reward portion, For distance-based rewards / penalties, This is a radar lock-on item. , , This represents the proportions of each part.

[0092] Step 4: Reinforcement learning environment design: Use training mode and application mode to dynamically control the aircraft and the adversary, and realize the data interface function of status, action and reward value.

[0093] The training round consists of a training step size, and each training round contains a finite training step size. The agent uses the state information as input to the deep neural network, and generates actions after processing. After the actions are formatted, they form instructions that the aircraft can execute and send them to the environment.

[0094] In this invention, since the agent needs to interact with the JSBSim simulation environment, the simulation is not performed in a continuous running mode. Instead, after each decision is made by the agent, a function is called to complete the step-by-step execution of the simulation.

[0095] The radar system simulation selects radar functional level simulation to design the radar system modules. For the typical air-to-air mode of airborne phased array radar, functional level system simulation of airborne phased array radar is carried out to provide a fast and accurate battlefield situational awareness for air combat intelligence in this project.

[0096] Airborne phased array radar modeling:

[0097] (1) Modeling of main working modes: The modeling of this system mainly simulates the air interception (AIC) and air combat (ACM) in the air / air function.

[0098] (2) Radar data processing modeling:

[0099] First, the target detection point information is preprocessed.

[0100] Next, the system enters the track management module. If the track information is determined to be a real new target, a new track is opened. If the track can be associated with existing track information at this time, it becomes a stable moving track.

[0101] Then, the distance, azimuth, and elevation angle information of the target in the spherical coordinate system are converted to the three-dimensional position coordinates in the rectangular coordinate system, so as to perform filtering and prediction, and the filtering results are sent to the output interface.

[0102] If the target is lost for a period of time, the flight path is considered terminated, and the interface is cleared at this time.

[0103] (3) Data preprocessing:

[0104] In this system, to improve the real-time performance of radar processing, the transmitter module, receiver module, and target processing module in the overall radar system design are merged and simplified.

[0105] For beam transmission and reception, the maximum theoretical detectable range is determined by the radar system's scanning range, transmission power, target distance, and the target's radar cross-section, under conditions of no electronic interference, clear weather, and the target falling in a clutter-free area.

[0106] Track correlation: Calculates whether the currently detected target point falls within a set range centered on the point previously predicted by an existing track. When the aircraft target performs large maneuvers, a larger threshold is selected to maintain accurate track updates.

[0107] 1) If the target detection module detects a target point, and the detected point fails to be associated with the established track, it indicates that it is a new target. If the radar can successfully associate the point with the target twice in a row, the track can be started.

[0108] 2) If the target detection module detects target information, but the quasi-target information fails to be associated with the established track, and the target points detected by the airborne radar fail to be associated with the previously established track in the following period of time, it is judged as a false alarm and the track must be terminated.

[0109] 3) If the target detection module fails to detect the target point, track association will definitely fail. When the radar performs a tracking event, track maintenance is normally required, but this time the target point needed for track maintenance is equal to the predicted result of that target point in the previous moment. A small search event is scheduled for the next moment. Without beam modeling, the small search behavior is equivalent to widening a certain threshold. When the radar fails to detect the lost target during the small search, track termination can proceed.

[0110] 4) If the target detection module detects a target point and the detected target point is successfully associated with the established track in terms of distance, azimuth, and pitch in three dimensions, then the target point is determined to be a new observation point for the track, and the track needs to be maintained.

[0111] Case study analysis of intelligent decision-making in swarm air combat at the track control level:

[0112] Observation space setting: Based on the motion calculation equations under six freedoms, with These represent pitch, yaw, and roll angles, respectively, reflecting the aircraft's attitude relative to the ground coordinate system. This represents the three-dimensional spatial coordinates of the aircraft with the ground as the reference frame. These represent the decomposition of the aircraft's flight speed in three directions.

[0113] Action Space Setting: Based on the characteristics of the trajectory control task and the design of the simulation environment, the target point's north-east-sky coordinates and target velocity are selected as the decision dimensions for continuous space decision control. Simultaneously, to narrow the search space, the decision location range is concentrated within a certain range of the target aircraft. The target aircraft's identifier is also generated via a network. The decision result is represented as a one-dimensional vector as shown in the equation. .

[0114] The following four technologies are organically integrated to form a simulation architecture based on HLA and hybrid real-time networks:

[0115] 1) Utilize DIS distributed management, time-based advancement mechanism, and load balancing control technology

[0116] In the system's upper-level architecture, distributed management and data communication are achieved through the DIS network, enabling efficient collaboration among simulation nodes. Simultaneously, the system utilizes a time-advancement mechanism to ensure the accuracy and synchronization of simulation results, and employs load balancing control technology to guarantee system stability and reliability.

[0117] 2) Utilizing the high real-time performance and deterministic latency of reflective memory networks

[0118] In the lower-level architecture of the system, reflective memory network technology is used to achieve high real-time performance and deterministic latency, enabling the modules to quickly and accurately interact and collaborate.

[0119] 3) Utilizing RTX's precise clock and preemptive task scheduling mechanism

[0120] In the lower-level architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system enable the system to perform fine-grained control and scheduling of tasks, thereby ensuring the efficiency and accuracy of data interaction and collaborative work between modules.

[0121] 4) Data communication mechanism using CAN bus

[0122] In the lower-level architecture of the system, the data communication mechanism of the CAN bus is used to realize efficient data transmission and communication between modules, thereby ensuring the stability and reliability of the system.

Claims

1. A method for intelligent cooperative confrontation decision of an aircraft based on reinforcement learning, characterized in that, Comprising the following steps: Step 1: Observation value design: based on flight dynamics, simulation modeling is carried out on different aircraft types and their weapons, radars; The information obtained by the aircraft includes: self-position information, relative position relationship between self and enemy aircraft, speed of self-aircraft, speed difference between self-aircraft and enemy aircraft; Step 2: Action space design: design the action space of each aircraft, including target aircraft number and four heading instruction values, instruction action values including angle of attack, roll angle, throttle amount, written as follows: wherein, represents the number of the target aircraft selected by the machine, respectively represent the instruction values made by the intelligent agent in the four track dimensions; Step 3: Reward function design: design survival reward, distance reward / punishment and radar locking item, the reward function is written as follows: wherein, is a survival reward part, is a distance reward / punishment term, is a radar lock term; , and are the proportions of the respective parts; Step 4: Reinforcement learning environment design: adopt training mode and application mode for dynamic control of aircraft and opponents, and realize the data interface function of state, action and reward value; A training round is composed of training steps, and each training round contains a limited number of training steps; The agent takes the state information as the input of the deep neural network, and generates the action after operation; After format conversion, the action forms an executable instruction of the aircraft, which is sent to the environment; Step 1.1: Radar data processing modeling: Step a: data preprocessing of target detection point information; Step b: enter the track management module, judge whether the point information is a real new target, and then open a new track; if the point information can be associated with the existing track information at this time, it becomes a stable moving point; Step c: convert the distance, azimuth and pitch angle information of the target in the spherical coordinate system to the three-dimensional position coordinates in the rectangular coordinate system, so as to filter and predict, and send the filtering result to the output interface; Step d: if the target is lost for a period of time, it is judged as track termination, and the interface is cleared; Step 1.2: Data preprocessing The transmitter module, receiver module and target processing module in the overall design of the radar system are combined and simplified; for the transmission and reception of the beam, the theoretical maximum detectable distance under the condition of no electronic interference, fine weather and target falling in the non-clutter area is determined by the scanning range of the radar system, transmission power, target distance and target radar scattering cross section area. The track association of the track management module is as follows:

2. The method of claim 1, wherein, Calculate whether the currently detected target point information falls within the set range of the point information predicted by the last time of the existing track: 1) If the target detection module detects the target point information, and the detected target point information fails to associate with the established track, it is determined as a new target, and when the radar can successfully associate with the target point information for two times at the same time, it enters the track initiation; 2) If the target detection module detects target information, and the target information fails to associate with the established track, and in a period of time, the target point information detected by the airborne radar fails to associate with the previously established track, it is judged as false alarm, and the track is terminated; ​ 3) If the target detection module does not detect the target track, the track association fails, and the next time is arranged to perform a small search event. If the radar does not detect the lost target during the small search, the track is terminated; 4) If the target detection module detects the target track, and the detected target track is successfully associated with the established track in distance, azimuth, and elevation, the track is judged to be a new observation point of the track, that is, the track maintenance is performed.

3. A reinforcement learning based intelligent cooperative adversarial decision system for aircrafts, characterized in that, The upper layer architecture and the lower layer architecture are included. The upper layer architecture includes a guide control simulation node, a tactical command simulation node, a battlefield environment management node, a tactical deduction node, and a tactical simulator. The simulation nodes communicate and interact with each other through a DIS network. The lower layer architecture is located in a single tactical simulator and connects the fire control calculation module, the flight control calculation module, the visual solution module, the visual display module, the instrument calculation module, the multifunctional display module, and the device control and acquisition module in the simulator through a mixed real-time communication network. 1) The guide control simulation node is the management and monitoring center of the entire system, used to coordinate and control the operation of the entire system, monitor the system state, and record and replay data for evaluation. It interacts with other simulation nodes through the DIS network, including issuing instructions, querying states, and transmitting data. 2) The tactical command simulation node is responsible for the command and coordination of the aircraft, used to realize the communication and cooperation between the aircrafts, ensure team cooperation, and achieve the specified task target. It receives instructions from the guide control simulation node through the DIS network, sends instructions to the battlefield environment management node and the tactical deduction node, and receives data from the aircraft to update the state. 3) The battlefield environment management node builds a simulated air combat environment by establishing a JSBSim dynamics model, responsible for managing and monitoring the entire battlefield environment, used for environment modeling and simulation, and positioning and tracking the positions of the aircrafts on the battlefield. It receives instructions from the guide control simulation node through the DIS network, updates the battlefield environment information, and sends it to the tactical deduction node. 4) The tactical deduction node is responsible for tactical deduction and planning, used to collect information from other nodes and analyze it, develop tactical strategies and plan routes. It receives information from the battlefield environment management node and the tactical command simulation node through the DIS network, analyzes the information, and generates corresponding action plans. 5) The tactical simulator is responsible for simulating the behavior of the aircraft, used to predict the behavior and performance of the aircraft through simulation to guide the action of the aircraft. It receives information from the battlefield environment management node and the tactical deduction node through the DIS network, and simulates the behavior of the aircraft based on the information.

4. The aircraft intelligent cooperative confrontation decision system based on reinforcement learning according to claim 3, wherein 1) The fire control calculation module is responsible for calculating the fire control data of the aircraft, including the missile launch azimuth and elevation, target distance, and trajectory correction. This module receives data from the flight control calculation module and the visual solution module, generates corresponding fire control data through calculation, and sends it to the flight control calculation module. 2) The flight control calculation module is responsible for calculating the flight control data of the aircraft, including flight speed, height and attitude; this module receives data from the fire control calculation module, the visual calculation module and the instrument calculation module, generates corresponding flight control data through calculation, and sends it to the device control and acquisition module; 3) The visual calculation module is responsible for calculating the scene rendering of the aircraft, this module receives data from the fire control calculation module, the flight control calculation module and the device control and acquisition module, generates corresponding image data through calculation, and sends it to the visual display module; 4) The visual display module is responsible for displaying the visual data generated by the visual calculation module in the form of images, this module receives data from the visual calculation module and renders it into visual images; 5) The instrument calculation module is responsible for calculating various instrument data of the aircraft, including speed, height and attitude; This module receives data from the flight control calculation module, generates corresponding instrument data through calculation, and sends it to the multifunction display module; 6) The multifunction display module is responsible for displaying the instrument data generated by the instrument calculation module and other data related to the aircraft, including fire control data, task information and battery status; This module receives data from the instrument calculation module and the device control and acquisition module, and renders it into visual information; 7) The device control and acquisition module is responsible for communication and data acquisition with various devices of the aircraft.

5. The reinforcement learning based intelligent cooperative adversarial decision system for aircrafts according to claim 3, wherein, In the upper and lower architectures, the following four technologies are organically integrated to form a simulation architecture based on HLA and hybrid real-time network: 1) Use of DIS distributed management, time advance mechanism and load balancing control technology In the upper architecture of the system, distributed management and data communication are realized through the DIS network, so that the simulation nodes can work efficiently; at the same time, the system uses the time advance mechanism to ensure the accuracy and synchronization of the simulation results, and uses the load balancing control technology to ensure the stability and reliability of the system; 2) Use of high real-time characteristics and deterministic delay of reflective memory network In the lower architecture of the system, the reflective memory network technology is used to realize high real-time and deterministic delay, so that the modules can quickly and accurately interact and cooperate with each other; 3) Use of precise clock and preemptive task scheduling mechanism of RTX In the lower architecture of the system, the precise clock and preemptive task scheduling mechanism of the RTX real-time operating system are used to enable the system to finely control and schedule tasks, thereby ensuring the efficiency and accuracy of data interaction and cooperation between modules; 4) Use of CAN bus data communication mechanism In the lower architecture of the system, the CAN bus data communication mechanism is used to realize efficient data transmission and communication between modules, thereby ensuring the stability and reliability of the system.

6. The reinforcement learning based intelligent cooperative adversarial decision system for aircrafts according to claim 3, wherein, The basic characteristics of the JSBSim dynamics model include wing span, chord length, wing area, pilot eye position, aerodynamic force reference point, center of gravity position, moment of inertia, inertia product, front and main take-off and landing point position, and engine thrust line, landing gear model.

Images