Simulation environment construction method, device and system for beyond-visual-range air combat and medium

Abstract

The invention discloses a simulation environment construction method, device and system for over-the-horizon air combat and a medium. The method comprises the steps that a high-fidelity aerodynamic model is constructed, and the aerodynamic model comprises a six-degree-of-freedom aerodynamic model of an unmanned aerial vehicle and a three-degree-of-freedom aerodynamic model of an over-the-horizon air-to-air missile; constructing a sensor model, wherein the sensor model comprises an airborne fire control radar sensor model and a missile infrared seeker sensor model; a reward function is constructed, battlefield situation is quantitatively evaluated, and a learning target is provided for reinforcement learning algorithm training; and packaging the constructed simulation environment into a standardized reinforcement learning interface to support interaction of a state space and an action space, and realizing efficient coupling of a reinforcement learning algorithm and the simulation environment through a parallel simulation acceleration and asynchronous communication mechanism. According to the method, a simulation environment for the beyond-visual-range air combat is constructed, and high-adaptability simulation technical support is provided for algorithm training and tactical verification of the intelligent beyond-visual-range air combat.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer simulation, and in particular to a method, apparatus, system and medium for constructing a simulation environment for beyond-visual-range air combat. Background Technology

[0002] In recent years, unmanned combat aircraft (UCAVs) have gradually replaced manned aircraft as the core force in future air combat due to their low cost, high survivability, and strong maneuverability. Beyond-visual-range (BVR) air combat, as the main form of modern air combat, is characterized by long engagement distances, reliance on sensor detection and long-range weapon guidance, and rapidly changing battlefield situations. Against this backdrop, reinforcement learning, with its autonomous decision-making and online optimization capabilities, is considered a key technology for realizing intelligent air combat with UAVs. However, existing simulation environments face significant technical bottlenecks in supporting reinforcement learning training, specifically in the following three aspects:

[0003] Traditional beyond-visual-range (BVR) air combat simulation platforms are mostly based on point mass models or low-degree-of-freedom dynamic designs, failing to reflect the nonlinear physical characteristics of real air combat. For example, one open-source air combat simulation tool simplifies UAVs into a two-dimensional planar motion model, ignoring aerodynamic coupling effects (such as pitch-roll coupling), leading to severe instability of the trained algorithm in real six-degree-of-freedom flight control. Furthermore, existing missile seeker models often employ deterministic hit detection (such as fixed hit probabilities) without considering the impact of atmospheric disturbances on infrared signal attenuation, resulting in a missile miss rate in simulations deviating by as much as 40% from real battlefield data. While such simplifications reduce computational costs, they create a simulation-to-reality gap when reinforcement learning algorithms are transferred to real systems.

[0004] Current sensor models in simulation environments are mostly designed for visualization training or pilot operation feedback, lacking adaptation to the input features of reinforcement learning algorithms. For example, a research team directly mapped the airborne radar output to the precise coordinates of the enemy in an open-source UAV simulation platform, ignoring uncertainties such as radar detection probability and false alarm rate. This caused the agent to rely excessively on idealized information and be unable to cope with missed detections and false alarms from real radar. Similarly, the infrared seeker models of mainstream missile simulation tools only determine target lock based on a fixed signal-to-noise ratio threshold, without modeling the dynamic changes in background radiation noise with weather conditions, causing the trained guidance strategy to fail in complex weather environments.

[0005] While commercial simulation platforms possess high-precision physics engines, their closed architecture makes efficient integration with mainstream reinforcement learning frameworks difficult. For example, a defense research project attempted to train an air combat decision-making algorithm in FLAMES (a simulation development software), but because the platform lacked a standardized state / action space interface, additional data conversion middleware needed to be developed, resulting in a 70% increase in training latency. Furthermore, existing open-source environments generally lack support for multi-agent collaboration, and their reward functions are crudely designed, failing to guide agents to learn complex tactical behaviors such as energy management and sensor-based game theory.

[0006] To address the above issues, the academic community has attempted to improve the situation through two approaches: one is to build high-fidelity vacuum combat scenarios based on game engines, but the physics engine focuses on rigid body collisions rather than aerodynamic simulation, making it difficult to meet the long-term trajectory prediction requirements of beyond-visual-range air combat; the other is to adopt a lightweight simulation framework, which is easy to integrate with reinforcement learning algorithms, but loses its tactical research value due to oversimplification of sensors and dynamic models.

[0007] In summary, existing technologies struggle to balance simulation realism, algorithm compatibility, and computational efficiency, failing to provide a dedicated simulation environment for reinforcement learning and to offer crucial technical support for the training and tactical innovation of intelligent air combat algorithms. Consequently, they severely hinder the engineering application of intelligent air combat algorithms. Summary of the Invention

[0008] This invention aims to at least partially solve the technical problems in related technologies. Therefore, the first objective of this invention is to provide a method for constructing a simulation environment for beyond-visual-range (BVR) air combat. Through high-fidelity aerodynamic modeling, sensor simulation with dynamic environmental disturbance injection, reward function design for battlefield situation quantification, and standardized reinforcement learning interface encapsulation, this method supports reinforcement learning training requirements while ensuring simulation accuracy and improving computational efficiency. It constructs the first dedicated reinforcement learning simulation environment for BVR air combat, providing key technical support for the training of intelligent air combat algorithms and tactical innovation.

[0009] The second objective of this invention is to provide a simulation environment construction device for beyond-visual-range air combat.

[0010] A third objective of this invention is to provide a computer-readable storage medium.

[0011] The fourth objective of this invention is to provide a simulation environment construction system for beyond-visual-range air combat.

[0012] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0013] A method for constructing a simulation environment for beyond-visual-range air combat includes:

[0014] A high-fidelity aerodynamic model is constructed, which includes a six-degree-of-freedom aerodynamic model of the UAV and a three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile;

[0015] Construct sensor models, including sensor models of airborne fire control radar and missile infrared seeker;

[0016] Construct a reward function to quantitatively assess the battlefield situation and provide a learning objective for training reinforcement learning algorithms;

[0017] The constructed simulation environment is encapsulated as a standardized reinforcement learning interface to support the interaction between the state space and the action space, and efficient coupling between the reinforcement learning algorithm and the simulation environment is achieved through parallel simulation acceleration and asynchronous communication mechanisms.

[0018] Preferably, when constructing a high-fidelity aerodynamic model, the aerodynamic forces, torques, and propulsion system characteristics are calculated to accurately simulate the attitude, speed, and maneuverability of the aircraft, ensuring that the physical behavior in the simulation environment conforms to the real air combat scenario.

[0019] Preferably, the sensor model of the airborne fire control radar is constructed with the goal of simulating detection range, azimuth resolution and multi-target tracking capability, and environmental disturbances are introduced into the design. The environmental disturbances include atmospheric attenuation and clutter noise. Based on the target thermal radiation characteristics, the sensor model of the missile infrared seeker is constructed by combining atmospheric transmission loss and background radiation noise. In this way, environmental parameters are injected to enhance battlefield uncertainty.

[0020] Preferably, the reward function is characterized by including a state reward mechanism and a state transition reward mechanism, wherein the state reward mechanism guides the UAV swarm to optimize its battlefield perception and guidance coordination capabilities through progressive incentives; and the state transition reward mechanism guides the UAV swarm to learn tactical strategies that are both aggressive and disciplined through differentiated reward and punishment rules.

[0021] Preferably, the state reward mechanism is used to quantify the incremental tactical advantage gained by our drone swarm under a specific battlefield situation, and the state reward mechanism includes the following two core components:

[0022] Fire control radar detection rewards provide positive rewards when the UAV effectively detects and locks onto enemy targets;

[0023] Missile guidance rewards provide positive incentives when UAVs provide mid-course guidance support for missiles launched by our side;

[0024] The state transition reward mechanism is used to evaluate the change in advantage gained by our drone swarm during changes in battlefield state. The state transition reward mechanism includes the following three key parts:

[0025] The aircraft crash reward reflects the battle loss exchange ratio. A negative reward is applied when our drone is shot down, and a positive reward is given when the enemy target is destroyed.

[0026] Missile launch rewards are used to regulate combat behavior. Launching a missile without meeting mid-course guidance conditions will result in a negative reward, while launches that meet tactical coordination requirements will receive a positive reward.

[0027] Missile failure rewards are used to constrain the waste of ammunition; a negative reward is generated when a missile misses its target and crashes.

[0028] Preferably, the standardized reinforcement learning interface includes at least an environment initialization interface, a state interaction interface, an observation space definition interface, an action space definition interface, a battlefield situation visualization interface, and a random seed control interface.

[0029] Preferably, the state space is the set of all possible states in the simulation environment, including the fighter jet's own state, sensor data, and battlefield situation; the action space is the set of all possible actions that the UAV swarm can perform under a given state.

[0030] To achieve the above objectives, a second aspect of the present invention provides a device for constructing a simulation environment for beyond-visual-range air combat, comprising:

[0031] The first building unit is used to build a high-fidelity aerodynamic model, which includes a six-degree-of-freedom aerodynamic model of an unmanned aerial vehicle and a three-degree-of-freedom aerodynamic model of a beyond-visual-range air-to-air missile.

[0032] The second building unit is used to build sensor models, which include sensor models of airborne fire control radar and missile infrared seeker.

[0033] The third building unit is used to construct the reward function, quantitatively evaluate the battlefield situation, and provide learning objectives for the training of reinforcement learning algorithms.

[0034] The encapsulation unit is used to encapsulate the constructed simulation environment into a standardized reinforcement learning interface to support the interaction between the state space and the action space, and to achieve efficient coupling between the reinforcement learning algorithm and the simulation environment through parallel simulation acceleration and asynchronous communication mechanisms.

[0035] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described above.

[0036] To achieve the above objectives, a fourth aspect of the present invention provides a simulation environment construction system for beyond-visual-range air combat, including the device described above or the computer-readable storage medium described above.

[0037] This invention has at least the following technical effects:

[0038] This invention, through collaborative modeling of a high-precision aircraft dynamics model and a highly realistic sensor system, achieves, for the first time, end-to-end simulation from the physical level to the tactical decision-making level. It overcomes the simulation-to-reality gap caused by traditional simplified models and solves the compatibility issues between commercial simulation platforms and reinforcement learning frameworks. In particular, the design of quantifying the battlefield situation into multi-dimensional reward signals provides a realistic, stable, and efficient training environment for reinforcement learning algorithms, enabling the agent—the autonomous decision-making entity trained through reinforcement learning—to learn near-real-world beyond-visual-range air combat tactical decision-making capabilities. This technical solution fills the gap in current intelligent air combat algorithm development due to the lack of dedicated simulation support, providing key technical support for the rapid development of future unmanned air combat systems.

[0039] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0040] Figure 1 This is a flowchart of a simulation environment construction method for beyond-visual-range air combat according to an embodiment of the present invention.

[0041] Figure 2 This is a structural block diagram of a simulation environment construction device for beyond-visual-range air combat according to an embodiment of the present invention. Detailed Implementation

[0042] The following describes this embodiment in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0043] The following description, with reference to the accompanying drawings, outlines a method, apparatus, system, and medium for constructing a simulation environment for beyond-visual-range air combat according to this embodiment.

[0044] Figure 1 This is a flowchart illustrating a method for constructing a simulation environment for beyond-visual-range air combat, according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0045] Step S1: Construct a high-fidelity aerodynamic model, which includes a six-degree-of-freedom aerodynamic model of the UAV and a three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile.

[0046] In this implementation, the spatial motion characteristics of an aircraft, including velocity variations, trajectory characteristics, and basic maneuverability, can be realistically reproduced by accurately calculating aerodynamic properties, propulsion system output, and flight control response, providing a realistic physical basis for tactical combat. Specifically, by calculating aerodynamic forces, torques, and propulsion system characteristics, the attitude, velocity, and maneuverability of an aircraft can be accurately simulated, ensuring that the physical behavior in the simulation environment conforms to real air combat scenarios.

[0047] In this embodiment, a high-precision aircraft dynamics simulation system can be constructed by building a six-degree-of-freedom dynamics model of an unmanned aerial vehicle and a three-degree-of-freedom aerodynamics model of a beyond-visual-range air-to-air missile.

[0048] The six-degree-of-freedom dynamics model of the UAV utilizes the high-fidelity flight dynamics engine provided by JSBSim (an open-source high-precision flight dynamics model engine). The dynamic response of the UAV is driven by aileron deflection, elevator deflection, rudder deflection and throttle change as control inputs.

[0049] The three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile was implemented using Python (a programming language). In the inertial coordinate system, the missile's kinematic equations are:

[0050]

[0051] In the formula, x(t), y(t), and z(t) are the three-dimensional coordinates of the missile in the inertial coordinate system at time t, respectively. Let v(t), θ(t), and θ(t) be the derivatives corresponding to the three-dimensional coordinates, respectively. These are the missile's velocity, pitch angle, and yaw angle, respectively, all functions of flight time t. The missile's flight is divided into an active phase and a passive phase. In an inertial frame, the forces acting on the missile during the active phase mainly include: missile thrust T(t), gravity G = m(t)g, and aerodynamic drag D(t). Therefore, in the ballistic coordinate system, the missile's particle dynamics equation is:

[0052]

[0053] in, For overload in the velocity direction, n y (t), n z (t) represents the lateral control overloads of the missile in the yaw and pitch directions at time t, respectively, calculated using the proportional guidance method. m(t) is the current mass of the missile, and g is the gravitational acceleration constant, taken as g = 9.81 m / s². 2 .

[0054] T(t) is the missile thrust, and its magnitude satisfies:

[0055]

[0056] In the formula, I sg For specific impulse (assuming it to be constant), The combustion mass is the combustion rate.

[0057] D(t) is the aerodynamic drag, and its magnitude satisfies:

[0058]

[0059] In the formula, v(t) is the velocity of the missile at time t, z(t) is the horizontal height of the missile at time t, S is the calculated area (defined as the cross-sectional area of ​​the missile body orthogonal to the velocity), and c D (v(t)) is the drag coefficient, a function of the velocity magnitude v(t), and can be approximated as a constant. ρ(z(t)) is the air density, a function of the horizontal height z(t), and its magnitude satisfies the approximate relationship: ρ(z(t))=ρ0e -z(t) / k In the formula, ρ0 = 1.225 kg / m 3 k = 9300m.

[0060] For the calculated area S of the missile, its size approximately satisfies:

[0061]

[0062] In the formula, d is the missile diameter, L is the missile radius, and Δθ, The difference between the missile's attitude and trajectory in pitch and yaw angles can be approximated as the trajectory pitch angle and trajectory yaw angle θ. The change within the time increment Δt.

[0063] In this embodiment, the six-degree-of-freedom (DOF) dynamic model of the UAV comprehensively describes the aircraft's motion characteristics in three-dimensional space. It includes not only the translational degrees of freedom for position and velocity but also accurately simulates the dynamic changes of the three rotational degrees of freedom: roll, pitch, and yaw. Through the coupling effects of aerodynamic forces, thrust, gravity, and control surface deflections (ailerons, elevators, and rudders), this model fully reflects the aircraft's maneuverability and stability, making it suitable for high-precision flight simulation, control system design, and complex maneuver analysis. This embodiment, by accurately simulating nonlinear aerodynamic effects (such as pitch-roll coupling) using a six-DOF dynamic model, solves the control instability problem caused by traditional two-dimensional models.

[0064] The missile's three-degree-of-freedom aerodynamic model focuses on the trajectory of the missile's center of mass, describing its flight state through velocity magnitude, pitch angle, and yaw angle. This model simplifies attitude dynamics, directly translating control commands into overload commands to adjust the flight direction, thereby efficiently implementing guidance algorithms such as proportional guidance. This simplification significantly reduces computational complexity while meeting the requirements of missile trajectory simulation, hit accuracy analysis, and tactical-level adversarial simulation, making it suitable for large-scale battlefield simulation and rapid tactical assessment. While maintaining computational efficiency, the missile's three-degree-of-freedom model reduces the simulation error of missile miss rate by calculating the dynamic drag coefficient using formula (4) and the projected area of ​​the missile body using formula (5).

[0065] Step S2: Construct sensor models, including sensor models for airborne fire control radar and missile infrared seeker.

[0066] In this embodiment, the sensor model of the airborne fire control radar simulates the detection range, azimuth resolution, and multi-target tracking capability, and introduces environmental disturbances such as atmospheric attenuation and clutter noise; the infrared sensor model of the missile's infrared seeker is modeled based on the target's thermal radiation characteristics, combined with atmospheric transmission loss and background radiation noise. By injecting environmental parameters (such as weather and distance attenuation), the physical realism of the sensor data generation and the uncertainty of the battlefield can be enhanced.

[0067] In this embodiment, the airborne fire control radar model operates under specific detection range and operating angle parameters. Based on a standard fire control radar for advanced fighter jets, the radar has a detection range of 200 kilometers for targets with a radar cross-section (RCS) of 1 square meter, and approximately 80 kilometers for targets with an RCS of 0.01 square meters. The radar's operating angle is ±60°, and the scanning range is always horizontally expanded with the fuselage axis as the reference (the beam plane is parallel to the ground plane), with no deflection of the radar's central axis, always pointing directly forward of the aircraft's nose. This radar operates continuously without adjustment using an open-loop strategy; once a target enters the airborne fire control radar's coverage area, it is tracked and intercepted by the aircraft. This allows for the acquisition of precise target position and velocity information, enabling the aircraft to launch from any point and providing mid-course guidance for missiles.

[0068] Furthermore, addressing the common issue of look-down clutter blind spots in advanced fighter jets, if a target's radial vector velocity along its direction exceeds 50 km / h and it lies within the ground / sea clutter background of the fire control radar, the radar will be unable to acquire target information, leading to the missile's mid-course guidance failure. When the enemy fire control radar illuminates the fighter jet, a radar tracking warning will be triggered. It is worth noting that the fire control radar should switch to lock-on mode and increase antenna transmission power during missile launch; therefore, an enemy missile launch will trigger a local radar lock-on warning, which is distinct from the radar tracking warning indicating the start of enemy missile launch.

[0069] The detection range of an infrared seeker is affected by the target's radiation intensity, azimuth angle, and system sensitivity, exhibiting a non-linear attenuation characteristic. When the target is directly aligned with the seeker's axis (without azimuth angle) and has high radiation intensity, the detection range reaches its maximum. If the target deviates from the axis or the radiation intensity decreases, the detection range will decrease sharply according to an exponential law. For example, a certain type of seeker can achieve a forward detection range of up to 20 kilometers against a typical jet fighter (radiation intensity of approximately 1000 W / sr), but when the target deviates from the axis by 30°, the detection range may plummet to less than 8 kilometers. The system's detection capability against low-radiation targets (such as stealthy UAVs) will further decrease, potentially maintaining only 3-5 kilometers of effective detection under the same conditions.

[0070] Specifically, the detection range model of the airborne fire control radar in this embodiment is as follows:

[0071]

[0072] In the formula, R max P represents the maximum detection range. t The radar transmit power is represented by G, and the target radar cross-section is represented by s. rcs P represents the sensitivity of the radar receiver. min α represents the minimum signal power that radar can detect. max and β max These represent the maximum search angles of the radar in the yaw and pitch directions, respectively.

[0073] In the simulation environment, the mid-course guidance system of the airborne fire control radar uses a simplified logic model to implement its core functions. The basic working principle of this system is as follows: when an enemy target enters the pre-set radar detection range, it is immediately identified as detectable and automatically enters the tracking process. The radar's detection range is based on the parameters of a typical fighter jet's fire control radar, with a detection range of 200 kilometers for a target with a radar cross-section of 1 square meter, and a detection angle of 60 degrees to the left and right of the fuselage axis in a fan-shaped area. As long as the target remains within this spatial range, it will be continuously tracked, and real-time position and velocity data will be generated.

[0074] Once the target is successfully detected and tracked, the system gains the capability to provide mid-course guidance for the missile. After launch, the airborne fire control radar continuously transmits real-time motion parameters of the target to the missile, guiding it towards the target. This guidance process continues until one of three scenarios occurs: the missile approaches a sufficient distance and activates its terminal guidance system, the target leaves the radar's detection range, or the missile completes its interception mission. The entire process employs an idealized data transmission model, assuming a consistently stable and reliable data link between the radar and the missile, and neglecting potential signal delays, interference, or loss in the real-world environment.

[0075] The system also includes basic threat warning functionality. When the missile is illuminated by an enemy radar beam, a simple "radar locked" warning will be triggered. If the target suddenly moves out of radar detection range during guidance, the system will immediately terminate the transmission of guidance commands, and the missile will continue its flight based on the last target information received.

[0076] The infrared detection range of the seeker is modeled as follows:

[0077]

[0078] In the formula, R represents the maximum locking distance, I represents the infrared radiation intensity of the target, h represents the angle between the target and the line of sight of the infrared seeker, σ represents the sensitivity attenuation coefficient of the infrared seeker, and S0 represents the receiving sensitivity threshold of the infrared seeker.

[0079] As the missile approaches its target, the infrared seeker takes over radar guidance, providing high-precision tracking. At this point, there is no need for our UAVs to continue providing detection results from the fire control radar.

[0080] In this embodiment, the airborne fire control radar subsystem simulates key performance indicators including detection probability, angle measurement accuracy, and multi-target processing capability, while also considering interference from complex electromagnetic environments; the missile infrared seeker sensor model, i.e., the infrared detection system, is based on target radiation characteristics and combines atmospheric transmission models and background noise to construct a high-fidelity detection simulation environment.

[0081] Step S3: Construct a reward function to quantitatively assess the battlefield situation and provide a learning objective for the reinforcement learning algorithm training.

[0082] In this embodiment, a reward function can be designed to quantitatively assess the battlefield situation. Specifically, a dynamic reward function can be designed based on the battlefield situation to quantitatively assess key air combat indicators (such as missile hit rate, information superiority, threat avoidance, etc.). Furthermore, through multi-objective weighted fusion and sparse reward optimization, interpretable training signals are provided for the reinforcement learning algorithm to guide the agent in learning tactical decisions.

[0083] In this embodiment, the reward mechanism represented by the reward function includes a state reward mechanism and a state transition reward mechanism.

[0084] The state-based reward mechanism quantifies the incremental tactical advantage gained by the Red Force (our own drone swarm) under specific battlefield conditions. This reward system comprises two core components: a fire control radar detection reward, which provides a positive bonus when a drone effectively detects and locks onto an enemy target; and a missile guidance reward, which provides a positive bonus when a drone provides mid-course guidance support for a missile launched by our side. These two reward functions together constitute the real-time tactical effectiveness evaluation index.

[0085] The state transition reward mechanism is used to evaluate the change in advantage gained by the Red Force's UAV swarm during changes in battlefield state. This mechanism comprises three key components: Aircraft crash reward reflects the kill-to-kill ratio, with a negative reward applied when a friendly UAV is shot down and a positive reward when an enemy target is destroyed; missile launch reward regulates combat behavior, with a negative reward for launching missiles without meeting mid-course guidance conditions, and a positive reward for launches that meet tactical coordination requirements; and missile failure reward constrains ammunition waste, generating a negative reward when a missile crashes without hitting its target. These rewards collectively constitute tactical feedback signals in a dynamic game environment, driving agents—autonomous decision-making entities trained through reinforcement learning algorithms, such as UAV swarms or virtual flight decision-making systems—to learn more efficient cooperative combat strategies.

[0086] Specifically, as shown in Table 1 below, the reward function includes a state reward mechanism and a state transition reward mechanism.

[0087] Table 1. Schematic diagram of state reward mechanism and state transition reward mechanism

[0088]

[0089] The status-based reward mechanism guides the UAV swarm to optimize its battlefield awareness and guidance coordination capabilities through progressive incentives. An immediate reward of +0.02 is given when the airborne fire control radar detects an enemy aircraft, +0.03 is given when providing mid-course guidance for a missile, and an even higher reward of +0.06 is given when the missile enters the infrared terminal guidance phase. These fine-grained positive feedbacks continuously enhance the UAV's reconnaissance-strike coordination effectiveness.

[0090] Its progressive reward structure is as follows:

[0091] Detection Reward (+0.02): When a UAV detects an enemy aircraft (Blue Force) through its onboard fire control radar, it receives a basic reward to incentivize the cluster to prioritize improving its battlefield awareness capabilities (such as expanding the reconnaissance range and optimizing target identification).

[0092] Midcourse guidance bonus (+0.03): UAVs receive a midcourse bonus when they provide midcourse guidance for missiles (such as data link to correct the trajectory), enhancing the real-time coordination capabilities of the cluster and strike units.

[0093] Terminal guidance bonus (+0.06): The highest bonus is given when the missile switches to the infrared autonomous guidance phase, which helps the UAV ensure the stability of the guidance link (such as anti-jamming and precise locking).

[0094] In this embodiment, the significance of fine-grained feedback lies in the independent reward of each step, avoiding the univariate result theory (such as only rewarding the final hit), and prompting the cluster to optimize the overall process efficiency.

[0095] The state transition reward mechanism regulates key tactical behaviors through significant rewards and penalties: shooting down an enemy aircraft earns a high reward of +200, while losing one results in a penalty of -200; a well-timed missile launch rewards +20, while a reckless launch incurs a penalty of -5; and a missile missing its target incurs a penalty of -50. This differentiated reward design highlights the value of combat results while constraining resource waste, driving agents to develop combat strategies that combine both aggressiveness and discipline in intense combat. In other words, it guides agents to learn tactical strategies that combine both aggressiveness and discipline through differentiated reward and penalty rules. A detailed analysis follows:

[0096] Core rewards for achievements (high rewards and penalties)

[0097] Shoot down enemy aircraft: +200

[0098] Function: Highlight the ultimate combat objective and motivate agents to prioritize effective kills.

[0099] Our losses: -200

[0100] Purpose: To emphasize survival priority and avoid unnecessary sacrifices (such as blind charges).

[0101] Weapon Usage Guidelines (Precise Rewards and Punishments)

[0102] Reasonable missile launch: +20

[0103] Function: To encourage firing at times with a high probability of success (such as after target lock), thereby improving resource utilization.

[0104] Blind launch: -5

[0105] Function: To suppress ineffective attacks (such as blind volleys beyond visual range) and avoid wasting ammunition.

[0106] Feedback on the effectiveness of the crackdown (severe punishment)

[0107] Missile miss: -50

[0108] Function: To punish inefficient attacks (such as uncorrected trajectories under interference conditions) and to encourage agents to optimize guidance strategies.

[0109] In this embodiment, a battlefield situation assessment system is designed to transform key elements of air combat into quantifiable reward signals. This system dynamically evaluates core indicators such as missile attack effectiveness, information superiority assurance, and threat response. Through multi-dimensional reward fusion, it provides clear and effective learning guidance signals for the agent and promotes the rapid improvement of tactical decision-making capabilities.

[0110] The reward function uses a configurable weight matrix W = [w1, w2, w3] for dynamic adjustment, where w1 is the kill ratio weight (default 1.0), w2 is the missile hit weight (default 0.8), and w3 is the coordination effectiveness weight (default 0.6). Real-time adjustment is supported through the interface set_reward_weights(weights). For example, in "escort missions," setting W = [1.2, 0.5, 0.3] emphasizes survival priority, while in "annihilation missions," setting W = [0.7, 1.5, 0.9] enhances strike effectiveness.

[0111] Step S4: Encapsulate the constructed simulation environment into a standardized reinforcement learning interface to support the interaction between the state space and the action space, and achieve efficient coupling between the reinforcement learning algorithm and the simulation environment through parallel simulation acceleration and asynchronous communication mechanisms.

[0112] The state space is the set of all possible states in the simulation environment, including the fighter jet's own state, sensor data, and battlefield situation; the action space is the set of all possible actions that the UAV swarm can perform under a given state.

[0113] In this embodiment, the simulation environment is encapsulated as a standardized reinforcement learning interface, supporting interaction between the state space (such as aircraft position and radar data) and the action space (such as maneuver commands and weapon launches). Through parallel simulation acceleration and asynchronous communication mechanisms, efficient coupling between the algorithm and the environment can be achieved, thereby meeting the needs of large-scale training.

[0114] The standardized reinforcement learning interface in this embodiment includes:

[0115] (1) Environment initialization interface reset(): Resets the simulation environment to its initial state and returns the initial observation state. It supports parameterized initial condition settings (such as the initial position, speed, and battlefield scale of the aircraft), and is suitable for training needs in multiple scenarios.

[0116] Specifically, this interface is responsible for resetting the simulation environment to a preset initial state. Its input can receive parameterized initial condition configurations, including the initial latitude and longitude coordinates of the red and blue team's aircraft, the number of air-to-air missiles they carry, and the battlefield airspace range. Its output is a standardized format of the initial observation state. By supporting custom initial conditions, this interface can meet the training needs of multiple scenarios, from basic single-aircraft combat to complex swarm operations, ensuring that the agent has the ability to generalize to different battlefield situations.

[0117] (2) State Interaction Interface step(action): Used to receive action commands from the agent (such as maneuver control, weapon firing, etc.). The action space must support a hybrid encoding of discrete (e.g., tactical decision-making) and continuous (e.g., flight actions). That is, the agent sends action commands to the simulation environment through the state interaction interface using a hybrid encoding method of discrete and continuous methods. This interface is also used to return a four-tuple (observation, reward, done, info), i.e., a four-tuple state observation output, where:

[0118] Observation: Current status observation (such as aircraft attitude, radar data, number of remaining missiles, etc.) requires standardization of dimensions and numerical ranges.

[0119] reward: The immediate reward value calculated based on the reward function.

[0120] done: A termination marker that triggers termination conditions (such as mission completion, aircraft being destroyed, etc.).

[0121] info: A dictionary structure used to assist in debugging information (such as battlefield situation logs, raw physical state, etc.).

[0122] Specifically, the state interaction interface serves as the core interaction channel between the environment and the agent. This interface receives multi-dimensional action command inputs, where the environment specifically refers to the digital simulation environment constructed for training the air combat decision-making agent. The action space employs a hybrid coding design: the discrete dimension handles tactical decisions, including missile launch and radar on / off. The continuous dimension controls flight parameters, including four control variables: aileron deflection, elevator deflection, rudder deflection, and throttle change. The interface output is a standard four-tuple, where observation is a normalized state observation vector containing the following multi-dimensional information: Self-state, including altitude, sine and cosine values ​​of roll / pitch angles, three-axis velocity components, vacuum speed, and remaining missile count; Launched missile state, including relative speed difference, relative altitude difference, entry angle, exit angle, relative distance, azimuth marker, and target / enemy aircraft code; Friendly aircraft state, including relative speed difference, relative altitude difference, entry angle, exit angle, relative distance, azimuth marker, and detected state; Enemy aircraft state, including relative speed difference with error, relative altitude difference with error, entry angle, exit angle, relative distance with error, azimuth marker, and detected state by radar; and Incoming missile threat, including relative speed difference, relative altitude difference, entry angle, exit angle, relative distance, and azimuth marker. All observations are normalized. Position information uses the NED (a local reference coordinate system) coordinate system, and angle information uses the spherical coordinate system. Dynamic offsets ensure alignment of observation dimensions, and unused observation slots are automatically filled with zeros. The reward is calculated in real time based on the reward function. The done flag triggers the termination condition. The info dictionary provides extended information such as raw physical quantities and debugging logs.

[0123] Multi-agent collaboration is achieved through a hierarchical communication mechanism: 1) The tactical layer employs a Qmix network (an agent reinforcement learning algorithm) based on an attention mechanism, through hidden_state = [h1, h2, ..., h n Share the observation features of each UAV, where hidden_state is the joint hidden state of one party, h n 1) Represent the hidden state of the nth agent; 2) Establish a radar data link at the sensor layer and realize the fusion of detection information according to formula (8):

[0124]

[0125] Among them, R fusion This indicates that one party shares radar data, R i For the radar data of the i-th UAV, d i For relative distance, α = 0.1 is the attenuation coefficient, N is the total number of UAVs, and e is the natural constant. 3) The weapon coordination layer achieves missile launch timing coordination through distributed constraint optimization, avoiding resource waste caused by multiple UAVs attacking the same target simultaneously.

[0126] (3) The observation space definition interface get_observation_space(): returns the dimension, data type and value range of the state space.

[0127] Specifically, this parameterless interface returns a named tuple containing three key attributes: `shape` defines the dimensions of the state vector, `dtype` specifies the data type, and `bounds` describes the range of values ​​for each dimension. This definition ensures that different algorithms can correctly parse the state structure, while also supporting automatic scaling to accommodate the incorporation of new sensor data. Each dimension represents an observable physical quantity in the environment.

[0128] (4) Action space definition interface get_action_space(): Defines the format of actions that the agent can execute.

[0129] Specifically, the interface output adopts a multi-level container structure of gym.Space (an abstract base class and a fundamental component for building reinforcement learning environments), where the composite action space includes: a continuous space of type Box (representing a multi-dimensional continuous real number space) and a discrete space of type Discrete (representing a finite discrete action space). This structured definition satisfies the needs of traditional RL (a machine learning method that allows agents to learn optimal decision policies through trial and error) algorithms, and is also compatible with hierarchical policy training based on action masks.

[0130] (5) Battlefield situation visualization interface render(): Outputs acmi file, supports visualization of battlefield situation through Tacview.

[0131] Specifically, the input parameter `mode` of the battlefield situation visualization interface controls the output mode. When 'acmi' (Air Combat Maneuvering Instrumentation) is selected, the interface generates a trajectory file in real time that conforms to the Tacview (3D flight data analysis and visualization tool) parsing standard, containing over 50 fields such as latitude and longitude, attitude, and weapon status for all entities. The visualized data is updated at a frequency of 10Hz, supporting third-party tools for post-battle 3D situation review and analysis. In this embodiment, the battlefield situation visualization interface is used to record and visualize dynamic information of battlefield entities in real time.

[0132] (6) Random seed control interface seed(seed_value): Fixed environment random seed to ensure experiment reproducibility.

[0133] Specifically, this interface takes an integer seed_value as input. By fixing the seed value of the internal random number generator, it ensures that the experience trajectories obtained by the agent under the same experimental configuration are completely consistent. This interface can synchronously return a list of locked seeds, thus facilitating controllable randomness during distributed training.

[0134] In this embodiment, a standardized reinforcement learning training platform is constructed, fully encapsulating interfaces for state perception, decision control, and environmental interaction. This platform supports multi-dimensional state representation, including spatial situation and sensor data, as well as complex action spaces encompassing maneuver control and weapon usage. Through an optimized parallel computing architecture and efficient data communication mechanisms, this standardized reinforcement learning training platform achieves high-performance collaboration between the simulation environment and the learning algorithm, thereby meeting the large-scale computing requirements of modern intelligent air combat training.

[0135] In summary, at the technical implementation level, this method creatively integrates high-fidelity dynamics simulation with the needs of intelligent decision-making training. The six-degree-of-freedom dynamics model of the UAV, built using the JSBSim engine, accurately simulates the aerodynamic characteristics and control response of the aircraft, while the independently developed three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile achieves lightweight simulation of the guidance process. The sensor system adopts a hierarchical modeling strategy, preserving key sensor features while ignoring complex scanning logic, achieving a balance between simulation accuracy and speed. In terms of system architecture, this solution overcomes the limitations of traditional simulation tools. By designing a standardized reinforcement learning interface, end-to-end connectivity from low-level physical simulation to high-level decision-making training is achieved. Furthermore, the unique hybrid action space design supports the collaborative optimization of discrete tactical decision-making and continuous flight control, while the dynamic expansion mechanism of the multi-dimensional observation space ensures the system's rapid adaptability to new equipment and tactics.

[0136] The core advantage of this method lies in its practical engineering value. A carefully designed composite reward mechanism guides tactical behavior with fine-grained precision, enabling the agent to naturally learn the winning mechanisms of beyond-visual-range air combat. Experiments show that agents trained in this simulation environment exhibit significantly improved tactical decision-making capabilities compared to traditional methods. This system not only provides an efficient verification platform for the development of intelligent air combat algorithms but also lays a crucial technological foundation for the future practical application of unmanned air combat systems.

[0137] The technological innovation and practicality of this solution have been verified in actual applications. Its modular design facilitates functional expansion and performance optimization, and it has broad prospects for military applications and industrialization promotion value.

[0138] Figure 2 This is a structural block diagram of a simulation environment construction device for beyond-visual-range air combat according to an embodiment of the present invention. Figure 2As shown, the simulation environment construction device 100 for beyond-visual-range air combat includes a first construction unit 10, a second construction unit 20, a third construction unit 30, and an encapsulation unit 40.

[0139] The first building unit 10 is used to construct a high-fidelity aerodynamic model, including a six-degree-of-freedom aerodynamic model of the UAV and a three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile. The second building unit 20 is used to construct sensor models, including sensor models of the airborne fire control radar and the missile's infrared seeker. The third building unit 30 is used to construct a reward function to quantitatively assess the battlefield situation and provide learning objectives for the reinforcement learning algorithm. The encapsulation unit 40 is used to encapsulate the constructed simulation environment into a standardized reinforcement learning interface to support interaction between the state space and action space, and to achieve efficient coupling between the reinforcement learning algorithm and the simulation environment through parallel simulation acceleration and asynchronous communication mechanisms.

[0140] It should be noted that the specific implementation of the simulation environment construction device for beyond-visual-range air combat in this embodiment can be found in the specific implementation of the simulation environment construction method for beyond-visual-range air combat described above. To avoid redundancy, it will not be repeated here.

[0141] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for constructing a simulation environment for beyond-visual-range air combat.

[0142] Furthermore, the present invention also provides a simulation environment construction system for beyond-visual-range air combat, including the above-described device or the above-described computer-readable storage medium.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a simulation environment for beyond-visual-range air combat, characterized in that, include: A high-fidelity aerodynamic model is constructed, which includes a six-degree-of-freedom aerodynamic model of the UAV and a three-degree-of-freedom aerodynamic model of the beyond-visual-range air-to-air missile; Construct sensor models, including sensor models of airborne fire control radar and missile infrared seeker; Construct a reward function to quantitatively assess the battlefield situation and provide a learning objective for training reinforcement learning algorithms; The constructed simulation environment is encapsulated as a standardized reinforcement learning interface to support the interaction between the state space and the action space, and efficient coupling between the reinforcement learning algorithm and the simulation environment is achieved through parallel simulation acceleration and asynchronous communication mechanisms.

2. The method as described in claim 1, characterized in that, When constructing a high-fidelity aerodynamic model, the aerodynamic forces, torques, and propulsion system characteristics are calculated to accurately simulate the attitude, speed, and maneuverability of the aircraft, ensuring that the physical behavior in the simulation environment conforms to real air combat scenarios.

3. The method as described in claim 1, characterized in that, With the goal of simulating detection range, azimuth resolution, and multi-target tracking capability, an airborne fire control radar sensor model is constructed by introducing environmental disturbances, including atmospheric attenuation and clutter noise. Based on the target's thermal radiation characteristics, combined with atmospheric transmission loss and background radiation noise, a missile infrared seeker sensor model is constructed, where environmental parameters are injected to enhance battlefield uncertainty.

4. The method as described in claim 1, characterized in that, The reward function is characterized by a state reward mechanism and a state transition reward mechanism. The state reward mechanism guides the UAV swarm to optimize its battlefield perception and guidance coordination capabilities through progressive incentives. The state transition reward mechanism guides the UAV swarm to learn tactical strategies that are both aggressive and disciplined through differentiated reward and punishment rules.

5. The method as described in claim 4, characterized in that, The state reward mechanism is used to quantify the incremental tactical advantage gained by our drone swarm under specific battlefield conditions. The state reward mechanism includes the following two core components: Fire control radar detection rewards provide positive rewards when the UAV effectively detects and locks onto enemy targets; Missile guidance rewards provide positive incentives when UAVs provide mid-course guidance support for missiles launched by our side; The state transition reward mechanism is used to evaluate the change in advantage gained by our drone swarm during changes in battlefield state. The state transition reward mechanism includes the following three key parts: The aircraft crash reward reflects the battle loss exchange ratio. A negative reward is applied when our drone is shot down, and a positive reward is given when the enemy target is destroyed. Missile launch rewards are used to regulate combat behavior. Launching a missile without meeting mid-course guidance conditions will result in a negative reward, while launches that meet tactical coordination requirements will receive a positive reward. Missile failure rewards are used to constrain the waste of ammunition; a negative reward is generated when a missile misses its target and crashes.

6. The method as described in claim 1, characterized in that, Standardized reinforcement learning interfaces include at least an environment initialization interface, a state interaction interface, an observation space definition interface, an action space definition interface, a battlefield situation visualization interface, and a random seed control interface.

7. The method according to any one of claims 1-6, characterized in that, The state space is the set of all possible states in the simulation environment, including the aircraft's own state, sensor data, and battlefield situation; the action space is the set of all possible actions that the UAV swarm can perform under a given state.

8. A simulation environment construction device for beyond-visual-range air combat, characterized in that, include: The first building unit is used to build a high-fidelity aerodynamic model, which includes a six-degree-of-freedom aerodynamic model of an unmanned aerial vehicle and a three-degree-of-freedom aerodynamic model of a beyond-visual-range air-to-air missile. The second building unit is used to build sensor models, which include sensor models of airborne fire control radar and missile infrared seeker. The third building unit is used to construct the reward function, quantitatively evaluate the battlefield situation, and provide learning objectives for the training of reinforcement learning algorithms. The encapsulation unit is used to encapsulate the constructed simulation environment into a standardized reinforcement learning interface to support the interaction between the state space and the action space, and to achieve efficient coupling between the reinforcement learning algorithm and the simulation environment through parallel simulation acceleration and asynchronous communication mechanisms.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. A simulation environment construction system for beyond-visual-range air combat, characterized in that, It includes the apparatus of claim 8 or the computer-readable storage medium of claim 9.

Images