An air combat intelligent decision-making method, device and equipment based on curriculum reinforcement learning and a medium

Abstract

The application provides an air combat intelligent decision-making method and device based on course reinforcement learning, equipment and medium, and relates to the field of reinforcement learning. It comprises the following steps: determining corresponding battle advantage indexes based on multiple sets of initial state data; storing the initial state data in a difficulty-guided state pool after sorting based on the battle advantage indexes; updating the initial state data in the progressive sliding buffer in an increasing difficulty manner from the difficulty-guided state pool; dynamically formulating intermediate task targets until the final task target is formulated; sampling the current initial state data from the progressive sliding buffer and inputting the current initial state data into the policy network of the SAC model to obtain the current action; training the SAC model based on at least the intermediate task target, the current action and the current initial state data until the final task target is reached to obtain the trained SAC model for making intelligent decisions, thereby improving the decision-making ability and adaptability of the aircraft in complex environments.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning, and in particular to a method, apparatus, device, and medium for intelligent air combat decision-making based on curriculum reinforcement learning. Background Technology

[0002] In recent years, with the continuous development of UCAV (Unmanned Combat Aerial Vehicle) technology, how to enable it to make autonomous decisions in complex air combat environments has become a research focus. Reinforcement Learning (RL) has shown great potential in achieving autonomous control of UCAVs, especially for high-risk missions such as air combat that require rapid adaptation and complex tactical maneuvers. Although RL-based methods have achieved significant success in areas such as path planning, obstacle avoidance, and autonomous flight control, their application in UCAV combat missions faces significant challenges due to the large amount of computational resources and time required for training. The RL training process in UCAVs is particularly slow and computationally intensive, mainly due to the inherent complexity of the air combat environment. These scenarios require exploring high-dimensional action spaces, adapting to adversarial dynamics in real time, and precisely coordinating tactical maneuvers. The computational cost of achieving operational proficiency under these constraints poses a significant obstacle to scalability and deployment readiness. Furthermore, the dynamic and adversarial nature of air combat exacerbates these challenges, requiring systems to rapidly adapt to constantly changing threats and make real-time decisions. Achieving this capability typically requires extended training time, highlighting the urgent need for more efficient RL training paradigms that need to be tailored to the unique requirements of UCAV combat scenarios.

[0003] It is evident that current technologies for training UCAVs for combat based on reinforcement learning suffer from technical shortcomings such as slow training convergence speed, high computational resource requirements, and poor adaptability to complex tactical environments. As a result, it is currently impossible to improve the decision-making and adaptability of UCAVs in complex air combat environments while effectively accelerating the training process and reducing computational costs. Summary of the Invention

[0004] In view of the above-mentioned technical problems, the present invention provides an air combat intelligent decision-making method, device, equipment and medium based on course reinforcement learning, which aims to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of this invention provides an intelligent air combat decision-making method based on curriculum reinforcement learning, the method comprising:

[0006] Based on multiple sets of initial state data, the combat advantage index corresponding to each set of initial state data is determined. The combat advantage index represents the difficulty of the combat scenario for the first friendly aircraft.

[0007] Based on the combat advantage index, the multiple sets of initial state data are sorted, and the sorted multiple sets of initial state data are stored in the difficulty-guided state pool.

[0008] The initial state data in the progressive sliding buffer is updated from the difficulty-guided state pool in an increasing manner;

[0009] Intermediate task objectives are dynamically formulated for the policy network in the deep reinforcement learning SAC model to be trained. The intermediate task objectives guide the policy network from simple tasks to complex tasks until the final task objective is formulated for the policy network.

[0010] The current initial state data is sampled from the progressive sliding buffer, and the current initial state data is input into the policy network in the deep reinforcement learning SAC model to be trained to obtain the current action of the first local aircraft.

[0011] Based at least on the intermediate mission objective, the current actions of the first friendly aircraft, and the current initial state data, the deep reinforcement learning SAC model to be trained is trained until the first friendly aircraft reaches the final mission objective, thus obtaining a trained deep reinforcement learning SAC model.

[0012] The current air combat environment information is input into the policy network of the trained deep reinforcement learning SAC model, and the action information of the second friendly aircraft is output.

[0013] A second aspect of the present invention provides an intelligent air combat decision-making device based on curriculum reinforcement learning, the device comprising:

[0014] The indicator determination module is used to determine the combat advantage indicator corresponding to each set of initial state data based on multiple sets of initial state data. The combat advantage indicator represents the difficulty of the combat scenario for the first friendly aircraft.

[0015] The state pool storage module is used to sort the multiple sets of initial state data based on the combat advantage index and store the sorted multiple sets of initial state data into the difficulty-guided state pool.

[0016] The buffer update module is used to update the initial state data in the progressive sliding buffer from the difficulty-guided state pool in an increasing manner;

[0017] The goal setting module is used to dynamically set intermediate task goals for the policy network in the deep reinforcement learning SAC model to be trained. The intermediate task goals guide the policy network to transition from simple tasks to complex tasks until the final task goal is set for the policy network.

[0018] The action output module is used to sample the current initial state data from the progressive sliding buffer, input the current initial state data into the policy network in the deep reinforcement learning SAC model to be trained, and obtain the current action of the first local aircraft.

[0019] The model training module is used to train the deep reinforcement learning SAC model to be trained based at least on the intermediate mission objective, the current action of the first friendly aircraft, and the current initial state data, until the first friendly aircraft reaches the final mission objective, and obtain the trained deep reinforcement learning SAC model.

[0020] The decision output module is used to input the current air combat environment information into the policy network of the trained deep reinforcement learning SAC model and output the action information of the second friendly aircraft.

[0021] A third aspect of the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the air combat intelligent decision-making method based on curriculum reinforcement learning as described in the first aspect of the present invention.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the air combat intelligent decision-making method based on curriculum reinforcement learning, as described in the first aspect of the present invention.

[0023] The air combat intelligent decision-making method based on curriculum reinforcement learning provided by this invention integrates automatic curriculum learning into the aircraft's air combat training process: It determines the corresponding combat indicator advantages for initial state data, sorts the initial state data based on these advantages, generates a difficulty-guided state pool, and then updates the initial state data in a progressive sliding buffer from the difficulty-guided state pool in an increasing difficulty manner. This dynamically adjusts the progressive sliding buffer as the state pool for training samples, gradually introducing more complex combat scenarios during training to ensure the diversity of training samples while avoiding overly simple samples from affecting training effectiveness, thus supporting effective curriculum learning in air combat missions. Simultaneously, the difficulty of the mission objectives in this invention increases progressively, starting with simple tasks and dynamically adjusting the difficulty of the mission objectives. Intermediate mission objectives are used to gradually guide the policy network to approach the final mission objective. This dynamic adjustment mechanism ensures that the policy network can gradually master complex tactical behaviors during training, improving its tactical decision-making capabilities. Thus, this invention adopts a progressive difficulty approach, starting with simplified tasks and progressing to complex scenarios through automatic complexity measurement and adaptive target setting. This adaptive curriculum optimizes learning efficiency, reduces training costs, significantly accelerates the training process, reduces computational resource requirements, and enhances the decision-making and adaptability of the second-party aircraft in complex air combat environments, enabling the second-party aircraft to effectively and quickly acquire advanced tactical strategies. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the steps of an intelligent air combat decision-making method based on curriculum reinforcement learning, as shown in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of air combat geometry shown in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the process of calculating combat advantage indicators and generating a state pool according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram illustrating the progressive task generation process according to an embodiment of the present invention;

[0029] Figure 5This is a schematic diagram illustrating the update of a progressive sliding buffer according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram of a weapons engagement zone shown in an embodiment of the present invention;

[0031] Figure 7 This is a schematic diagram illustrating an automatic combat course learning framework according to an embodiment of the present invention;

[0032] Figure 8 This is a schematic diagram of the overall process of an intelligent air combat decision-making method based on curriculum reinforcement learning, as shown in an embodiment of the present invention.

[0033] Figure 9 This is a structural block diagram of an air combat intelligent decision-making device based on curriculum reinforcement learning, provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Currently, several key technical challenges exist in training unmanned combat aerial vehicles (UCAVs) for combat using Deep Reinforcement Learning (DRL). These challenges include the following three aspects: 1. Slow training convergence speed: Traditional DRL methods typically rely on a large number of samples and repeated computational iterations during training. Especially in the highly complex and dynamically changing environment of air combat, the convergence speed of model training is often slow, making it difficult to meet the real-time requirements of practical applications. Long training cycles not only increase development costs but also affect the timeliness of actual combat deployment. 2. High computational resource requirements: Air combat missions involve extremely large and high-dimensional state and action spaces. UCAVs need to make immediate and accurate decisions in these complex spaces. Traditional DRL methods require a large amount of computational resources and data samples during the training phase, resulting in exceptionally high computational overhead and becoming a bottleneck for application promotion. This problem is particularly prominent in environments with limited resources or computational power, severely affecting the system's scalability and practical operational capabilities. 3. Poor adaptability to complex tactical environments: Air combat environments are highly dynamic and adversarial. Tactical strategies need to be adjusted in real time according to the opponent's actions and changes in the battlefield environment. Traditional reinforcement learning methods, especially course learning models based on fixed task difficulty, lack the ability to dynamically adjust and adapt training difficulty or optimization strategies according to training progress and environmental changes. This makes it difficult for intelligent decision-making models to continuously improve in complex and rapidly changing air combat scenarios, affecting their effectiveness and robustness in actual combat.

[0036] The dynamic, adversarial, and high-dimensional nature of air combat exacerbates the challenges of reinforcement learning training. Unlike standard UAV missions such as obstacle avoidance or autonomous landing, UCAV combat scenarios require constant adaptation to intelligent adversaries. Environmental changes are unpredictable, and reward signals are sparse, necessitating extensive exploration and extended convergence times, making the training process both time-consuming and resource-intensive. Furthermore, the risk of catastrophic forgetting during incremental learning and the computational burden of high-fidelity simulations further hinder the development of efficient training processes. Currently, there is an urgent need for a new method that can effectively accelerate the training process, reduce computational costs, and simultaneously improve the decision-making and adaptability of UCAVs in complex air combat environments, thereby better meeting the actual needs of unmanned combat aerial vehicle (UCAV) combat missions.

[0037] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes an intelligent air combat decision-making method based on curriculum reinforcement learning. In this method, corresponding combat advantage indicators are determined for each set of initial state data, and the initial state data are sorted based on the combat indicator advantages to generate a difficulty-guided state pool, thereby achieving automated combat complexity assessment. By adopting a progressive sliding buffer mechanism, the initial state data in the progressive sliding buffer are updated from the difficulty-guided state pool in an increasing manner to dynamically adjust the state pool of the progressive sliding buffer as training samples. More complex combat scenarios are gradually introduced during the training process to ensure the diversity of training samples. By using intermediate mission objectives to gradually guide the policy network to approach the final mission objective, goal-driven progressive mission generation is achieved. This dynamic adjustment mechanism ensures that the policy network can gradually master complex tactical behaviors during the training process, thereby improving tactical decision-making capabilities. Thus, this invention adopts a progressive difficulty approach, starting with simplified tasks and progressing to complex scenarios through automatic complexity measurement and adaptive target setting. This adaptive curriculum optimizes learning efficiency, reduces training costs, accelerates the training process, reduces computational resource requirements, and enhances the decision-making and adaptability of the second-party aircraft in complex air combat environments, enabling the second-party aircraft to effectively and quickly acquire advanced tactical strategies.

[0038] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of an intelligent air combat decision-making method based on curriculum reinforcement learning, as shown in an embodiment of the present invention. Figure 1 As shown, the air combat intelligent decision-making method based on curriculum reinforcement learning provided in this embodiment includes at least the following steps:

[0039] Step S11: Based on multiple sets of initial state data, determine the combat advantage index corresponding to each set of initial state data. The combat advantage index represents the difficulty of the combat scenario for the first friendly aircraft.

[0040] In this embodiment, multiple sets of initial state data can be acquired, each set representing the multidimensional initial situational information of both sides during air combat training. After acquiring these multiple sets of initial state data, this embodiment can determine the combat advantage index corresponding to each set. In this embodiment, the first friendly aircraft is the friendly aircraft during model training (the friendly aircraft among the two combatants), and the combat advantage index represents the difficulty of the combat scenario for the first friendly aircraft (referred to as the combat scenario difficulty). This embodiment can dynamically evaluate multiple key factors in the combat scenario through each set of initial state data, quantify the difficulty of the combat scenario, and obtain the combat advantage index corresponding to each set of initial state data. In an optional implementation, the multiple key factors in the combat scenario include at least: the engagement angles of the two combatants, energy advantage (capability consumption), and positional relationship.

[0041] In an optional implementation, the initial state data of this embodiment can be obtained by processing scenario data. Specifically, for the scenario data, this embodiment can randomly initialize the air combat training scenario, randomly generating multiple sets of scenario data. The scenario is a hypothetical scenario of the basic situation, combat intentions, and combat development of both sides; for example, each set of scenario data can include at least one or more of the following: the initial latitude and longitude, initial speed, initial altitude, and initial yaw angle of the first friendly aircraft; and the relative distance, relative speed, relative altitude, relative heading angle, and relative bearing angle of the enemy aircraft relative to the first friendly aircraft.

[0042] Step S12: Sort the multiple sets of initial state data based on the combat advantage index, and store the sorted multiple sets of initial state data into the difficulty-guided state pool.

[0043] In this embodiment, multiple sets of initial state data can be sorted based on the combat advantage index corresponding to each set of initial state data. The sorted sets of initial state data are then stored in a difficulty-guided state pool. This ensures that the deep reinforcement learning (SAC) model to be trained gradually adapts to increasingly complex combat scenarios during training, ultimately generating the difficulty-guided state pool. The difficulty-guided state pool in this embodiment stores multiple sets of initial state data sorted according to the combat advantage index, and each set of initial state data carries a corresponding combat advantage index.

[0044] Step S13: Update the initial state data in the progressive sliding buffer from the difficulty-guided state pool in an increasing difficulty manner.

[0045] In this embodiment, a progressive sliding buffer is set up as the state pool for training samples in air combat training. This embodiment uses a progressive sliding buffer mechanism to progressively update the initial state data in the difficulty-guided state pool according to the increasing difficulty of the combat scenario, thereby dynamically adjusting the state pool of the training samples. It can be understood that the initial state of the progressive sliding buffer consists of multiple sets of initial state data with the lowest combat scenario difficulty from the difficulty-guided state pool.

[0046] Step S14: Dynamically formulate intermediate task objectives for the policy network in the deep reinforcement learning SAC model to be trained. The intermediate task objectives guide the policy network to transition from simple tasks to complex tasks until the final task objective is formulated for the policy network.

[0047] In this embodiment, the difficulty of the task objective (the difficulty of the combat scenario for the first friendly aircraft) set for the policy network in the deep reinforcement learning SAC model to be trained is gradually increased. Intermediate task objectives are dynamically set for the policy network in the deep reinforcement learning SAC model to be trained. Specifically, the difficulty of the intermediate task objectives can be dynamically adjusted according to the training progress of the deep reinforcement learning SAC model to be trained. The intermediate task objectives guide the policy network to transition from simple tasks to complex tasks until the final task objective is set for the policy network in the deep reinforcement learning SAC model to be trained.

[0048] It should be noted that this embodiment does not specifically limit the execution order between step S14 and step S13. Step S13 can be executed first and then step S14, or step S14 can be executed first and then step S13, or steps S13 and S14 can be executed simultaneously.

[0049] Step S15: Sample the current initial state data from the progressive sliding buffer, and input the current initial state data into the policy network in the deep reinforcement learning SAC model to be trained to obtain the current action of the first local aircraft.

[0050] In this embodiment, the deep reinforcement learning SAC algorithm is used for training to initialize the deep reinforcement learning SAC model to be trained. The deep reinforcement learning SAC model to be trained is a component of the agent in deep reinforcement learning. The deep reinforcement learning SAC model to be trained includes at least a policy network, which is used to output the action decision of the first friendly aircraft based on the current state information (such as position, speed and altitude) of the first friendly aircraft and the opponent aircraft.

[0051] In this embodiment, a set of initial state data can be sampled from a progressive sliding buffer to serve as the current initial state data. This current initial state data is then input into the policy network of the deep reinforcement learning SAC model to be trained, yielding the current actions of the first friendly aircraft output by the policy network for the intermediate task objective. In other words, the current initial state data in this embodiment is the current set of initial state data. In an optional example, the current actions of the first friendly aircraft output by the policy network may include: elevator, rudder, aileron, and throttle actions of the first friendly aircraft. In one example, the aircraft (such as the first friendly aircraft and the second friendly aircraft) may include: an unmanned combat aerial vehicle (UCAV).

[0052] Step S16: Train the deep reinforcement learning SAC model to be trained based at least on the intermediate mission objective, the current action of the first friendly aircraft, and the current initial state data, until the first friendly aircraft reaches the final mission objective, and obtain the trained deep reinforcement learning SAC model.

[0053] In this embodiment, the deep reinforcement learning SAC model to be trained can be trained based at least on the intermediate task objective set for the policy network in the deep reinforcement learning SAC model to be trained, the current action of the first friendly aircraft output by the policy network in the deep reinforcement learning SAC model to be trained, and the current initial state data input to the policy network in the deep reinforcement learning SAC model to be trained, until it is determined that the first friendly aircraft has reached the final task objective, thus obtaining the trained deep reinforcement learning SAC model.

[0054] In this embodiment, a reward mechanism is also designed during the training process: the first friendly aircraft to successfully chase the enemy aircraft is given a positive reward, otherwise it is given a negative reward; through multiple training iterations, the chasing strategy of the first friendly aircraft is gradually optimized until the first friendly aircraft reaches the final mission objective, and the trained deep reinforcement learning SAC model is obtained.

[0055] Step S17: Input the current air combat environment information into the policy network of the trained deep reinforcement learning SAC model, and output the action information of the second friendly aircraft.

[0056] In this embodiment, after obtaining the trained deep reinforcement learning SAC model, the policy network within the trained deep reinforcement learning SAC model can be used to make intelligent decisions for friendly aircraft in actual air combat, outputting the actions of friendly aircraft. Specifically, the current air combat environment information can be input into the policy network of the trained deep reinforcement learning SAC model, outputting the action information of a second friendly aircraft. Here, the second friendly aircraft in this embodiment is the friendly aircraft (the friendly aircraft in the combat relationship) at the time of model application, and the current air combat environment information is a set of air combat environment information, including multiple initial situational information of both sides during actual air combat.

[0057] To address the shortcomings of traditional learning methods that rely on static curriculum design and lack the ability to dynamically adjust difficulty based on model progress and actual conditions, especially in highly adversarial and rapidly changing environments like air combat, where fixed curriculum designs cannot effectively cope with complex combat scenarios, this embodiment integrates automatic curriculum learning into the aircraft's air combat training process. It determines the corresponding combat indicator advantages for initial state data, sorts the initial state data based on these advantages, and generates a difficulty-guided state pool. Then, it updates the initial state data in a progressive sliding buffer from this difficulty-guided state pool in an increasing difficulty manner, dynamically adjusting the progressive sliding buffer as the state pool for training samples. During training, more complex combat scenarios are gradually introduced to ensure the diversity of training samples while avoiding overly simplistic samples from negatively impacting training effectiveness, thus supporting effective curriculum learning in air combat missions. Furthermore, the difficulty of the mission objectives in this invention increases progressively, starting with simple tasks and dynamically adjusting the difficulty of the mission objectives. Intermediate mission objectives are used to gradually guide the policy network towards the final mission objective. This dynamic adjustment mechanism ensures that the policy network can gradually master complex tactical behaviors during training, improving its tactical decision-making capabilities. Thus, this invention adopts a progressive difficulty approach, starting with simplified tasks and progressing to complex scenarios through automatic complexity measurement and adaptive target setting. This adaptive curriculum optimizes learning efficiency, reduces training costs, significantly accelerates the training process, reduces computational resource requirements, and enhances the decision-making and adaptability of the second-party aircraft in complex air combat environments, enabling the second-party aircraft to effectively and quickly acquire advanced tactical strategies.

[0058] In one implementation, in conjunction with the above embodiments, to enable dynamic course learning to better utilize training samples, reduce ineffective exploration, and improve learning efficiency and stability, this embodiment designs different course scenarios based on the advantages and disadvantages of the initial spatial positions of the first friendly aircraft and the opposing aircraft. This design can simulate various situations that may be encountered in actual air combat, allowing the agent to be trained in diverse environments, thereby acquiring the ability to cope with various initial situations (initial state data).

[0059] Specifically, when randomly generating multiple sets of scenario data, the range of random variables in the scenario can be designed, for example, starting from the initial yaw angle of the first local aircraft. Initial altitude h0, initial velocity v0, relative distance d, relative heading angle θ of the enemy aircraft relative to the first friendly aircraft. heading and relative azimuth angle θ azimuth The decision was made jointly. In one example, by changing the values ​​of the relative heading angle and relative azimuth angle, the task curriculum was designed according to the principle of increasing difficulty, generating four sets of courses with varying difficulty levels, each satisfying the following conditions:

[0060] ①θ azimuth ∈[-Π / 2, Π / 2], θ heading =0, d∈[1000, 5000]m, h0∈[100, 12000]m, v0∈[120, 340]m / s

[0061] ②θ azimuth ∈[-Π,Π],θ heading =0, d∈[1000, 5000]m, h0∈[100, 12000]m, v0∈[120, 340]m / s

[0062] ③θ azimuth ∈[-Π,Π],θ heading ∈[-Π,0],d∈[1000,5000]m, h0∈[100, 12000]m, V0∈[120, 340]m / s

[0063] ④θ azimuth ∈[-Π,Π],θ heading ∈[-Π, Π], d∈[1000, 5000]m, h0∈[100, 12000]m, v0∈[120, 340]m / s;

[0064] This embodiment can randomly generate multiple sets of scenario data belonging to multiple sets of difficulty courses (such as the four sets of difficulty courses mentioned above) based on multiple sets of difficulty courses. Thus, multiple sets of initial state data can be obtained based on multiple sets of scenario data. In this way, when determining the combat advantage index corresponding to the initial state data and the subsequent sorting based on the combat advantage index, the initial state data is obtained in sequence based on the difficulty of the multiple sets of difficulty courses, the combat advantage index is calculated and the subsequent sorting is performed. This simplifies the sorting algorithm and saves computing resources.

[0065] In conjunction with the above embodiments, in one implementation, the present invention also provides an intelligent air combat decision-making method based on curriculum reinforcement learning. In this method, the step S11 above, "determining the combat advantage index corresponding to each set of initial state data based on multiple sets of initial state data," may specifically include steps S21 to S24:

[0066] Step S21: Determine the angle advantage value based on the engagement angles in each set of initial state data.

[0067] In this embodiment, each set of initial state data includes at least: the engagement angle, relative distance, relative flight altitude, and flight speed of the first friendly aircraft and the enemy aircraft; wherein, the enemy aircraft refers to the opposing aircraft in air combat (the opposing side in the combat). This embodiment mainly considers three aspects—angle advantage, distance advantage, and energy advantage—to calculate the combat advantage index. Specifically, regarding angle advantage, the angle advantage value corresponding to each set of initial state data can be determined based on the engagement angle in each set of initial state data.

[0068] Step S22: Determine the distance advantage value based on the relative distance in each set of initial state data.

[0069] In this embodiment, regarding the distance advantage, the distance advantage value corresponding to each set of initial state data can be determined based on the relative distance in each set of initial state data.

[0070] Step S23: Determine the energy advantage value based on the relative flight altitude and flight speed in each set of initial state data.

[0071] In this embodiment, regarding the energy advantage, the energy advantage value corresponding to each set of initial state data can be determined based on the relative flight altitude and the flight speed of the first home aircraft in each set of initial state data.

[0072] Step S24: Based at least on the angle advantage value, the distance advantage value, and the energy advantage value, determine the combat advantage index corresponding to each set of initial state data, wherein the larger the value of the combat advantage index, the lower the difficulty of the combat scenario for the first friendly aircraft.

[0073] In this embodiment, after obtaining the angle advantage value, range advantage value, and energy advantage value corresponding to each set of initial state data, the combat advantage index corresponding to each set of initial state data can be determined at least based on these values. In this embodiment, a larger value for the combat advantage index indicates a higher advantage for the first friendly aircraft, signifying a lower difficulty level for the first friendly aircraft in the combat scenario, and also indicating a lower difficulty level for the mission's combat scenario.

[0074] In conjunction with the above embodiments, in one implementation, the present invention also provides an intelligent air combat decision-making method based on curriculum reinforcement learning. In this method, step S21 may specifically include steps S31 to S34, step S22 may specifically include steps S35 to S37, and step S23 may specifically include steps S38 to S310.

[0075] Step S31: Determine the detection advantage value of the attack angle based at least on the radar's maximum detection angle and the attack angle.

[0076] In this embodiment, the angle takes into account the aircraft's angle of attack δ. ATA and escape angle σ AA The first local aircraft (e.g.) Figure 2 (Red machine in the middle) velocity direction v r and the other party's aircraft (e.g.) Figure 2 (Blue machine in the middle) velocity direction v b The angle between the line connecting the two machines (LOS line), such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the air combat geometry according to an embodiment of the present invention. That is, the engagement angles in this embodiment include the attack angle and the escape angle.

[0077] In this embodiment, based on angle and energy theories in air combat, situational advantage in a joint state can be captured and evaluated from detection and threat advantages, ultimately yielding combat advantage indicators. Specifically, the detection advantage value of the attack angle can be determined at least based on the radar's maximum detection angle and the attack angle in the initial state data.

[0078] In an optional example, the detection advantage function of the attack angle can be determined. Based on the detection advantage function of the attack angle and the attack angle in each set of initial state data, the detection advantage value of the attack angle is determined. For example, the detection advantage function of the attack angle is shown in equation (1) below:

[0079]

[0080] Where, δ Rmax This represents the radar's maximum detection angle; φ0 is an adjustable parameter. The attack angle is the angle in each set of initial state data.

[0081] Step S32: Determine the threat advantage value of the attack angle based at least on the angle corresponding to the maximum engagement area, the missile lock-on angle, the angle of the no-escape attack zone, and the attack angle.

[0082] In this embodiment, the threat advantage value of the attack angle can be determined based at least on the angle corresponding to the maximum engagement area, the missile lock-on angle, the angle of the no-escape attack zone, and the attack angle in each set of initial state data.

[0083] In an optional example, the threat advantage function of the attack angle can be determined. Based on the threat advantage function of the attack angle and the attack angle in each set of initial state data, the threat advantage value of the attack angle is determined. For example, the threat advantage function of the attack angle is shown in equation (2) below:

[0084]

[0085] Where, δ R δ represents the angle corresponding to the maximum combat zone. MK Indicates the missile lock-on angle, δ MO This indicates the angle of the no-escape attack zone; φ1 and φ2 are adjustable parameters. The attack angle is the angle in each set of initial state data.

[0086] Step S33: Determine the threat advantage value of the escape angle, at least based on the escape angle.

[0087] In this embodiment, the threat advantage value of the escape angle can be determined from at least the escape angle in each set of initial state data.

[0088] In an alternative example, a threat advantage function for the escape angle can be determined. Based on the threat advantage function for the escape angle and the escape angle in each set of initial state data, the threat advantage value of the escape angle is determined. For example, the threat advantage function for the escape angle is shown in equation (3) below:

[0089]

[0090] Wherein, φ3 is an adjustable parameter. Let be the escape angle in each set of initial state data, where

[0091] Step S34: Determine the angle advantage value based at least on the detection advantage value of the attack angle, the threat advantage value of the attack angle, and the threat advantage value of the escape angle.

[0092] In this embodiment, after determining the detection advantage value of the attack angle, the threat advantage value of the attack angle, and the threat advantage value of the escape angle, the angle advantage value can be determined at least based on the detection advantage value of the attack angle, the threat advantage value of the attack angle, and the threat advantage value of the escape angle.

[0093] In an alternative example, the joint threat advantage function for the two angles is shown in equation (4) below:

[0094]

[0095] Where, α ATA and α AA These represent the factors influencing the attack angle and escape angle, respectively, with the constraint condition: α ATA +α AA =1.

[0096] Thus, the overall angular advantage function is shown in equation (5) below, and the angular advantage value can be obtained based on this overall angular advantage function.

[0097]

[0098] Where, α det and α thr These represent the factors influencing detection advantage and threat advantage, respectively, with the constraint: α det +α thr =1.

[0099] Step S35: Determine the detection advantage value of the distance based at least on the radar's maximum detection range, the missile lock-on range, and the relative distance.

[0100] In this embodiment, the detection advantage and threat advantage based on range are primarily considered in relation to radar and missiles. This embodiment can determine the range detection advantage value based at least on the radar's maximum detection range, the missile's lock-on range, and the relative distances in each set of initial state data.

[0101] In an alternative example, a probe dominance function for distance can be determined, based on the probe dominance function and the relative distance in each set of initial state data, to determine the probe dominance value for distance. For example, the probe dominance function for distance is shown in equation (6) below:

[0102]

[0103] Among them, D Rmax D represents the maximum detection range of the radar. Lock This indicates the missile lock-on distance, φ4 is an adjustable parameter, and d is the relative distance in each set of initial state data.

[0104] Step S36: Determine the threat advantage value of the distance based at least on the missile's maximum attack range, no-escape range, minimum attack range, and the relative distance.

[0105] In this embodiment, the threat advantage value of distance can be determined based at least on the missile's maximum attack range, no-escape range, minimum attack range, and the relative distance in each set of initial state data.

[0106] In an alternative example, a distance threat advantage function can be determined, based on the distance threat advantage function and the relative distance in each set of initial state data, to determine the distance threat advantage value. For example, the distance threat advantage function is shown in equation (7) below:

[0107]

[0108] Among them, D Mmax D represents the missile's maximum attack range. Mnoesp D represents the non-escape distance. Mmin This represents the minimum attack distance, φ5 is an adjustable parameter, and d is the relative distance in each set of initial state data.

[0109] Step S37: Determine the distance advantage value based at least on the detection advantage value and the threat advantage value of the distance.

[0110] In this embodiment, after determining the detection advantage value and the threat advantage value of the distance, the distance advantage value can be determined at least based on the detection advantage value and the threat advantage value of the distance.

[0111] In an alternative example, the total distance advantage function is shown in equation (8) below, and the distance advantage value can be obtained based on this total distance advantage function.

[0112]

[0113] in, and These represent the factors influencing detection advantage and threat advantage, respectively, with the following constraints:

[0114] Step S38: Determine the potential energy advantage value based at least on the optimal flight altitude, the maximum flight altitude, and the relative flight altitude.

[0115] In this embodiment, energy during air combat is expressed as potential energy (e.g., expressed as relative flight altitude) and kinetic energy (e.g., expressed as relative flight speed). Regarding potential energy advantage, this embodiment can determine the potential energy advantage value based at least on the optimal flight altitude, maximum flight altitude, and the relative flight altitude in each set of initial state data.

[0116] In an alternative example, a potential advantage function can be determined, based on the potential advantage function and the relative flight altitude in each set of initial state data, to determine the potential advantage value. For example, the potential advantage function is shown in equation (9) below:

[0117]

[0118] Among them, H opt H represents the optimal flight altitude.max This represents the maximum flight altitude, and h is the relative flight altitude in each set of initial state data.

[0119] Step S39: Determine the kinetic energy advantage value based at least on the kinetic energy of the opponent's aircraft and the kinetic energy of the first friendly aircraft determined based on the flight speed of the first friendly aircraft.

[0120] In this embodiment, regarding the kinetic energy advantage, the kinetic energy advantage value can be determined by at least the kinetic energy of the opposing aircraft and the kinetic energy of the first friendly aircraft; wherein, the kinetic energy E of the first friendly aircraft is determined based on the flight speed v of the first friendly aircraft in each set of initial state data: E = v 2 / 2g. In an optional example, the kinetic energy of the opposing aircraft is determined based on the speed of the opposing aircraft in each set of initial state data.

[0121] In one alternative example, a kinetic energy dominance function can be determined, and based on this function, a kinetic energy dominance value can be determined. For example, the kinetic energy dominance function is shown in equation (10) below:

[0122]

[0123] Among them, E T φ represents the kinetic energy of the opposing aircraft, E represents the kinetic energy of the first friendly aircraft, and φ6 is an adjustable parameter.

[0124] Step S310: Determine the energy advantage value based at least on the potential energy advantage value and the kinetic energy advantage value.

[0125] In this embodiment, after determining the potential energy advantage value and the kinetic energy advantage value, the energy advantage value can be determined based at least on the potential energy advantage value and the kinetic energy advantage value.

[0126] In an alternative example, the total energy advantage function is shown in equation (11) below, and the energy advantage value can be obtained based on this total energy advantage function.

[0127]

[0128] Where, β h and β v These represent the influencing factors of potential energy advantage and kinetic energy advantage, respectively, with the constraint condition: β h +β v =1.

[0129] In conjunction with the above embodiments, in an optional implementation, step S24 can be specifically obtained by the following formula (12):

[0130]

[0131] in, As an indicator of combat advantage, The angle advantage value, This is the distance advantage value. λ0, λ1, and λ2 are the energy advantage values, and the weighting coefficients for the contribution of each component in balancing the final combat advantage index.

[0132] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of calculating combat advantage indicators and generating a state pool, as shown in an embodiment of the present invention. Figure 3 In this process, at least by calculating the combat advantage indicators from the speed, altitude, and distance in each set of initial state data, it is possible to obtain the initial state data for each set. Difficulty score ψ i (i.e., combat advantage index), and then based on the combat advantage index ψ i The system automatically sorts multiple sets of initial state data from simple to complex, forming a difficulty-guided state pool. This difficulty-guided state pool facilitates the training process and optimizes learning efficiency by gradually introducing increasingly complex initial state data into the deep reinforcement learning SAC model to be trained.

[0133] In conjunction with the above embodiments, in one implementation, the present invention also provides an intelligent air combat decision-making method based on curriculum reinforcement learning. In this embodiment, step S14 may specifically include steps S41 to S43:

[0134] Step S41: The target generator generates an intermediate target angle and sets an intermediate task target for the policy network to achieve the intermediate target angle.

[0135] This embodiment maintains a pair of adaptive angle target generators and target discriminators. In air combat scenarios, a key parameter affecting mission difficulty is the aircraft's angle of attack δ. ATA It has been shown to significantly impact mission complexity. In this embodiment, the mission objective is to control the enemy aircraft within the weapons engagement zone; therefore, this embodiment uses the angle of attack as the range angle of the weapons engagement zone. The ultimate mission objective is to directly train the agent to achieve an attack angle equal to the range angle of the weapon engagement zone. This is typically quite challenging and time-consuming; therefore, an intermediate target angle is introduced in this embodiment. As accessible intermediate task objectives, these objectives can more effectively guide the agent and accelerate the learning process. Therefore, this embodiment generates intermediate target angles through a target generator, defining an intermediate task objective for the policy network of the deep reinforcement learning SAC model to be trained, which aims to achieve the intermediate target angle.

[0136] Step S42: During the training process of the deep reinforcement learning SAC model to be trained, when the target discriminator determines that the attack angle of the first friendly aircraft reaches the intermediate target angle, the target generator determines the updated intermediate target angle based at least on the intermediate target angle and the difference between the intermediate target angle and the range angle of the weapon engagement zone.

[0137] In this embodiment, during the training process of the deep reinforcement learning SAC model to be trained, when the target discriminator determines that the attack angle of the first friendly aircraft reaches the intermediate target angle, the target generator can update the intermediate target angle: at least based on the intermediate target angle and the difference between the intermediate target angle and the range angle of the weapon engagement zone, the updated intermediate target angle is determined. Wherein, the intermediate target angle is greater than the range angle of the weapon engagement zone.

[0138] In an alternative example, the target generator can obtain the updated intermediate target angle based on the following equation (13).

[0139]

[0140] Where α is a decay factor that gradually reduces the impact of each adjustment. Let f(Δδ) be the intermediate target angle, and f(Δδ) represent the amount of angle reduction. Δδ is the difference between the intermediate target angle and the range angle of the weapon engagement zone, β is the attenuation constant, and k is the iteration index determined by the target discriminator, which can ensure a smooth reduction in angle adjustment when the agent approaches the target.

[0141] In one example, the target discriminator can evaluate the training process of the deep reinforcement learning SAC model by collecting simulation data (such as replay data from the most recent k rounds) in the air combat simulation environment to determine whether the attack angle of the first friendly aircraft has reached the intermediate target angle. If the intermediate target angle has been reached, the target generator will input the updated intermediate target angle into the policy network of the deep reinforcement learning SAC model to be trained in order to formulate a new intermediate mission target.

[0142] Step S43: The target generator sets a new intermediate mission objective for the strategy network, with the attack angle reaching the updated intermediate target angle, until the final mission objective is set for the strategy network, with the attack angle reaching the range angle of the weapon engagement zone.

[0143] In this embodiment, after the target generator obtains the updated intermediate target angle, it can formulate a new intermediate task objective for the policy network of the deep reinforcement learning SAC model to be trained, so that the attack angle reaches the updated intermediate target angle. This guides the agent to transition from a simple task to a complex task. The target generator continues to adjust until it converges to the range angle of the weapon engagement zone, that is, until the final task objective of the policy network is formulated to reach the range angle of the weapon engagement zone, which represents the final challenge faced by the agent.

[0144] In this embodiment, the difficulty of the target task is dynamically adjusted according to the model's training progress. Intermediate task targets are used to gradually guide the model to approach the final target. Specifically, this embodiment refines task parameters (adjusting the target attack angle) to achieve effective learning. The target generator gradually increases the task difficulty in a controlled manner based on the performance of the policy network. This dynamic adjustment mechanism ensures that the model can gradually master complex tactical behaviors during training, improving its tactical decision-making ability. The aim is to guide the agent to complete increasingly challenging tasks using an incremental difficulty adjustment strategy.

[0145] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the progressive task generation process according to an embodiment of the present invention. Figure 4 This paper proposes an adaptive target angle generation module. In this module, the target discriminator acquires replay data from an air combat simulation environment and determines whether the attack angle has reached the intermediate target angle based on the replay data. Should At the intermediate target angle In this case, the target generator generates a new intermediate target angle. New intermediate target angle The input is fed into the UCAV agent for further training, with continuous adjustments made until the angle of attack converges to the final target angle (i.e., the range angle of the weapon engagement zone). This represents the ultimate challenge faced by UCAV agents.

[0146] In conjunction with the above embodiments, in one implementation, the present invention also provides an intelligent air combat decision-making method based on curriculum reinforcement learning. In this method, step S13 may specifically include steps S51 and S52:

[0147] Step S51: Based on the most recent K current actions of the first friendly aircraft output by the policy network in the deep reinforcement learning SAC model to be trained, perform simulation in the air combat simulation environment to obtain the win rate of the first friendly aircraft in the most recent K rounds.

[0148] In this embodiment, after obtaining the current action of the first friendly aircraft output by the policy network in the deep reinforcement learning SAC model to be trained, the current action of the first friendly aircraft output by the policy network in the deep reinforcement learning SAC model to be trained can be input into the air combat simulation environment to perform simulation and obtain simulation results. The simulation results can include at least one of the following: policy reward, the win or loss result of the first friendly aircraft and the opponent's aircraft in this round.

[0149] In this embodiment, the most recent K current actions of the first friendly aircraft, output by the policy network in the deep reinforcement learning SAC model to be trained, can be used to perform simulations in an air combat simulation environment to obtain the win rate of the first friendly aircraft in the most recent K rounds. For example, by simulating the most recent 10 current actions, it can be determined that the first friendly aircraft won 6 out of the most recent 10 rounds, and the win rate of the most recent 10 rounds is 60%.

[0150] Step S52: If the win rate of the first home plane in the most recent K rounds exceeds the win rate threshold, the progressive sliding buffer reads and stores M sets of new initial state data from the difficulty-guided state pool in an increasing order of difficulty, and discards M sets of old initial state data in an increasing order of difficulty.

[0151] In this embodiment, a win rate threshold (which can be freely set) is preset, and this win rate threshold determines the update frequency of the progressive sliding buffer. This embodiment can control the update frequency of the progressive sliding buffer through a win rate threshold. Whenever the win rate of the first player's aircraft in the most recent K rounds exceeds the win rate threshold, the initial state data in the progressive sliding buffer is updated: the progressive sliding buffer reads and stores M sets of new initial state data from the difficulty-guided state pool in an increasing order of difficulty (e.g., in descending order of corresponding combat advantage indicators), and discards M sets of old initial state data in an increasing order of difficulty, to ensure that the progressive sliding buffer always stores L sets of initial state data; L and M are natural numbers greater than 0, with L greater than M, and can be freely set.

[0152] This embodiment designs a progressive sliding buffer mechanism to dynamically adjust the training sample state pool: during training, more complex combat scenarios are gradually introduced based on the current learning state of the model, ensuring the diversity of training samples while avoiding overly simple samples from affecting the training effect. This mechanism continuously optimizes the quality of samples during training through a sliding window approach, reducing the impact of invalid samples and improving training efficiency.

[0153] In one embodiment, such as Figure 5 As shown, Figure 5 This is a schematic diagram illustrating the update of a progressive sliding buffer according to an embodiment of the present invention. Figure 5In [reference], a progressive sliding buffer module is proposed. In the training scheduler, the progressive sliding buffer module maintains a dynamic state buffer (i.e., the progressive sliding buffer), whose function is similar to a sliding window. The capacity of the progressive sliding buffer module is fixed at N, and each update integrates k s new and more difficult data points (i.e., k s sets of new initial state data with higher representational difficulty), while discarding k' s (which can be the same as or different from k s ) of the oldest and less difficult entries (i.e., k' s sets of the oldest initial state data with lower representational difficulty). Specifically, the state of the progressive sliding buffer at time t is represented as The update of the progressive sliding buffer is controlled by a sliding trigger. When the win rate W t of the first friendly aircraft exceeds the win rate threshold θ s1 , the sliding trigger will be activated: triggering the selection of new and more challenging states from the difficulty-guided state pool, ensuring that the policy network gradually encounters increasingly difficult scenarios, thus supporting effective curriculum learning in air combat missions.

[0154] Combined with the above embodiments, in one implementation, the present invention also provides an air combat intelligent decision-making method based on curriculum reinforcement learning. In this method, "sampling the current initial state data from the progressive sliding buffer" in step S15 above specifically may include step S61:

[0155] Step S61: Sampling the current initial state data from the progressive sliding buffer or a random sampling generator through a dynamic hybrid sampling strategy.

[0156] In this embodiment, a dynamic hybrid sampling strategy is used to balance exploration and exploitation during training: instead of only sampling the current initial state data from the progressive sliding buffer, the current initial state data is sampled from the progressive sliding buffer or a random sampling generator through a dynamic hybrid sampling strategy.

[0157] In an optional example, the current initial state data can be sampled from the sliding buffer with probability p and from the random sampling generator (i.e., the random state sampler oracle(·)) with probability 1 - p, where 0 < p < 1. This method allows the agent to explore new states while exploiting the stored experience from the progressive sliding buffer , and these experiences are dynamically adjusted to adapt to the agent's evolving skill level, thus promoting a steady improvement in performance.

[0158] In this embodiment, a hybrid sampling strategy is adopted, which combines sampling from a progressive sliding buffer with random sampling to improve the balance between exploration and utilization. In this way, the agent can maintain the utilization of known tasks while exploring new combat scenarios, thereby rapidly improving its tactical response capabilities.

[0159] In conjunction with the above embodiments, in one implementation, the present invention also provides an intelligent air combat decision-making method based on curriculum reinforcement learning. In this method, step S16 may specifically include steps S71 to S74:

[0160] Step S71: Input the current actions and current initial state data of the first friendly aircraft into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the intermediate mission objective, and update the network parameters of the deep reinforcement learning SAC model to be trained after each simulation.

[0161] In this embodiment, the current actions and corresponding initial state data of the first friendly aircraft output by the policy network of the deep reinforcement learning SAC model to be trained can be input into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the corresponding intermediate mission objective, and update the network parameters of the deep reinforcement learning SAC model to be trained after each simulation.

[0162] Step S72: If the first friendly aircraft fails to reach the intermediate mission objective, new first initial state data is sampled from the progressive sliding buffer. The new first initial state data is input into the policy network in the deep reinforcement learning SAC model to be trained to obtain a new first action of the first friendly aircraft. The simulation is performed based on the new first initial state data and the new first action to determine that the first friendly aircraft has reached the intermediate mission objective.

[0163] In this embodiment, if it is determined that the first friendly aircraft has not reached the intermediate mission objective, new first initial state data is sampled from the progressive sliding buffer. This new first initial state data is a new set of initial state data. The new first initial state data is input into the policy network of the deep reinforcement learning SAC model to be trained to obtain the new first action of the first friendly aircraft. The new first action of the first friendly aircraft output by the policy network of the deep reinforcement learning SAC model to be trained and the corresponding new first initial state data are input into the air combat simulation environment to perform simulation, and determine whether the first friendly aircraft has reached the corresponding intermediate mission objective, until it is determined that the first friendly aircraft has reached the corresponding intermediate mission objective.

[0164] Step S73: If the first friendly aircraft reaches the intermediate mission objective, determine a new intermediate mission objective, sample new second initial state data from the progressive sliding buffer, and input the new second initial state data into the policy network in the deep reinforcement learning SAC model to be trained to obtain a new second action of the first friendly aircraft.

[0165] In this embodiment, when it is determined that the first friendly aircraft has reached the corresponding intermediate mission objective, a new intermediate mission objective is determined (e.g., the objective generator formulates a new intermediate mission objective for the policy network of the deep reinforcement learning SAC model to be trained). New second initial state data is sampled from the progressive sliding buffer. This new second initial state data is a new set of initial state data, which is different from the new first initial state data. The new second initial state data is input into the policy network of the deep reinforcement learning SAC model to be trained to obtain a new second action of the first friendly aircraft.

[0166] Step S74: Input the new second action and new second initial state data of the first friendly aircraft into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the new intermediate mission objective, until the simulation determines that the first friendly aircraft has achieved the final mission objective, and obtain the trained deep reinforcement learning SAC model.

[0167] In this embodiment, after obtaining the new second action of the first friendly aircraft output by the policy network of the deep reinforcement learning SAC model to be trained, the new second action of the first friendly aircraft and the corresponding new second initial state data are input into the air combat simulation environment to perform simulation and determine whether the first friendly aircraft has achieved the corresponding new intermediate mission objective. If it is determined that the first friendly aircraft has not achieved the new intermediate mission objective, new second initial state data is resampled from the progressive sliding buffer. The resampled new second initial state data is input into the policy network of the deep reinforcement learning SAC model to be trained to obtain the new second action of the first friendly aircraft. The newly obtained new second action of the first friendly aircraft and the corresponding resampled new second initial state data are input into the air combat simulation environment to perform simulation and determine whether the first friendly aircraft has achieved the corresponding new intermediate mission objective, until it is determined that the first friendly aircraft has achieved the corresponding new intermediate mission objective. This process is repeated through multiple simulation training iterations until it is determined through simulation that the first friendly aircraft has achieved the final mission objective, resulting in a trained deep reinforcement learning SAC model.

[0168] In conjunction with any of the above embodiments, in one embodiment, the deep reinforcement learning SAC model constructed in this embodiment is an algorithm based on policy gradient, which achieves stable learning and better exploration by maximizing the expected reward and entropy of the policy. The objective function of the deep reinforcement learning SAC model mainly consists of two parts: policy entropy. and expected returns This algorithm introduces a soft Q function, which can be summarized as follows:

[0169] Where α is a temperature parameter used to balance the effects of reward and entropy; s t For state variables, i.e., the current initial state data; a t This is the action vector, representing the current action of the first friendly aircraft.

[0170] The deep reinforcement learning SAC model in this embodiment includes two value networks (a base value network and a target value network) with the same network structure but different parameters, and a policy network. The policy network is a multilayer perceptron (MLP) with 128, 128 hidden layers and ReLU activation function. The value network has a similar structure to the policy network. During model training, this embodiment can use the Adam optimizer, and parameter updates are performed using gradient descent, updating the network parameters by minimizing the loss function.

[0171] In one embodiment, the state variable s t The state space can be constructed based on the aircraft's six degrees of freedom (6-DOF) motion characteristics, including translational motion (position and velocity) and rotational motion (Euler angles and angular velocity). These variables enable the agent to accurately capture the aircraft's spatial position and orientation, thus allowing the state space to effectively represent and process key state information in dynamic air combat scenarios. Although continuous state spaces are typically represented by basic motion parameters, these parameters are affected by translational and rotational effects, making it difficult to accurately capture the relative relationships between agents. Therefore, in this embodiment, the state variables are mainly defined by the aircraft's basic state variables and relative variables between aircraft, thereby more accurately describing their relative positions and orientations. In this embodiment, the state consists of two consecutive frames, and the motion state vector in each frame is defined as follows:

[0172]

[0173] Where h is the altitude of the first aircraft, v n v e and v d Let a represent the velocity components of the first aircraft in the north, east, and ground-to-air directions, respectively. x a y and a zLet denoted as the acceleration component of the first aircraft, ψ as the yaw angle of the first aircraft, θ as the pitch angle of the first aircraft, φ as the roll angle of the first aircraft, α as the angle of attack of the first aircraft, and β as the sideslip angle of the first aircraft. These represent the rates of change of yaw, pitch, and roll angles, respectively. Represents the relative variables between aircraft, including the relative distance d. rel Attack angle δ ATA escape angle σ AA and relative velocity υ rel .like Figure 2 As shown, the aircraft's center of mass is connected by the line of sight (LOS line), which is also used to determine the relative distance d. rel Escape angle σ AA Defined as LOS line and target aircraft ( Figure 2 The angle between the velocity vectors of the blue machine (in the image) and the attack angle δ. ATA Indicates LOS line and attack aircraft ( Figure 2 The angle between the velocity vectors of the red engine (in the equation). The following equation defines the calculation of these two angles:

[0174]

[0175] Where, vector r rel Representing relative position, the aircraft's total velocity υ is characterized as the velocity vector v = [υ n υ e υ d ] T The 2-norm, where v n v e and v d Let the velocity components representing the north, east, and directions relative to the ground be respectively expressed as:

[0176] In one alternative embodiment, the Weapons Engagement Zone (WEZ) is defined as a cone-shaped area with a cone angle of 30 degrees and a slant distance of 1 kilometer. Aircraft entering this area will be subject to lock-on and attack by enemy missiles, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of a weapons engagement zone shown in an embodiment of the present invention. Figure 6 Considering the relative distance r between the two aircraft, the thresholds for WEZ are d1 = 200m and d2 = 1000m. The calculation process is as follows:

[0177]

[0178] In one embodiment, the action vector a tIncludes four consecutive control commands: aileron deflection command elevator deflection command rudder deflection command and throttle command The first three commands control the aircraft's roll, pitch, and yaw, respectively, achieving precise attitude adjustments. Simultaneously, the throttle command regulates engine power, thereby managing the aircraft's speed and energy. Therefore, the motion vector at time step t can be represented as...

[0179] In one embodiment, during an autonomous air combat mission, a reward function provides feedback to the agent's actions, guiding it to achieve mission objectives. The primary objective is to rapidly and strategically maneuver the unmanned combat aircraft to force enemy aircraft into the WEZ (Weather Zone) to ensure optimal engagement conditions. The reward function in this embodiment is designed with process rewards (short-term rewards) and terminal rewards (long-term rewards), serving as signals for the agent's learning and decision-making at different stages of the mission. A process reward is calculated at each decision step to assess the aircraft's current position and engagement situation. This reward incorporates factors such as relative distance to the enemy aircraft, angle of attack, escape angle, and time penalty to incentivize efficient mission completion.

[0180] In a specific example, the process reward is shown in the following formula, where c1, c2, c3, and c4 are weighting coefficients that balance the contribution of each sub-reward item:

[0181]

[0182] in, This is a distance bonus designed to encourage aircraft to close the distance to enemy aircraft and maintain a tactical advantage by keeping the effective engagement range intact. Where ρ1 is the sensitivity coefficient, d′ t d represents the distance to the previous frame. t This represents the distance at the current time step t. The function is bounded, and the reward provided decreases as the distance approaches the optimal engagement range.

[0183] As an attack angle bonus, the objective is to minimize the attack angle, incentivizing the aircraft to effectively align itself with the opponent's aircraft. Where ρ2 is the sensitivity coefficient, δ′ t δ represents the angle of attack in the previous frame. t It is the attack angle at the current time step t.

[0184] An escape angle bonus is awarded to promote a favorable relative bearing and help the aircraft establish an effective tactical position. Where ρ3 is the sensitivity coefficient, σ′ tIt is the escape angle of the previous frame, σ t It is the escape angle at time t.

[0185] tanh(-t) is a time penalty term that penalizes the agent for extending the task time, thereby incentivizing it to complete the task objective quickly. The longer the agent spends, the heavier the penalty, thus cultivating its preference for taking swift and decisive action.

[0186] In this embodiment, the terminal reward is determined based on the final state of the round task, providing a high reward or penalty based on the specific result of each round. The terminal reward includes the following conditions:

[0187] a. If your aircraft successfully enters the WEZ area first, you will receive a reward of +50;

[0188] b. If the opposing aircraft enters the WEZ area first, it will be penalized with -50.

[0189] c. If the distance between our aircraft and the enemy aircraft exceeds a predefined threshold, indicating disengagement, a penalty of -50 will be applied.

[0190] d. If the altitude of one's own aircraft exceeds the permitted flight range, a penalty of -50 will be imposed;

[0191] e. If the task duration exceeds the final time threshold, a penalty of -50 will be imposed.

[0192] In this embodiment, the combination of short-term rewards (tactical positioning and engagement indicators) and long-term rewards (mission completion status) can guide UCAV agents to balance immediate actions with overall mission objectives. This dual structure can achieve optimal decision-making, thereby improving mission efficiency and tactical effectiveness.

[0193] In conjunction with the above embodiments, in one embodiment, this embodiment can train the deep reinforcement learning SAC model to be trained based on the aforementioned state variables, action vectors, and rewards. The training process includes at least the following steps:

[0194] 1. Through random network parameters ω, ω * and θ for the basic value network Q(s) t a t |ω Q ), Target Value Network Q * (s t a t |ω Q* ) and policy network π(a t |s t Initialize; where the policy network π(a) is initialized. t |s t ): Input is state s t The output is action a.t ; Basic value network Q(s) t a t |ω Q ): Input is state s t and action a t The output is the Q-value; the target value network Q * (s t a t |ω Q* ): Similar in structure to the base value network, used for stable training, and synchronized with the base value network through soft updates. Initialize the experience replay buffer R to store the agent's sampled experience data in the environment (current state, current action, reward, next state);

[0195] 2. Obtain the state vector s of the air combat simulation environment at time t. t If it is the initial time, then t = 0, and the initial state is s. t From the progressive sliding buffer via the hybrid sampler Alternatively, it can be obtained from random sampling in Oracle(·);

[0196] 3. Set the initial state s t Input policy network π(a) t |s t ), to obtain the output a t In an alternative embodiment, action a can be selected. t ~Π θ (·|s t )+ε, where ε is the noise term used to explore the environment; that is, based on the policy network π(a t |s t The output of ) plus the noise term yields action a. t ;

[0197] 4. Obtain the aircraft control actions (i.e., a) t The environment is input and a simulation is performed to obtain the state s of the next frame. t+1 Reward value r t and (s t a t r t s t+1 Store in the experience replay buffer R;

[0198] 5. If the number of samples in the experience replay buffer R is greater than the batch size, then randomly sample an N tuple {(s) from the experience replay buffer. i a i r i s i+1 )} i＝1，...，NThe learning process involves calculating the target Q-value using a target value network. The calculation process is as follows:

[0199] a i+1 ~Π θ (·|s s+1 );

[0200] Where, r i For rewards, γ is the influencing factor, and α is the temperature coefficient of the entropy regularization term, used to control the balance between exploration and exploitation. This is achieved through the loss function. The underlying value network is updated. Then, a reparameterized sampling action is used. Through the loss function Update the policy network and perform a soft update on the target value network: ω * ←τω+(1-τ)ω * , where τ is the soft update coefficient;

[0201] 6. Update the environment state to the next state s t+1 Then return to step 2;

[0202] 7. If the simulation reaches a termination state (completion of the current intermediate task objective, aircraft altitude exceeding the limit, aircraft distance exceeding the limit, or time step exceeding the maximum value, etc.), the simulation environment is reset and the progressive sliding buffer is updated. Once the intermediate task objectives are achieved, the next round of training will begin, continuing until the training concludes.

[0203] In one embodiment, this embodiment proposes an intelligent air combat decision-making method based on curriculum reinforcement learning, and designs an automatic engagement curriculum learning framework, such as... Figure 7 As shown, Figure 7 This is a schematic diagram illustrating an automatic engagement curriculum learning framework according to an embodiment of the present invention. This embodiment combines dynamic curriculum learning and deep reinforcement learning techniques to effectively perform flight control and intelligent decision-making for a six-degree-of-freedom unmanned combat aerial vehicle. Figure 7 In this automatic combat course learning framework, the following are included:

[0204] Automated Combat Complexity Assessment: This embodiment designs a Combat Advantage Index (CSI) to quantify the difficulty (complexity) of a combat scenario by dynamically evaluating multiple key factors (such as engagement angle, energy advantage, and the positional relationship between the two sides). The CSI index includes the relative distance between the two aircraft, attack angle, and energy consumption, ensuring that the model gradually adapts to increasingly complex combat situations during training. Ultimately, it generates a difficulty-guided state pool (i.e., a state pool sorted by complexity), allowing for dynamic adjustment of training difficulty based on model feedback.

[0205] Progressive Sliding Buffer Mechanism: This embodiment designs a progressive sliding buffer mechanism to dynamically adjust the training sample state pool (i.e., the progressive sliding buffer). During training, more complex combat scenarios are gradually introduced based on the current learning state of the model, ensuring the diversity of training samples while avoiding overly simple samples from affecting the training effect. This mechanism continuously optimizes the quality of samples during training through a sliding window approach, reducing the impact of invalid samples and improving training efficiency.

[0206] Goal-driven progressive task generation: In this framework, the difficulty of the task increases progressively. The system dynamically adjusts the difficulty of the target task based on the model's training progress, using intermediate goals (such as adjusting the target's attack angle) to gradually guide the model towards the final goal (endgame objective). By designing an intermediate goal guidance mechanism, the difficulty of combat mission objectives (such as attack angle) is dynamically adjusted, gradually guiding the model to complete complex tactical decision-making tasks. This ensures that the model can gradually master complex tactical behaviors during training, improving its tactical decision-making capabilities.

[0207] Hybrid Sampling Strategy: This embodiment employs a hybrid sampling strategy that combines sampling from a progressive sliding buffer with random sampling to improve the balance between exploration and utilization. In this way, the model can both maintain its utilization of known tasks and explore new combat scenarios, rapidly improving its tactical response capabilities.

[0208] This embodiment, through the integrated optimization of the automatic engagement curriculum learning framework, including automated combat complexity assessment, mission objective generation, progressive buffering mechanism, and hybrid sampling strategy, can significantly accelerate the training process, reduce computational resource requirements, ensure the efficiency and adaptability of the training process, and effectively improve air combat decision-making capabilities while ensuring the quality of policy network decision-making.

[0209] In one embodiment, such as Figure 8 As shown, Figure 8 This is a schematic diagram illustrating the overall process of an intelligent air combat decision-making method based on curriculum reinforcement learning, as shown in an embodiment of the present invention. Figure 8 The method includes the following steps:

[0210] Step 1: Randomly initialize the training scenario and generate randomly initialized training scenario data, which should include at least: the initial latitude and longitude, initial speed, initial altitude and initial yaw angle of the first local aircraft, and the relative distance, relative speed, relative altitude, relative heading angle and relative bearing angle of the opponent aircraft relative to the first local aircraft, such as generating 10,000 sets of training scenario data.

[0211] Step 2: Calculate the CSI index of each initial state data corresponding to each training scenario data in the above steps, including the relevant angles, distances, energy consumption, etc. of the aircraft of both sides. Sort the 10,000 generated CSI indexes in descending order. The larger the value, the higher the advantage of the first aircraft and the lower the difficulty of the combat scenario. The 10,000 initial state data after sorting according to the 10,000 CSI indexes are stored in the difficulty-guided state pool.

[0212] Step 3: Generate a progressive sliding buffer with a capacity of k, design a sliding trigger update mechanism, calculate the win rate of the current first player's aircraft in the last k rounds, control the update frequency of the buffer through the win rate threshold, whenever the win rate reaches the specified threshold, progressively slide and update the initial state data in the progressive sliding buffer in an increasing manner from the difficulty-guided state pool. The initialization of the progressive sliding buffer is the k sets of initial state data sorted first in the difficulty-guided state pool.

[0213] Step 4: Design a mission objective generator, which includes a objective generator and an objective discriminator. Select the replay data from the most recent k rounds in the air combat simulation environment, perform calculations and determine whether the current first friendly aircraft has achieved the specified intermediate mission objective. If the objective is achieved, the objective generator will input a new intermediate mission objective into the agent model, guiding the agent to transition from simple tasks to complex tasks, until the final mission objective is input into the agent model.

[0214] Step 5: Train the deep reinforcement learning SAC algorithm. Its input includes the current state information of the aircraft of both sides, i.e., the current initial state data (such as position, speed, and altitude). The output is the strategy decision of the first friendly aircraft, i.e., the current action of the first friendly aircraft, including elevator, rudder, aileron, and throttle values. At the same time, a reward mechanism is designed to give a positive reward for successfully chasing the enemy aircraft and a negative reward for otherwise. During the training process, the enemy aircraft keeps flying straight. Through multiple training iterations, the pursuit strategy of the first friendly aircraft is gradually optimized.

[0215] Step Six: Use the target discriminator to judge the strategic performance of the current first friendly aircraft. If the pre-specified final mission objective has not been achieved, return to Step Four to continue training. If the final mission objective has been achieved, proceed to the next step.

[0216] Step 7: Deploy the optimized strategy model to the decision production unit. The decision production unit will make intelligent decisions by calling the optimal strategy in real time based on the current air combat situation.

[0217] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0218] Based on the same inventive concept, one embodiment of the present invention provides an intelligent air combat decision-making device based on curriculum reinforcement learning. (Reference) Figure 9 , Figure 9 This is a structural block diagram of an intelligent air combat decision-making device based on curriculum-based reinforcement learning, provided in an embodiment of the present invention. Figure 9 As shown, the air combat intelligent decision-making device based on curriculum reinforcement learning in this embodiment may include:

[0219] The indicator determination module is used to determine the combat advantage indicator corresponding to each set of initial state data based on multiple sets of initial state data. The combat advantage indicator represents the difficulty of the combat scenario for the first friendly aircraft.

[0220] The state pool storage module is used to sort the multiple sets of initial state data based on the combat advantage index and store the sorted multiple sets of initial state data into the difficulty-guided state pool.

[0221] The buffer update module is used to update the initial state data in the progressive sliding buffer from the difficulty-guided state pool in an increasing manner;

[0222] The goal setting module is used to dynamically set intermediate task goals for the policy network in the deep reinforcement learning SAC model to be trained. The intermediate task goals guide the policy network to transition from simple tasks to complex tasks until the final task goal is set for the policy network.

[0223] The action output module is used to sample the current initial state data from the progressive sliding buffer, input the current initial state data into the policy network in the deep reinforcement learning SAC model to be trained, and obtain the current action of the first local aircraft.

[0224] The model training module is used to train the deep reinforcement learning SAC model to be trained based at least on the intermediate mission objective, the current action of the first friendly aircraft, and the current initial state data, until the first friendly aircraft reaches the final mission objective, and obtain the trained deep reinforcement learning SAC model.

[0225] The decision output module is used to input the current air combat environment information into the policy network of the trained deep reinforcement learning SAC model and output the action information of the second friendly aircraft.

[0226] Optionally, each set of initial state data includes at least: the engagement angle, relative distance, relative flight altitude, and flight speed of the first friendly aircraft and the enemy aircraft; the index determination module includes:

[0227] The first determining module is used to determine the angle advantage value based on the engagement angle in each set of initial state data;

[0228] The second determining module is used to determine the distance advantage value based on the relative distance in each set of initial state data;

[0229] The third determining module is used to determine the energy advantage value based on the relative flight altitude and flight speed in each set of initial state data;

[0230] The fourth determining module is used to determine the combat advantage index corresponding to each set of initial state data based at least on the angle advantage value, the distance advantage value, and the energy advantage value, wherein the larger the value of the combat advantage index, the lower the difficulty of the combat scenario for the first friendly aircraft.

[0231] Optionally, the engagement angle includes: an attack angle and an escape angle; the first determining module includes:

[0232] The first advantage determination module is used to determine the detection advantage value of the attack angle based at least on the radar's maximum detection angle and the attack angle;

[0233] The second advantage determination module is used to determine the threat advantage value of the attack angle based at least on the angle corresponding to the maximum engagement area, the missile lock-on angle, the angle of the no-escape attack zone, and the attack angle.

[0234] The third advantage determination module is used to determine the threat advantage value of the escape angle, at least based on the escape angle.

[0235] The fourth advantage determination module is used to determine the angle advantage value based at least on the detection advantage value of the attack angle, the threat advantage value of the attack angle, and the threat advantage value of the escape angle;

[0236] The second determining module includes:

[0237] The fifth advantage determination module is used to determine the range detection advantage value based at least on the radar's maximum detection range, the missile lock-on range, and the relative range;

[0238] The sixth advantage determination module is used to determine the threat advantage value of the distance based at least on the missile's maximum attack range, no-escape range, minimum attack range, and the relative distance.

[0239] The seventh advantage determination module is used to determine the distance advantage value based at least on the detection advantage value of the distance and the threat advantage value of the distance;

[0240] The third determining module includes:

[0241] The eighth advantage determination module is used to determine the potential energy advantage value based at least on the optimal flight altitude, the maximum flight altitude, and the relative flight altitude;

[0242] The ninth advantage determination module is used to determine a kinetic energy advantage value based at least on the kinetic energy of the opponent's aircraft and the kinetic energy of the first friendly aircraft determined based on the flight speed of the first friendly aircraft.

[0243] The tenth advantage determination module is used to determine the energy advantage value based at least on the potential energy advantage value and the kinetic energy advantage value.

[0244] Optionally, intermediate task objectives are dynamically formulated for the policy network in the deep reinforcement learning SAC model to be trained. These intermediate task objectives guide the policy network from simple tasks to complex tasks until a final task objective is formulated for the policy network, including:

[0245] An angle generation module is used by the target generator to generate intermediate target angles, and to formulate intermediate task objectives for the policy network to achieve the intermediate target angles by attacking the intermediate target angles.

[0246] An angle update module is used to determine, during the training process of the deep reinforcement learning SAC model to be trained, when the target discriminator determines that the attack angle of the first friendly aircraft reaches the intermediate target angle, the target generator determines the updated intermediate target angle based at least on the intermediate target angle and the difference between the intermediate target angle and the range angle of the weapon engagement zone.

[0247] An angle setting module is used by the target generator to set a new intermediate mission objective for the strategy network, which is to set an attack angle that reaches the updated intermediate target angle, until the final mission objective is set for the strategy network, which is to set an attack angle that reaches the range angle of the weapon engagement zone.

[0248] Wherein, the angle of the intermediate target is greater than the range angle of the weapon engagement zone.

[0249] Optionally, the buffer update module includes:

[0250] The win rate determination module is used to perform simulation in an air combat simulation environment based on the most recent K current actions of the first aircraft output by the policy network in the deep reinforcement learning SAC model to be trained, and to obtain the win rate of the first aircraft in the most recent K rounds.

[0251] The update processing module is used to read and store M sets of new initial state data from the difficulty-guided state pool in an increasing manner when the win rate of the first local aircraft in the most recent K rounds exceeds the win rate threshold, and to discard M sets of old initial state data in an increasing manner.

[0252] Where K and M are natural numbers greater than 0.

[0253] Optionally, the motion output module includes:

[0254] The hybrid sampling module is used to sample the current initial state data from the progressive sliding buffer or random sampling generator using a dynamic hybrid sampling strategy.

[0255] Optionally, the model training module includes:

[0256] The simulation execution module is used to input the current actions and current initial state data of the first friendly aircraft into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the intermediate mission objective, and update the network parameters of the deep reinforcement learning SAC model to be trained after each simulation.

[0257] The task judgment module is used to sample new first initial state data from the progressive sliding buffer when the first home aircraft has not reached the intermediate task objective, input the new first initial state data into the policy network in the deep reinforcement learning SAC model to be trained, obtain a new first action of the first home aircraft, and so on until the simulation is performed based on the new first initial state data and the new first action to determine that the first home aircraft has reached the intermediate task objective.

[0258] The task update module is used to determine a new intermediate task objective when the first friendly aircraft reaches the intermediate task objective, sample new second initial state data from the progressive sliding buffer, input the new second initial state data into the policy network in the deep reinforcement learning SAC model to be trained, and obtain a new second action of the first friendly aircraft.

[0259] The training result determination module is used to input the new second action and new second initial state data of the first friendly aircraft into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the new intermediate mission objective, and so on until the simulation determines that the first friendly aircraft has achieved the final mission objective, thereby obtaining the trained deep reinforcement learning SAC model.

[0260] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the air combat intelligent decision-making method based on curriculum reinforcement learning as described in any of the above embodiments of the present invention.

[0261] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps of the air combat intelligent decision-making method based on curriculum reinforcement learning described in any of the above embodiments of the present invention.

[0262] For the apparatus embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually.

[0263] The above provides a detailed description of the air combat intelligent decision-making method, device, equipment, and medium based on course reinforcement learning provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for intelligent air combat decision-making based on curriculum reinforcement learning, characterized in that, The method includes: Based on multiple sets of initial state data, the combat advantage index corresponding to each set of initial state data is determined. The combat advantage index represents the difficulty of the combat scenario for the first friendly aircraft. Based on the combat advantage index, the multiple sets of initial state data are sorted, and the sorted multiple sets of initial state data are stored in the difficulty-guided state pool. The initial state data in the progressive sliding buffer is updated from the difficulty-guided state pool in an increasing manner; Intermediate task objectives are dynamically formulated for the policy network in the deep reinforcement learning SAC model to be trained. The intermediate task objectives guide the policy network from simple tasks to complex tasks until the final task objective is formulated for the policy network. The current initial state data is sampled from the progressive sliding buffer, and the current initial state data is input into the policy network in the deep reinforcement learning SAC model to be trained to obtain the current action of the first local aircraft. Based at least on the intermediate mission objective, the current actions of the first friendly aircraft, and the current initial state data, the deep reinforcement learning SAC model to be trained is trained until the first friendly aircraft reaches the final mission objective, thus obtaining a trained deep reinforcement learning SAC model. The current air combat environment information is input into the policy network of the trained deep reinforcement learning SAC model, and the action information of the second home aircraft is output. Specifically, intermediate task objectives are dynamically formulated for the policy network in the deep reinforcement learning SAC model to be trained. These intermediate task objectives guide the policy network from simple tasks to complex tasks until a final task objective is formulated for the policy network, including: The target generator generates intermediate target angles and sets intermediate task objectives for the policy network to achieve these intermediate target angles. During the training process of the deep reinforcement learning SAC model to be trained, when the target discriminator determines that the attack angle of the first friendly aircraft reaches the intermediate target angle, the target generator determines the updated intermediate target angle based at least on the intermediate target angle and the difference between the intermediate target angle and the range angle of the weapon engagement zone. The target generator sets a new intermediate mission objective for the strategy network, which is to achieve the updated intermediate target angle, until the final mission objective is set for the strategy network, which is to achieve the range angle of the weapon engagement zone. Wherein, the angle of the intermediate target is greater than the range angle of the weapon engagement zone.

2. The air combat intelligent decision-making method based on curriculum reinforcement learning according to claim 1, characterized in that, Each set of initial state data includes at least: the engagement angle, relative distance, relative flight altitude, and flight speed of the first friendly aircraft and the enemy aircraft; based on multiple sets of initial state data, the combat advantage indicators corresponding to each set of initial state data are determined, including: Based on the engagement angles in each set of initial state data, determine the angle advantage value; Based on the relative distances in each set of initial state data, a distance advantage value is determined; Based on the relative flight altitude and flight speed in each set of initial state data, the energy advantage value is determined; Based at least on the angle advantage value, the distance advantage value, and the energy advantage value, a combat advantage index corresponding to each set of initial state data is determined, wherein the larger the value of the combat advantage index, the lower the difficulty of the combat scenario for the first friendly aircraft.

3. The air combat intelligent decision-making method based on curriculum reinforcement learning according to claim 2, characterized in that, The engagement angles include: attack angle and escape angle; based on the engagement angles in each set of initial state data, an angle advantage value is determined, including: The detection advantage value of the attack angle is determined based at least on the radar's maximum detection angle and the attack angle. The threat advantage value of the attack angle is determined based at least on the angle corresponding to the maximum engagement area, the missile lock-on angle, the angle of the no-escape attack zone, and the attack angle. The threat advantage value of the escape angle is determined at least based on the escape angle; The angle advantage value is determined based at least on the detection advantage value of the attack angle, the threat advantage value of the attack angle, and the threat advantage value of the escape angle; Based on the relative distances in each set of initial state data, a distance advantage value is determined, including: The detection advantage value of the distance is determined based at least on the radar's maximum detection range, the missile's locking range, and the relative distance. The threat advantage value of the distance is determined based at least on the missile's maximum attack range, no-escape range, minimum attack range, and the relative distance. The distance advantage value is determined based at least on the detection advantage value and the threat advantage value of the distance; Based on the relative flight altitude and flight speed in each set of initial state data, the energy advantage value is determined, including: The potential energy advantage value is determined based at least on the optimal flight altitude, the maximum flight altitude, and the relative flight altitude. The kinetic energy advantage value is determined based at least on the kinetic energy of the opponent's aircraft and the kinetic energy of the first friendly aircraft determined based on the flight speed of the first friendly aircraft. The energy advantage value is determined based at least on the potential energy advantage value and the kinetic energy advantage value.

4. The air combat intelligent decision-making method based on curriculum reinforcement learning according to any one of claims 1 to 3, characterized in that, The initial state data in the progressive sliding buffer is updated from the difficulty-guided state pool in an increasing difficulty manner, including: Based on the most recent K current actions of the first-side aircraft output by the policy network in the deep reinforcement learning SAC model to be trained, the simulation is performed in the air combat simulation environment to obtain the win rate of the first-side aircraft in the most recent K rounds. If the win rate of the first home plane in the most recent K rounds exceeds the win rate threshold, the progressive sliding buffer reads and stores M sets of new initial state data from the difficulty-guided state pool in an increasing order of difficulty, and discards M sets of old initial state data in an increasing order of difficulty. Where K and M are natural numbers greater than 0.

5. The air combat intelligent decision-making method based on curriculum reinforcement learning according to any one of claims 1 to 3, characterized in that, The current initial state data is obtained by sampling from the progressive sliding buffer, including: The current initial state data is obtained by sampling from the progressive sliding buffer or random sampling generator using a dynamic hybrid sampling strategy.

6. The air combat intelligent decision-making method based on curriculum reinforcement learning according to any one of claims 1 to 3, characterized in that, The deep reinforcement learning SAC model to be trained is trained based at least on the intermediate mission objective, the current actions of the first friendly aircraft, and the current initial state data, until the first friendly aircraft reaches the final mission objective, resulting in a trained deep reinforcement learning SAC model, including: The current actions and initial state data of the first friendly aircraft are input into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the intermediate mission objective, and update the network parameters of the deep reinforcement learning SAC model to be trained after each simulation. If the first home aircraft fails to reach the intermediate mission objective, new first initial state data is sampled from the progressive sliding buffer, and the new first initial state data is input into the policy network in the deep reinforcement learning SAC model to be trained to obtain a new first action of the first home aircraft, until simulation is performed based on the new first initial state data and the new first action to determine that the first home aircraft has reached the intermediate mission objective. If the first local aircraft reaches the intermediate mission objective, a new intermediate mission objective is determined, new second initial state data is sampled from the progressive sliding buffer, and the new second initial state data is input into the policy network in the deep reinforcement learning SAC model to be trained to obtain a new second action of the first local aircraft. The new second action and new second initial state data of the first friendly aircraft are input into the air combat simulation environment to perform simulation, determine whether the first friendly aircraft has achieved the new intermediate mission objective, and continue until the simulation determines that the first friendly aircraft has achieved the final mission objective, thus obtaining the trained deep reinforcement learning SAC model.

7. An intelligent air combat decision-making device based on curriculum reinforcement learning, characterized in that, The device includes: The indicator determination module is used to determine the combat advantage indicator corresponding to each set of initial state data based on multiple sets of initial state data. The combat advantage indicator represents the difficulty of the combat scenario for the first friendly aircraft. The state pool storage module is used to sort the multiple sets of initial state data based on the combat advantage index and store the sorted multiple sets of initial state data into the difficulty-guided state pool. The buffer update module is used to update the initial state data in the progressive sliding buffer from the difficulty-guided state pool in an increasing manner; A target setting module is used to dynamically set intermediate task targets for the policy network in the deep reinforcement learning SAC model to be trained. These intermediate task targets guide the policy network from simple tasks to complex tasks, until a final task target is set for the policy network. A target generator generates intermediate target angles and sets intermediate task targets for the policy network where the attack angle reaches the intermediate target angle. During the training process of the deep reinforcement learning SAC model to be trained, if the target discriminator determines that the attack angle of the first friendly aircraft reaches the intermediate target angle, the target generator determines an updated intermediate target angle based at least on the intermediate target angle and the difference between the intermediate target angle and the range angle of the weapon engagement zone. The target generator sets new intermediate task targets for the policy network where the attack angle reaches the updated intermediate target angle, until a final task target is set for the policy network where the attack angle reaches the range angle of the weapon engagement zone; wherein the intermediate target angle is greater than the range angle of the weapon engagement zone. The action output module is used to sample the current initial state data from the progressive sliding buffer, input the current initial state data into the policy network in the deep reinforcement learning SAC model to be trained, and obtain the current action of the first local aircraft. The model training module is used to train the deep reinforcement learning SAC model to be trained based at least on the intermediate mission objective, the current action of the first friendly aircraft, and the current initial state data, until the first friendly aircraft reaches the final mission objective, and obtain the trained deep reinforcement learning SAC model. The decision output module is used to input the current air combat environment information into the policy network of the trained deep reinforcement learning SAC model and output the action information of the second friendly aircraft.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the air combat intelligent decision-making method based on curriculum reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the air combat intelligent decision-making method based on curriculum reinforcement learning as described in any one of claims 1 to 6.

Images