Unmanned aerial vehicle air combat confrontation method based on hierarchical reinforcement learning

Abstract

The invention provides an unmanned aerial vehicle air combat confrontation method based on hierarchical reinforcement learning, and belongs to the technical field of unmanned aerial vehicle control. According to the invention, the technical problem of improving the autonomous decision-making and control capability of the unmanned aerial vehicle in a complex air combat environment is solved. According to the technical scheme, the method comprises the following steps: S1, situation acquisition; s2, layered action space design; s3, training a layered reward function and a strategy; and S4, performing model deployment and real-time optimization. The method has the beneficial effects that a complex problem is decomposed into a plurality of sub-problems, and learning and decision making are carried out on different levels, so that the complexity can be better handled, the autonomy and adaptability of the unmanned aerial vehicle are improved, and the development of an unmanned aerial vehicle control technology is promoted; and the autonomous decision-making and control capability of the unmanned aerial vehicle in a complex air combat environment is obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a unmanned aerial vehicle air combat confrontation method based on hierarchical reinforcement learning. Background Art

[0002] In the field of UAV technology, the autonomy and adaptability of UAVs in complex environments have become a hot topic of research. Existing UAV control methods mostly rely on pre-set rules or simple feedback control, which often seem inadequate when faced with dynamic and changeable adversarial environments. Especially in adversarial environments, UAVs need to make fast and accurate decisions in real time to deal with complex tasks such as enemy attacks and obstacle avoidance. Traditional control algorithms have the disadvantages of slow response speed and poor adaptability when dealing with such problems.

[0003] In recent years, deep reinforcement learning, as an emerging machine learning method, has shown great potential in complex tasks by learning optimal strategies through interaction with the environment. As an effective deep reinforcement learning algorithm, the deterministic policy gradient (DDPG) algorithm has been shown to have good performance in decision-making problems in continuous action spaces. However, applying the DDPG algorithm to the control of drones in adversarial environments still faces many challenges, such as how to design effective state representation, action space, and how to construct a reasonable reward function.

[0004] The high dynamics and complexity of the air combat environment require UAVs to be able to quickly adapt to changing tactical situations, which requires a more flexible and efficient learning method to guide the UAV's decision-making process. Summary of the invention

[0005] The purpose of the present invention is to provide a UAV air combat confrontation method based on hierarchical reinforcement learning. As an advanced method, hierarchical reinforcement learning can better handle such complexity and improve the autonomy and adaptability of UAVs by decomposing complex problems into multiple sub-problems and performing learning and decision-making at different levels. Therefore, the present invention aims to solve the deficiencies in the prior art and promote the development of UAV control technology. The method of the present invention significantly improves the autonomous decision-making and control capabilities of UAVs in complex air combat environments.

[0006] The present invention is achieved by the following measures: a UAV air combat confrontation method based on hierarchical reinforcement learning, comprising the following steps:

[0007] S1: Situation acquisition, real-time acquisition of UAV flight status and target information through a variety of high-precision sensors and detection and recognition systems;

[0008] S2: Hierarchical action space design, fine-grained definition of the action space including speed, posture, path and tactical action;

[0009] S3: Hierarchical reward function and strategy training, using deep deterministic policy gradient algorithm combined with experience replay and target network technology to train the strategy network, and designing a hierarchical reward function that comprehensively considers task completion, survival time, energy consumption and tactical effect;

[0010] S4: Model deployment and real-time optimization, testing and evaluating drone performance in a simulated environment, and achieving online update optimization through wireless communication.

[0011] Furthermore, in step S1, situation acquisition and definition of a multi-level state representation of the drone include the following steps:

[0012] S11. Flight status monitoring: real-time acquisition of the UAV's flight altitude, speed, attitude and position information through high-precision sensors, altimeters, pitot tubes, inertial measurement units (IMUs) and global positioning systems (GPSs), ensuring the accuracy and real-time nature of the information.

[0013] S12, target recognition, real-time identification of the location and characteristics of enemy fighters, missiles and obstacles, use of radar, infrared sensors and visual systems to identify and track targets on the battlefield, and analysis of their characteristics for accurate classification and tracking;

[0014] S13. Battlefield situation analysis: integrate flight status and target information to form battlefield situation awareness. By analyzing terrain data, environmental scanning data and geographic information system (GIS) data, it can identify the distribution of obstacles and terrain features in the battlefield, while integrating the location information of friendly units.

[0015] Furthermore, in step S2, the hierarchical action space design, designing the hierarchical action space of the drone, includes the following steps:

[0016] S21. Action space definition, defining the action space of the drone, including speed change, attitude adjustment, flight path selection and tactical avoidance action. The speed control range is 0m / s to the maximum speed of 250m / s, and the acceleration change range is 0 to 100m / s2. The attitude adjustment includes pitch angle -30° to +30°, roll angle -60° to +60° and yaw angle. The flight path is selected from the preset path library, and the enemy air defense system and terrain obstacles are taken into account. Tactical avoidance actions include sharp ascent, descent, roll and sharp turn, and the execution time is within 1 to 5 seconds.

[0017] S22. Action parameter optimization. According to different tactical requirements, the action parameters are optimized to achieve fine control, including dynamic adjustment of the maximum overload limit and the minimum turning radius. The action space is designed to be continuous, allowing for subtle action adjustments, and a variety of tactical action combinations are designed.

[0018] Furthermore, in step S3, the hierarchical reward function and strategy training, designing the hierarchical reward function and training the drone strategy through the hierarchical reinforcement learning algorithm, includes the following steps:

[0019] S31. Reward function design. Design a hierarchical reward function, taking into account task completion, survival time, energy consumption and tactical effect. Task completion reward is based on the number of key task targets completed, survival time reward is calculated based on survival time, energy consumption reward is related to energy consumption during the task, and tactical effect reward is based on the number of successful avoidance of enemy attacks. Weights are assigned to each reward function. Task completion reward function: R task =∑(10·Number of completed tasks)

[0020] Survival time reward function: R survival = Survival time

[0021] Energy consumption reward function: Tactical effect reward function: R tactics =∑(5·Number of successful evasions of enemy attacks)

[0022] Comprehensive reward function

[0023] R total =w task ·R task +w survival ·R survival +w energy ·R energy +w tactics .R tactics

[0024] Among them, the weight distribution of each reward function is w task , w survival , w energy , w tactics

[0025] S32, policy network training, using deep deterministic policy gradient algorithm to train policy network, improving learning efficiency through experience replay technology, including initializing policy network and value network parameters, collecting initial data, performing exploration actions and storing experience, sampling experience from experience replay buffer to calculate target Q value

[0026] y = r + γ · CriticNetworkTarget (s′, ActorNetworkTarget (s′)), calculate the value network loss function At the same time, update the value network parameters and calculate the policy network gradient Update the policy network parameters and regularly use the soft update mechanism to update the target network parameters: θ target =τθ+(1-τ)θtarget , repeat iterative optimization until the policy network converges;

[0027] Where r is the reward, γ is the discount factor, s′ is the new state, CriticNetworkTarget and ActorNetworkTarget are the target networks of the value network and the policy network respectively, N is the batch size, and s i and a i are states and actions, θ critic is the value network parameter, and τ is a small positive number.

[0028] Furthermore, in step S4, model deployment and real-time optimization, applying the trained strategy to the actual drone and performing real-time optimization, includes the following steps:

[0029] S41, simulated environment test, testing the performance of the UAV in a simulated environment to evaluate its survivability and mission execution efficiency. The simulated environment includes a three-dimensional terrain model, enemy units with real flight performance parameters, a dynamic weather system, and electromagnetic interference. The difficulty and complexity can be adjusted dynamically, and it has multiple mission types and confrontation scenario simulations and emergency simulations;

[0030] S42, online update and optimization, which is achieved through wireless communication, allowing the UAV to adjust its strategy in real time based on feedback from the actual air combat environment;

[0031] S43. Actual environment application: deploy the strategy to actual drones for testing and application in actual air combat environments.

[0032] Furthermore, the present invention enables the UAV to identify and track enemy fighters, missiles and obstacles on the battlefield through advanced target recognition technology. Using radar, infrared sensors and visual systems, the UAV can detect targets and analyze their characteristics, such as radar reflection area and heat signal, to achieve accurate target classification and tracking. In addition to flight status monitoring and target recognition, the present invention also includes an in-depth analysis of the battlefield environment. By analyzing terrain data, environmental scanning data and geographic information system (GIS) data, the UAV can identify the distribution of obstacles and terrain features in the battlefield. At the same time, the location information of friendly units received through the communication system is also integrated into the battlefield situation, providing the UAV with a comprehensive battlefield perspective. Finally, the multi-level state fusion technology integrates the data of flight status monitoring, target recognition and battlefield situation analysis to form a comprehensive situation awareness model. This model not only enables the UAV to understand its own flight status at the micro level, but also understands the entire battlefield environment at the macro level, thereby achieving comprehensive situation awareness. On this basis, the decision support system provides real-time battlefield information and decision suggestions to the UAV operator, enhancing the autonomous decision-making ability of the UAV. This multi-level state representation mechanism is one of the important innovations of the present invention in the field of UAV control technology.

[0033] Hierarchical Action Space Design:

[0034] The hierarchical action space provides the UAV with the ability to perform fine control and tactical decision-making in complex battlefield environments. The following are the specific implementation steps and parameters:

[0035] Action space definition:

[0036] Speed ​​control: The speed control action space of the drone includes continuous adjustment capability from 0 m / s to a maximum speed of 250 m / s, and can achieve fast response with acceleration changes ranging from 0 to 100 m / s2.

[0037] Attitude adjustment: In terms of attitude adjustment, the drone is able to perform actions including pitch, roll and yaw, with the pitch angle varying from -30 degrees to +30 degrees, the roll angle varying from -60 degrees to +60 degrees, and the yaw angle varying from continuous rotation.

[0038] Flight path selection: The UAV can select the optimal flight path from a preset path library based on the battlefield environment and mission requirements. The path planning takes into account the coverage of the enemy's air defense system, terrain obstacles, etc.

[0039] Tactical evasive maneuvers: A series of tactical evasive maneuvers have been designed, such as sharp ascent, descent, roll and sharp turn. The execution time of the evasive maneuvers is between 1 and 5 seconds, ensuring that the drone can respond quickly to enemy attacks.

[0040] Action parameter optimization:

[0041] Parameter adjustment: In response to different battlefield situations and mission requirements, the drone's action parameters can be dynamically adjusted, such as adjusting the maximum overload limit, minimum turning radius, etc., to achieve more precise flight control.

[0042] Continuity design: The action space is designed to be continuous, allowing the drone to make infinitesimal action adjustments, such as completing a 0.5-degree attitude adjustment in 0.1 seconds, or adjusting the speed by 2 meters per second in 1 second.

[0043] Diversity Implementation: A variety of tactical action combinations are designed, such as "rolling and sharp turns", "rapid descent and climb", etc., to adapt to changing battlefield conditions and enemy attack patterns.

[0044] Action execution and feedback:

[0045] Real-time execution: The UAV’s action execution system can respond to action instructions defined in the action space in real time and quickly execute flight control and tactical actions.

[0046] Feedback adjustment: After executing an action, the drone's sensor system will provide real-time feedback on the execution results, such as changes in speed, posture, and position. The system will further optimize the action parameters based on the feedback results.

[0047] Through the above-mentioned hierarchical action space design, the present invention enables the UAV to perform fine flight control and tactical decision-making in a complex battlefield environment, improving the survivability and mission execution efficiency of the UAV. This design not only enhances the autonomy of the UAV, but also provides a new technical solution for the application of UAV in military and civilian fields.

[0048] A key innovation of the present invention is to build a highly realistic and configurable simulated air combat environment, providing a training platform close to actual combat for drones. In this environment, the three-dimensional terrain model is generated using geographic information system (GIS) data, and the terrain features such as mountains, plains and oceans are carefully reproduced. In addition, the enemy units are designed as fighter models with real flight performance parameters. They can maneuver at a maximum speed of 1,500 kilometers per hour, and the overload range is from +9G to -3G, ensuring the challenging and practical nature of the training environment. In order to further enhance the realism of the simulated environment, we have designed a dynamic weather system that can simulate extreme meteorological conditions such as strong winds with a wind speed of 25 meters per second, wind direction changes, thunderstorms, rain and fog. These weather conditions will affect the perception and flight performance of the drone, forcing it to adapt to different flight conditions during training. At the same time, electromagnetic interference in the battlefield is also simulated, which will affect the communication and radar detection capabilities of the drone, increasing the complexity of training. The simulation environment of the present invention also has the ability to dynamically adjust the difficulty and complexity to meet the needs of different training stages. The number of enemy units, attack frequency and accuracy can all be adjusted according to the training plan, thereby gradually increasing the challenge of training. The battlefield environment layout will also be updated regularly, including the redistribution of obstacles and changes in terrain to meet the needs of different training scenarios. In addition, we have designed a variety of mission types, such as reconnaissance, strike, escort, etc., each with its own specific environment and enemy unit configuration. The confrontation scenario simulation includes different attack modes and tactics of enemy fighters, such as dogfights, long-range missile attacks, etc., allowing drones to experience real air combat scenarios in a simulated environment. In order to test the adaptability and decision-making ability of drones, we also introduced emergency simulations, such as the sudden appearance of friendly units, the arrival of enemy reinforcements, etc. These emergencies require drones to respond quickly and adjust tactics.

[0049] Hierarchical reward function design: This paper designs a hierarchical reward function to provide a clear incentive goal for the training of UAVs in a simulated air combat environment. The following are the specific implementation steps and function definitions:

[0050] Define reward metrics:

[0051] Task completion: define a reward indicator D task , which is based on the number of key mission objectives completed by the drone. For example, for each enemy target destroyed, D task Increase 10 points, T survival Increase by 1 point.

[0052] Energy consumption: Defining a reward metric E energy , which is based on the energy consumption of the drone during the mission. The less energy is consumed per unit of mission completion, the higher the E energy The more points you increase.

[0053] Tactical effect: define a reward indicator E tactics , which is based on the effect of the drone's tactical actions, such as the success rate of avoiding enemy attacks. For each successful avoidance of an attack, E tactics Increase by 5 points.

[0054] Design reward function:

[0055] Task completion reward function: R task =10×D task ;

[0056] Survival time reward function: (Assume T survival in seconds).

[0057] Energy consumption reward function: R energy =100-(E energy ×0.01)(assuming E energy is the total energy consumed during the task).

[0058] Tactical effect reward function: Rtractics = 5xEtagtics / total number of rounds

[0059] Weight distribution:

[0060] Assign weights to each reward function to reflect its importance in the overall training objective. For example, you can assign weights w task =0.5, w survival =0.2, w energy = 0.2, and W tactics =0.1.

[0061] S4.4 Calculation of comprehensive reward:

[0062] Comprehensive reward function: Multiply each reward function by its weight and sum it up to get the comprehensive reward R total : R total =w task ·R task +w survival ·R survival +w energy ·R energy +w tactics ·R tactics This comprehensive reward value will be used to evaluate each action of the drone in the simulated air combat environment.

[0063] Application Reward Feedback:

[0064] Strategy adjustment: During the training process, the drone’s strategy network will adjust the overall reward R totalIf the reward is positive, it means the action was beneficial; if the reward is negative, the policy needs to be adjusted to avoid similar actions.

[0065] Real-time optimization: After the drone performs an action in the simulation environment, it will immediately receive reward feedback, and the parameters of the policy network will be optimized in real time based on this feedback. Through this hierarchical reward function design, the present invention can ensure that every action of the drone in the simulation training is to optimize the above four key indicators, thereby improving the performance of the drone in the actual air combat environment.

[0066] Hierarchical reinforcement learning algorithm training: This invention uses a hierarchical reinforcement learning algorithm, especially the deep deterministic policy gradient (DDPG) algorithm, to systematically train the drone strategy. The following are the specific implementation steps and algorithm details:

[0067] Initialize network parameters:

[0068] Policy Network (ActorNetwork): Initializes the policy network parameters to generate actions to control the drone.

[0069] Value Network (CriticNetwork): Initialize the value network parameters to evaluate the expected returns of actions generated by the policy network.

[0070] Collect initial data:

[0071] Exploration maneuvers: The drone performs a series of exploratory maneuvers in the simulated environment to collect initial experience data.

[0072] Storing experience: Store the initial experience data in the experience replay buffer, including state, action, reward, and new state.

[0073] Training the value network:

[0074] Sampling experience: Randomly sample a batch of experience from the experience replay buffer and calculate the target Q value: Use the sampled experience to calculate the target Q value, the formula is: y = r + γ · CriticNetworkTarget (s′, ActorNetworkTarget (s′)) where r is the reward, γ is the discount factor, s′ is the new state, CriticNetworkTarget and ActorNetworkTarget are the target networks of the value network and policy network respectively.

[0075] Calculate loss: Calculate the loss function of the value network, the formula is: Where N is the batch size, s i and a i are states and actions, θ critic is the value network parameter.

[0076] Update value network: Use gradient descent to update value network parameters.

[0077] Train the policy network:

[0078] Calculate the policy gradient: Calculate the gradient of the policy network based on the value network. The formula is:

[0079]

[0080] Update policy network: Use gradient ascent method to update policy network parameters.

[0081] Target network soft update:

[0082] Soft update target network: Regularly use the soft update mechanism to update the target network parameters, the formula is: θ target =τθ+(1-τ)θ target Here, τ is a small positive number, such as 0.001.

[0083] Repeated iterative optimization:

[0084] Iterative training: Repeat the steps of data collection, value network training and policy network training to continuously optimize the strategy of the UAV. Through this hierarchical reinforcement learning algorithm training, the present invention enables the UAV to continuously learn and optimize in a simulated environment to improve its autonomous decision-making and control capabilities in a complex battlefield environment. This training method not only improves the training efficiency of the UAV, but also ensures its effectiveness and safety in practical applications.

[0085] In the present invention, experience replay technology plays a vital role, which enables drones to learn from historical interactive experiences in simulated air combat. Specifically, every step of the drone's operation when performing a mission, including the state, the action performed, the reward obtained, and the subsequent state, is recorded as a data point and stored in the experience replay buffer. This buffer acts as a large memory bank that saves the actions and results of the drone in different combat scenarios. In order to train the drone's policy network, we randomly extract data points from this buffer. The purpose of this is to avoid time correlation between samples, thereby enhancing the robustness of learning. Using these data points, the value network and policy network of the drone are continuously updated, where the value network evaluates the expected return of the action, and the policy network adjusts the action selection strategy accordingly to improve future returns. This process is repeated, allowing the drone's decision-making strategy to gradually adapt to the simulated environment and optimize. Another benefit of experience replay technology is that it improves the generalization ability of drone strategies. Since the training data comes from a variety of historical combat scenarios, the strategy learned by the drone is not for a specific scenario, but a general strategy that can perform well in a variety of different battlefield environments. As simulation training continues, the data in the experience replay buffer continues to increase, providing the UAV with rich learning material, enabling its strategies to be continuously improved and adapted, ultimately achieving higher combat effectiveness.

[0086] In this paper, we adopt the target network technology, by maintaining a target network with the same structure as the main network but different parameters, and soft-updating the target network parameters at a certain ratio τ, so as to effectively reduce the volatility of the strategy and enhance the stability of training during the hierarchical reinforcement learning process, where τ is usually set at around 0.001, ensuring the smoothness of the reward signal and the gradual improvement of the strategy.

[0087] In the present invention, the UAV's survivability and mission execution efficiency are accurately evaluated by conducting comprehensive tests on the UAV in a simulated environment, including simulating the attack strategies of enemy fighters in variable confrontation scenarios, evaluating the effectiveness of the UAV's strategies, and testing its robustness in randomly generated threats and obstacles.

[0088] In the present invention, the UAV training strategy that has been fully verified in the simulation environment is deployed in real time to the actual UAV system through wireless communication technology, and is updated and optimized online based on the feedback data collected during the actual flight, such as the response mode of the enemy fighter and the changes in the battlefield environment, thereby ensuring that the UAV can quickly adapt to new threats and environmental changes and always maintain optimal performance.

[0089] Through these technical steps, the present invention realizes the real-time status perception and rapid response of the UAV in the air combat environment, optimizes the task execution process, and enhances the survival and adaptability. These innovations not only improve the survivability and task execution efficiency of the UAV in complex air combat environments, but also provide new technical solutions for the application of UAVs in military and civilian fields.

[0090] Compared with the prior art, the present invention has the following beneficial effects:

[0091] 1. The present invention has an accurate and immediate reaction capability improvement: through layered reinforcement learning training, the UAV can identify the launch of an enemy missile within 0.5 seconds and perform evasive actions, such as sharp maneuvers with an acceleration of more than 3G, thereby successfully avoiding the initially locked missile in 95% of simulated confrontations. This rapid reaction mechanism significantly improves the survivability of UAVs in complex battlefield environments.

[0092] 2. The present invention has efficient optimization of task execution strategy: by using the hierarchical reward function, when performing reconnaissance missions, the UAV can choose a path that avoids the area covered by the enemy's air defense system under 60% visibility, while maintaining more than 80% observation coverage of the target area. The task completion time is shortened by 25% compared with the traditional control strategy, and the energy consumption is reduced by 15%.

[0093] 3. The invention has strong adaptability and enhanced battlefield survivability: After high-intensity training in a simulated environment, the UAV can autonomously perform 7G maneuver avoidance when the enemy fighter performs a high-G force turn interception, and successfully maintain the ability to track and strike the target in 80% of cases. In addition, through online update and optimization strategies, the UAV can complete the iterative update of the strategy within 5 minutes in actual battlefield deployment based on the threat data collected in real time to respond to new battlefield threats, thereby ensuring the continuity of the mission and the safety of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0095] Figure 1 This is the architecture diagram of the drone air combat confrontation system of the present invention

[0096] Figure 2 This is a flow chart showing the multi-level states in the present invention.

[0097] Figure 3 This is a flow chart of the drone countermeasure method in the present invention. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0099] Example 1

[0100] This embodiment provides a technical solution, which is a UAV air combat confrontation method based on hierarchical reinforcement learning. The multi-level state representation mechanism of this embodiment provides the UAV with fine environmental perception capabilities in air combat. Through the integrated sensor suite, the UAV can monitor its own flight status in real time, including maintaining a specified altitude range, such as 100 to 5000 meters; speed monitoring covers a wide speed range from low-speed takeoff to high-speed cruising, which may involve a speed range of 0 to 250 meters per second; attitude monitoring covers a full range of attitude changes from horizontal flight to sharp maneuvers, including ±90 degrees of pitch angle, ±60 degrees of roll angle and ±360 degrees of yaw angle; position monitoring uses the GPS system to ensure that the position accuracy of the UAV in three-dimensional space is within 1 meter.

[0101] In terms of target recognition, this embodiment uses a multimodal sensor system to detect and classify enemy targets. The radar system can detect enemy fighters within a range of 20 kilometers, the infrared sensor can identify heat source signals within 3 kilometers, and the optical camera provides high-resolution images for visual recognition. The target tracking algorithm can update the position and speed information of the target in real time, predict its future trajectory, and assess the potential threat level.

[0102] Battlefield situation analysis further enhances the battlefield awareness of the drone. By analyzing high-resolution terrain data from synthetic aperture radar, the drone can identify the distribution of obstacles ahead, such as mountains, buildings, etc., and assess their potential impact on the flight route. At the same time, through the communication link with friendly units, the drone can receive and integrate the location information of friendly units to build a networked battlefield situation map containing at least 10 friendly units. Finally, through advanced data fusion algorithms, this embodiment integrates the data of flight status monitoring, target identification and battlefield situation analysis into a unified situation awareness model. This model not only enables the drone to understand its own flight status in real time, but also understands the entire battlefield environment from a macro level to achieve comprehensive situation awareness. The decision support system uses this information to provide battlefield information and decision-making suggestions to the operator, enhancing the autonomous decision-making ability of the drone. For example, when an enemy missile is detected, the system may recommend performing an 8G sharp turn maneuver to evade the threat, while calculating a new route to maintain the continuity of mission execution. This multi-level state representation mechanism is one of the important innovations of this embodiment in the field of drone control technology.

[0103] The hierarchical action space significantly improves the adaptability and reaction speed of the drone in complex battlefield environments. In terms of speed control, the drone can quickly accelerate from a stationary state to a maximum speed of 250 meters per second, and can perform rapid maneuvers at an acceleration of 100 meters per second2. This enables the drone to respond quickly to battlefield changes, whether it is chasing targets or emergency evacuation. In terms of attitude adjustment, the drone is designed to be able to maneuver with extremely high flexibility. Specifically, it can complete a 30-degree pitch action in less than 2 seconds, or a 60-degree roll in 3 seconds. This design enables the drone to perform complex evasive tactics in air combat, such as getting rid of tracking through a sharp roll action when encountering an enemy missile lock. Flight path selection is another key component of the drone's action space. The drone can automatically select the optimal path to evade the enemy's air defense system based on the battlefield map updated in real time. For example, if the enemy's air defense system covers a range of 50 kilometers, the drone will automatically plan a safe path to bypass the area. Tactical evasive action is the key to the drone's survivability. To this end, this embodiment designs a series of avoidance actions that can be completed within 5 seconds, including rapid ascent, descent or sharp turns. The execution of these actions is not only rapid but also precise, ensuring that the drone can avoid enemy fire at critical moments. In addition, this embodiment also includes a feedback system for real-time monitoring of the execution of the drone's actions. Once an action is performed, such as a 5G turn, the system will immediately feedback the execution results, including changes in speed, posture and position. The operator can further adjust the action parameters based on this feedback information to adapt to the ever-changing battlefield environment. Through these designs, this embodiment not only improves the autonomy of the drone, but also provides strong technical support for the application of drones in military and civilian fields. This hierarchical action space design makes the drone more flexible, efficient and safe when performing tasks.

[0104] The simulated air combat environment can provide drones with a near-real combat training experience. This environment uses advanced simulation technology to generate a three-dimensional terrain model containing various terrain features. For example, the height of the mountains can reach more than 3,000 meters, while the vast plains provide open space for fighter jets to chase in the air. The simulation of enemy units is very detailed. For example, the enemy fighter model can simulate the performance of real aircraft, including maneuvers at a speed of 1,500 kilometers per hour at an altitude of 5,000 meters, and can perform overload maneuvers from +9G to -3G.

[0105] To ensure that the drone can perform missions under changing climatic conditions, our simulation environment also integrates a dynamic weather system. This system can simulate extreme weather conditions such as strong winds with speeds up to 25 meters per second, dense fog with visibility as low as 100 meters, and thunderstorms. These weather conditions not only test the drone's flight control system, but also the performance of its sensors and communication systems in harsh environments. In addition, our simulated battlefield environment includes electromagnetic interference, which will appear randomly with a certain probability and intensity to simulate the real battlefield electromagnetic environment. This simulation can train the drone's ability to maintain stable flight and accurately perform tasks under electromagnetic interference. The difficulty and complexity of the simulation environment can be dynamically adjusted according to training needs. For example, the number of enemy units can be gradually increased from a single fighter to the size of a squadron, and the attack frequency of enemy fighters can be increased from once per minute to once every 10 seconds. We also simulated enemy units attacking with different types of air-to-air missiles with different speeds, tail-chasing capabilities and maneuverability. In order to fully test the drone's tactical decision-making ability, we designed a variety of mission types, including covert reconnaissance within the coverage of the enemy's air defense system, or precision strike missions under the interception of enemy fighters. Each mission type has its own specific environment setting and enemy unit configuration, ensuring that drones can be trained in a variety of battlefield conditions.

[0106] In this example, we designed a hierarchical reward function that incentivizes the performance of the drone in a simulated air combat environment through four key indicators: mission completion, survival time, energy consumption, and tactical effect. The mission completion reward is quantified by adding 10 points to each enemy target destroyed, and the survival time reward is calculated by adding one point per second based on the survival time of the drone on the battlefield. The energy consumption reward is based on the energy usage of the drone during the mission, and the reward increases accordingly for each less energy consumed. The tactical effect reward is calculated based on the number of times the drone successfully avoids enemy attacks, and each successful avoidance can earn 5 points. These reward indicators are assigned different weights to reflect their importance in the overall training goal. For example, mission completion may be assigned a weight of 0.5, while tactical effect may be assigned a weight of 0.1. Then, we calculate a comprehensive reward value, which is the sum of all these weighted rewards, to evaluate each action of the drone. This comprehensive reward value not only provides immediate feedback on the drone's behavior, but is also used to optimize the drone's policy network in real time, ensuring that every action it takes in the simulated environment is to improve the performance of these four key indicators. With this approach, we are able to ensure that every action the drone performs during simulation training is designed to optimize its key performance indicators in an actual air combat environment.

[0107] In this embodiment, we use the deep deterministic policy gradient (DDPG) algorithm, which is a hierarchical reinforcement learning algorithm, to systematically train the control strategy of the drone. The training process starts with randomly initializing the parameters of the policy network and the value network, and then the drone performs exploratory actions in the simulated environment to collect initial data. This data, including state, action, reward, and new state, is stored in the experience replay buffer for later use. During the training process, we randomly extract batches of data from the buffer to calculate the target Q value of the value network, and use this Q value to update the value network parameters. Next, we use the updated value network to calculate the gradient of the policy network and use the gradient ascent method to update the policy network to optimize the control strategy of the drone. In order to stabilize the training process, we introduce the target network and regularly soft-update the parameters of these networks at a small ratio. This process is repeated in each training cycle, and through continuous iterations, the drone's strategy is gradually optimized, thereby achieving efficient learning and performance improvement in the simulated environment. The application of this hierarchical reinforcement learning algorithm enables drones to make more accurate and effective decisions in complex and changing battlefield environments.

[0108] In this embodiment, the application of experience replay technology enables the drone to learn and optimize its behavior strategy from the historical interactions of simulated air combat. Specifically, each step of the drone in the simulated environment, including the current state st, the action performed at, the immediate reward rt obtained, and the next state st+1, is encapsulated into an experience tuple and stored in the experience replay buffer. This buffer accumulates thousands or even tens of thousands of such quadruple, providing rich historical data for the drone. During the training process, we randomly extract a batch of 32 experience tuples from the buffer, each of which contains a multidimensional feature vector of the state, a continuous action space, an immediate reward value, and the subsequent state. These randomly sampled data batches help break the temporal association between experiences and improve the robustness of learning. In this way, the value network and policy network of the drone use these data batches to update themselves in each iteration, where the value network predicts the long-term return of each action, and the policy network optimizes its action selection based on these predictions. As the simulation training continues, the drone's strategy gradually improves, improving its adaptability and decision-making quality in a complex battlefield environment.

[0109] In this embodiment, we use the target network technology to maintain a target network with the same structure as the main network but different parameters, and soft-update the target network parameters at a certain ratio τ, so as to effectively reduce the volatility of the strategy and enhance the stability of training during the hierarchical reinforcement learning process. τ is usually set at around 0.001, ensuring the smoothness of the reward signal and the gradual improvement of the strategy.

[0110] In this embodiment, the UAV is comprehensively tested in a simulated environment, including simulating the attack strategies of enemy fighters in variable confrontation scenarios, evaluating the effectiveness of the UAV's strategies, and testing its robustness in randomly generated threats and obstacles, so as to accurately evaluate the UAV's survivability and mission execution efficiency.

[0111] In this embodiment, the UAV training strategy that has been fully verified in the simulation environment is deployed in real time to the actual UAV system through wireless communication technology, and is updated and optimized online based on feedback data collected during actual flight, such as the response patterns of enemy fighters and changes in the battlefield environment, thereby ensuring that the UAV can quickly adapt to new threats and environmental changes and always maintain optimal performance.

[0112] Through these technical steps, this embodiment realizes the real-time status perception and rapid response of the UAV in the air combat environment, optimizes the task execution process, and strengthens the survivability and adaptability. These innovations not only improve the survivability and task execution efficiency of the UAV in complex air combat environments, but also provide new technical solutions for the application of UAVs in military and civilian fields.

[0113] Example 2

[0114] Traditional methods of UAV air combat confrontation usually rely on preset rules and simple feedback control, which have obvious limitations when dealing with dynamically changing battlefield environments. Especially in scenarios that require rapid decision-making and response, traditional methods often cannot effectively handle complexity, resulting in insufficient autonomy and adaptability of UAVs.

[0115] In this embodiment, we compared the method of this embodiment with a typical traditional method - a rule-based control system. In a simulated air combat environment, the method of this embodiment combines the deep deterministic policy gradient algorithm with experience replay and target network technology, and the trained policy network can adapt to battlefield changes more quickly. In the simulated confrontation, the method of this embodiment achieved a success rate of 95%, while the success rate of the rule-based control system was only 70%. In addition, the method of this embodiment reduces energy consumption by 15% compared with the rule-based system, significantly improving the mission execution efficiency of the UAV.

[0116] Detailed description and parameters:

[0117] 1. Situation Acquisition: The method of this embodiment uses an integrated sensor suite to monitor the flight status of the drone in real time, including altitude, speed, attitude, and position information. Compared with traditional methods, this embodiment can provide more accurate and real-time information, thereby improving the quality of situation awareness. Specific parameters include: altitude monitoring range 0-5000 meters, speed monitoring range 0-250 meters / second, attitude monitoring accuracy ±0.5 degrees.

[0118] 2. Hierarchical action space design: This embodiment designs a sophisticated hierarchical action space, including speed control, attitude adjustment, flight path selection, and tactical avoidance action. This design enables the drone to perform more complex maneuvers, such as sharp ascents, descents, rolls, and sharp turns, while traditional methods can usually only perform simple straight-line flights or preset flight paths. Specific parameters include: speed control accuracy of 0.1 m / s, attitude adjustment response time of less than 2 seconds, and tactical avoidance action execution time of 1-5 seconds.

[0119] 3. Hierarchical reward function and strategy training: This embodiment designs a hierarchical reward function that comprehensively considers task completion, survival time, energy consumption and tactical effect. Compared with traditional methods, the method of this embodiment can more effectively motivate the learning and decision-making of drones in simulated air combat environments, thereby improving the training effect and performance in practical applications. Specific parameters include: task completion reward 10 points / target, survival time reward 1 point / second, energy consumption reward -0.01 points / unit of energy, tactical effect reward 5 points / successful avoidance.

[0120] 4. Model deployment and real-time optimization: The method of this embodiment tests the performance of the UAV in a simulated environment and implements online update optimization through wireless communication. This enables the UAV to adjust its strategy in real time based on feedback from the actual air combat environment, which is usually impossible with traditional methods. Specific parameters include: a simulation environment test cycle of 1 week and an online update frequency of 5 minutes.

[0121] Conclusion: Through the above comparison, it can be seen that the method of this embodiment is superior to the traditional rule-based control system in terms of autonomous decision-making, adaptability, task execution efficiency and survivability, which proves that the technical solution of this embodiment is preferred.

[0122] Example 3

[0123] Background: In the field of drone air combat, traditional control methods often rely on pre-set rules or simple feedback control. These methods often fail to provide sufficient autonomy and adaptability in the face of complex and changing battlefield environments. In particular, the limitations of traditional methods are more obvious in scenarios that require rapid response and decision-making.

[0124] Example 3: In this example, we compared the method of this example with an advanced traditional method, model-based predictive control (MPC). In a simulated air combat environment, the method of this example uses a deep deterministic policy gradient algorithm combined with experience replay and target network technology to train a policy network that can adapt to battlefield changes more quickly and achieve a 98% success rate in simulated confrontations, while the MPC-based method has a success rate of only 75%. In addition, the method of this example reduces energy consumption by 20% compared to the MPC-based method, significantly improving the mission execution efficiency of drones.

[0125] Detailed description and parameters:

[0126] 1. Situation Acquisition: The method of this embodiment uses an integrated sensor suite to monitor the flight status of the drone in real time, including altitude, speed, attitude, and position information. Compared with traditional methods, this embodiment can provide more accurate and real-time information, thereby improving the quality of situation awareness. Specific parameters include: altitude monitoring range 0-6000 meters, speed monitoring range 0-300 meters / second, attitude monitoring accuracy ±0.2 degrees.

[0127] 2. Hierarchical action space design: This embodiment designs a sophisticated hierarchical action space, including speed control, attitude adjustment, flight path selection, and tactical avoidance action. This design enables the drone to perform more complex maneuvers, such as sharp ascents, descents, rolls, and sharp turns, while traditional methods can usually only perform simple straight-line flights or preset flight paths. Specific parameters include: speed control accuracy of 0.05 m / s, attitude adjustment response time of less than 1 second, and tactical avoidance action execution time of 0.5-3 seconds.

[0128] 3. Hierarchical reward function and strategy training: This embodiment designs a hierarchical reward function that comprehensively considers task completion, survival time, energy consumption and tactical effect. Compared with traditional methods, the method of this embodiment can more effectively motivate the learning and decision-making of drones in simulated air combat environments, thereby improving the training effect and performance in practical applications. Specific parameters include: task completion reward 20 points / target, survival time reward 2 points / second, energy consumption reward -0.02 points / unit of energy, tactical effect reward 10 points / successful avoidance.

[0129] 4. Model deployment and real-time optimization: The method of this embodiment tests the performance of the UAV in a simulated environment and implements online update optimization through wireless communication. This enables the UAV to adjust its strategy in real time based on feedback from the actual air combat environment, which is usually not possible with traditional methods. Specific parameters include: a simulation environment test cycle of 1 week and an online update frequency of 3 minutes.

[0130] Conclusion: Through the above comparison, it can be seen that the method of the present invention is superior to the model-based predictive control method in terms of autonomous decision-making, adaptability, task execution efficiency and survivability, which proves that the technical solution of the present invention is preferred.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A UAV air combat confrontation method based on hierarchical reinforcement learning, characterized in that: The following steps are involved: S1: Situation acquisition, real-time acquisition of UAV flight status and target information through a variety of high-precision sensors and detection and recognition systems; S2: Hierarchical action space design, fine-grained definition of the action space including speed, posture, path and tactical action; S3: Hierarchical reward function and strategy training, using deep deterministic policy gradient algorithm combined with experience replay and target network technology to train the strategy network, and designing a hierarchical reward function that comprehensively considers task completion, survival time, energy consumption and tactical effect; S4: Model deployment and real-time optimization, testing and evaluating drone performance in a simulated environment, and achieving online update optimization through wireless communication.

2. The UAV air combat confrontation method based on hierarchical reinforcement learning according to claim 1 is characterized in that: In step S1, the situation acquisition and definition of the multi-level state representation of the UAV include the following steps: S11. Flight status monitoring: real-time acquisition of the UAV's flight altitude, speed, attitude and position information through high-precision sensors, altimeters, pitot tubes, inertial measurement units (IMUs) and global positioning systems (GPSs), ensuring the accuracy and real-time nature of the information. S12, target recognition, real-time identification of the location and characteristics of enemy fighters, missiles and obstacles, use of radar, infrared sensors and visual systems to identify and track targets on the battlefield, and analysis of their characteristics for accurate classification and tracking; S13, battlefield situation analysis, integrate flight status and target information to form battlefield situation awareness, identify obstacle distribution and terrain features in the battlefield by analyzing terrain data, environmental scanning data and geographic information system (GIS) data, and integrate friendly unit location information.

3. The UAV air combat confrontation method based on hierarchical reinforcement learning according to claim 1 is characterized in that: In step S2, the hierarchical action space design, designing the hierarchical action space of the drone, includes the following steps: S21. Action space definition, defining the action space of the drone, including speed change, attitude adjustment, flight path selection and tactical avoidance action. The speed control range is 0m / s to the maximum speed of 250m / s, and the acceleration change range is 0 to 100m / s2. The attitude adjustment includes pitch angle -30° to +30°, roll angle -60° to +60° and yaw angle. The flight path is selected from the preset path library, and the enemy air defense system and terrain obstacles are taken into account. Tactical avoidance actions include sharp ascent, descent, roll and sharp turn, and the execution time is within 1 to 5 seconds. S22. Action parameter optimization. According to different tactical requirements, the action parameters are optimized to achieve fine control, including dynamic adjustment of the maximum overload limit and the minimum turning radius. The action space is designed to be continuous, allowing for subtle action adjustments, and a variety of tactical action combinations are designed.

4. The UAV air combat confrontation method based on hierarchical reinforcement learning according to claim 1 is characterized in that: In step S3, the hierarchical reward function and strategy training, designing the hierarchical reward function and training the drone strategy through the hierarchical reinforcement learning algorithm, includes the following steps: S31. Reward function design. Design a hierarchical reward function, taking into account task completion, survival time, energy consumption and tactical effect. Task completion reward is based on the number of key task targets completed, survival time reward is calculated based on survival time, energy consumption reward is related to energy consumption during the task, and tactical effect reward is based on the number of successful avoidance of enemy attacks. Weights are assigned to each reward function. Task completion reward function: R task =∑(10·Number of completed tasks) Survival time reward function: R survival = Survival time Energy consumption reward function: Tactical effect reward function: R tactics =∑(5·Number of successful evasions of enemy attacks) Comprehensive reward function R total =w task ·R task +w survival ·R survival +w energy ·R energy +w tactics .R tactics Among them, the weight distribution of each reward function is w task , w survival , w energy , w tactics S32, policy network training, using deep deterministic policy gradient algorithm to train policy network, improving learning efficiency through experience replay technology, including initializing policy network and value network parameters, collecting initial data, performing exploration actions and storing experience, sampling experience from experience replay buffer to calculate target Q value y = r + γ · CriticNetworkTarget (s′, ActorNetworkTarget (s′)), calculate the value network loss function At the same time, update the value network parameters and calculate the policy network gradient Update policy network parameters, Periodically use the soft update mechanism to update the target network parameters: θ target =τθ+(1-τ)θ target , repeat the iterative optimization until the policy network converges; Where r is the reward, γ is the discount factor, s′ is the new state, CriticNetworkTarget and ActorNetworkTarget are the target networks of the value network and the policy network respectively, N is the batch size, and s i and a i are states and actions, θ critic is the value network parameter, and τ is a small positive number.

5. The UAV air combat confrontation method based on hierarchical reinforcement learning according to claim 1 is characterized in that: In step S4, model deployment and real-time optimization, applying the trained strategy to the actual drone and performing real-time optimization, includes the following steps: S41, simulated environment test, testing the performance of the UAV in a simulated environment to evaluate its survivability and mission execution efficiency. The simulated environment includes a three-dimensional terrain model, enemy units with real flight performance parameters, a dynamic weather system, and electromagnetic interference. The difficulty and complexity can be adjusted dynamically, and it has multiple mission types and confrontation scenario simulations and emergency simulations; S42, online update and optimization, which is achieved through wireless communication, allowing the UAV to adjust its strategy in real time based on feedback from the actual air combat environment; S43. Actual environment application: deploy the strategy to actual drones for testing and application in actual air combat environments.

Images