An unmanned aerial vehicle air combat confrontation method and system based on hierarchical reinforcement learning

Abstract

This invention belongs to the field of unmanned aerial vehicle (UAV) technology and discloses a UAV air combat confrontation method and system based on hierarchical reinforcement learning. The method includes: simulating the environment using a 3D terrain model and a 3D dynamic wind field model; correcting noise and enhancing features of multi-source sensor data using a dynamic battlefield environment and sensor attenuation model; quantizing multi-source perception datasets and multi-source state feature data into feature vectors; weighted fusion using a nonlinear transformation network; constructing multi-dimensional training target parameters by integrating the situational representation vector and mission objectives, assigning weights, and calculating a weighted total reward; an upper-layer reinforcement learning agent evaluating the optimal path command and issuing it to the lower-layer flight control terminal; generating actions for the propagation of sudden events; storing multi-source temporal interaction data in an experience replay buffer; updating the value network and policy network; and iteratively optimizing the target network to achieve autonomous decision-making and control strategies for the UAV. This invention improves the survivability of UAVs in complex battlefield environments through hierarchical reinforcement learning training.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a UAV air combat confrontation method and system based on hierarchical reinforcement learning. Background Technology

[0002] The application of unmanned aerial vehicle (UAV) technology has expanded to multiple fields, including military, civilian, and scientific research. The autonomy and adaptability of UAVs in complex environments have become a core research focus. Most existing UAV control methods rely on pre-set rules or simple feedback control mechanisms. These methods have significant limitations when facing dynamic and ever-changing adversarial environments. On the one hand, pre-set rules cannot cover all unexpected battlefield scenarios, such as the enemy suddenly changing its attack tactics or extreme weather conditions. On the other hand, simple feedback control has a slow response speed, which is difficult to meet the needs of second-level decision-making in air combat. In emergency situations such as dealing with enemy missile attacks or quickly avoiding obstacles, the delayed decision-making often leads to mission failure or damage to the drone.

[0003] In recent years, deep reinforcement learning, as an emerging machine learning method, has shown potential in complex tasks by learning optimal policies through interaction with the environment. Among them, the deterministic policy gradient (DDPG) algorithm performs well in continuous action space decision-making problems. However, applying the DDPG algorithm to UAV adversarial environment control still faces three core challenges: First, traditional state representations focus solely on the drone's own flight parameters, neglecting the overall battlefield situation (such as the positions of friendly units and the enemy's tactical intentions), resulting in limited field of vision for the drone. Second, the motion space design is crude, failing to balance precise flight control (e.g., 0.1-degree attitude adjustments) with tactical maneuvers (e.g., combinations of rolls and sharp turns). Third, the reward function is singular, focusing only on mission completion without considering key indicators such as survival time and energy consumption. This leads to extreme situations in actual combat where the drone prioritizes the mission but is destroyed, or excessively conserves energy but exceeds the mission timeout.

[0004] Therefore, how to provide a method and system for UAV air combat based on hierarchical reinforcement learning is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method and system for UAV air combat based on hierarchical reinforcement learning to solve the problems mentioned above in the prior art.

[0006] According to a first aspect of the present invention, a method for unmanned aerial vehicle (UAV) air combat based on hierarchical reinforcement learning is provided.

[0007] In one embodiment, the UAV air combat countermeasure method based on hierarchical reinforcement learning includes: A three-dimensional terrain model was constructed using geographic information system data to simulate electromagnetic interference conditions; a three-dimensional dynamic wind field model was constructed based on computational fluid dynamics principles to simulate meteorological conditions; and the dynamic battlefield environment was obtained based on the simulation results. Using a pre-defined sensor attenuation model and a dynamic battlefield environment, noise correction and feature enhancement are sequentially applied to multi-source sensor data to obtain a multi-source sensing dataset. The multi-source sensing dataset and multi-source state feature data are quantized into feature vectors, and the feature vectors are weighted and fused using a nonlinear transformation network to obtain a comprehensive situational representation vector. Based on the comprehensive situational representation vector and the task objective, multi-dimensional training objective parameters are constructed and weights are assigned. The weighted total reward is calculated using the weighted weights to obtain a hierarchical reward function system. By utilizing an upper-level reinforcement learning agent, the comprehensive situational representation vector and hierarchical reward function system are evaluated to obtain the optimal path instruction; this instruction is then sent to the lower-level flight control terminal for execution, and unexpected events during the execution process are propagated forward to generate actions, resulting in online optimized decision actions. Based on decision-making actions, comprehensive situation vectors, and hierarchical reward functions, multi-source temporal interaction data is constructed and stored in an experience replay buffer. The value network and policy network are updated according to the experience replay data, the target network is maintained and iteratively optimized to obtain the UAV autonomous decision-making and control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

[0008] In one embodiment, a three-dimensional terrain model is constructed using geographic information system data to simulate electromagnetic interference conditions; a three-dimensional dynamic wind field model is constructed based on computational fluid dynamics principles to simulate meteorological conditions; based on the simulation results, the dynamic battlefield environment is obtained, including: Based on the topographic features of mountains, plains, and oceans in the geographic information system data, a three-dimensional terrain model is constructed; Based on the principles of computational fluid dynamics, the spatiotemporally correlated wind speed vector field is obtained, and the wind direction data is calculated to obtain a three-dimensional dynamic wind field model. By using a three-dimensional terrain model and a three-dimensional dynamic wind field model, meteorological conditions and electromagnetic interference on the battlefield are simulated in a dynamic weather terminal to obtain the dynamic battlefield environment.

[0009] In one embodiment, using a preset sensor attenuation model and a dynamic battlefield environment, noise correction and feature enhancement are sequentially applied to multi-source sensor data to obtain a multi-source sensing dataset including: Based on the dynamic battlefield environment, and using the UAV six-degree-of-freedom dynamic model to obtain the environmental state parameters under the current battlefield scenario, wherein the environmental state parameters include meteorological condition parameters and electromagnetic interference parameters; By using a sensor attenuation model and combining environmental state parameters, the attenuation coefficients and measurement noise covariance of visual sensors and lidar under the current battlefield situation are determined. The attenuation coefficient and the measurement noise covariance are converted into correction parameters, and the correction parameters are input into the visual sensor imaging data and the lidar point cloud data to identify and separate the noise components in turn. Noise correction processing is performed on the multi-source sensing data from which noise components are separated to obtain benchmark sensing data, and feature extraction is performed on the benchmark sensing data to obtain multi-dimensional key feature information. Feature enhancement processing is performed on multi-dimensional key feature information to enhance the recognizability and effectiveness of features, resulting in a multi-source perception dataset.

[0010] In one embodiment, using a sensor attenuation model and incorporating environmental state parameters to determine the attenuation coefficients and measurement noise covariance of visual sensors and lidar under the current battlefield situation includes: Using the sensor attenuation model, we matched the atmospheric transmittance model of the visual sensor and the scattering model of the lidar, respectively. The meteorological condition parameters from the environmental state parameters are input into the attenuation model of the visual sensor to calculate the attenuation coefficient of the visual sensor. The meteorological and electromagnetic interference parameters in the environmental state parameters are input into the attenuation model of the lidar to calculate the attenuation coefficient of the lidar. Based on the calculated attenuation coefficients of the visual sensor and the lidar, the measurement noise covariance is quantified sequentially to obtain the attenuation coefficient and the measurement noise covariance.

[0011] In one embodiment, the multi-source sensing dataset and multi-source state feature data are quantized into feature vectors, and the feature vectors are weighted and fused using a nonlinear transformation network to obtain a comprehensive situational representation vector, including: The multi-source sensing dataset and multi-source state feature data are normalized to obtain a standardized feature data set. The feature data set is dimensionality normalized and vectorized to map discrete and continuous features into feature vector form, thus obtaining a feature vector set. By using a nonlinear transformation network, the feature vector set is nonlinearly mapped and weighted to construct a comprehensive correlation relationship of multi-source features, thereby obtaining a comprehensive situational representation vector.

[0012] In one embodiment, based on the comprehensive situational representation vector and the task objective, multi-dimensional training objective parameters are constructed and weights are assigned. The weighted total reward is then calculated using the weighted weighting results to obtain a hierarchical reward function system, including: Based on the comprehensive situational representation vector and mission objectives, four core dimensions are extracted: mission completion, survivability, energy efficiency, and tactical effectiveness, and multi-dimensional training objective parameters are constructed. Based on the principle of selecting the values ​​of each weight coefficient, weight coefficients are assigned to the multi-dimensional training target parameters to obtain the weight allocation results; Based on the weight allocation results and real-time quantization values, the training target parameters are weighted and the weighted results are accumulated and summed to calculate the weighted total reward. By integrating multi-dimensional training target parameters, weight allocation results, the calculation logic of weighted total reward, and quantification standards, a hierarchical reward function system is obtained.

[0013] In one embodiment, an upper-level reinforcement learning agent is used to evaluate the comprehensive situational representation vector and the hierarchical reward function system to obtain the optimal path instruction; this instruction is then sent to the lower-level flight control terminal for execution. During execution, unexpected events are propagated forward and actions are generated, resulting in online optimized decision actions, including: The integrated situational representation vector and hierarchical reward function system are input into the upper-layer reinforcement learning agent for collaborative evaluation to obtain the evaluation results. The upper-level reinforcement learning agent is used to evaluate the evaluation results and select strategies to obtain the optimal path instructions. The optimal path command is sent to the lower-level flight control terminal, which then executes the optimal path command to obtain the path command execution result. During the execution of the optimal path instruction, monitoring for sudden events is performed. If a sudden event occurs, the upper-layer reinforcement learning agent is used to perform forward propagation calculations on the sudden event to generate action instructions that adapt to the new situation and obtain the action instructions corresponding to the sudden event. The path instruction execution feedback is integrated with the action instructions generated by the sudden event, and the upper-level reinforcement learning agent is used to complete the online optimization of the decision action, resulting in the online optimized decision action.

[0014] In one embodiment, the upper-layer reinforcement learning agent performs value evaluation and policy selection on the evaluation results to obtain the optimal path instruction, including: Extract the attitude target parameters from the optimal path command based on the evaluation results, and clarify the target range of attitude adjustment; The attitude control terminal of the upper-layer reinforcement learning agent extracts the attitude target parameters to obtain the current flight attitude data; and calculates the attitude deviation between the current flight attitude data and the target attitude parameters. Based on the attitude deviation results, an attitude adjustment signal is generated, and the attitude is corrected within the flight control range. The adjusted flight attitude is monitored in real time to confirm that the attitude parameters have reached the target range, and the optimal path command after adjustment is obtained.

[0015] In one embodiment, based on decision actions, a comprehensive situation vector, and a hierarchical reward function, multi-source temporal interaction data is constructed and stored in an experience replay buffer. The value network and policy network are updated according to the experience replay data, the target network is maintained, and iterative optimization is performed to obtain an autonomous decision-making and control strategy for the UAV. This achieves hierarchical reinforcement learning-based autonomous UAV air combat confrontation decision-making and control optimization, including: Based on the decision-making actions performed by UAVs, the comprehensive situational representation vector, and the hierarchical reward function, multi-source time-series interactive data is collected and constructed. The multi-source time-series interactive dataset is stored in the experience playback buffer and stored according to preset rules. The target network of the updated value network and policy network is maintained, and the parameters of the target network are incrementally updated using a soft update mechanism to obtain the maintained target network. Repeat the network update and maintenance process, and iteratively optimize the value network, policy network, and target network to obtain the iteratively optimized network system; By integrating the learning results from the experience playback data with the network optimization results using the iteratively optimized network architecture, an autonomous decision-making and control strategy for UAVs is obtained, thereby achieving hierarchical reinforcement learning-based autonomous air combat decision-making and control optimization for UAVs.

[0016] According to a second aspect of the present invention, a drone air combat system based on hierarchical reinforcement learning is provided.

[0017] In one embodiment, the UAV air combat system based on hierarchical reinforcement learning includes: The battlefield environment construction module is used to build a three-dimensional terrain model using geographic information system data to simulate electromagnetic interference conditions; it also builds a three-dimensional dynamic wind field model based on computational fluid dynamics principles to simulate meteorological conditions; and based on the simulation results, it obtains the dynamic battlefield environment. The perception dataset generation module is used to perform noise correction and feature enhancement on multi-source sensing data in sequence using a preset sensor attenuation model and dynamic battlefield environment to obtain a multi-source perception dataset. The situation fusion representation module is used to quantize multi-source sensing datasets and multi-source state feature data into feature vectors, and to use a nonlinear transformation network to perform weighted fusion of the feature vectors to obtain a comprehensive situation representation vector. The hierarchical reward construction module is used to construct multi-dimensional training target parameters and assign weights based on the comprehensive situational representation vector and task objective, and calculate the weighted total reward based on the weighted weighting results to obtain the hierarchical reward function system. The decision command generation module is used to evaluate the comprehensive situational representation vector and hierarchical reward function system using the upper-level reinforcement learning agent to obtain the optimal path command; and then send it to the lower-level flight control terminal for execution. During the execution process, it performs forward propagation of unexpected events and generates actions to obtain the online optimized decision actions. The strategy learning optimization module is used to construct multi-source temporal interaction data based on decision actions, comprehensive situation vectors and hierarchical reward functions, and store it in the experience replay buffer. It updates the value network and policy network based on the experience replay data, maintains the target network and iteratively optimizes it to obtain the UAV autonomous decision control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

[0018] According to a third aspect of the present invention, a computer device is provided.

[0019] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0021] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0022] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: 1) This invention enables UAVs to identify enemy missile launches and perform evasive maneuvers within 0.5 seconds through hierarchical reinforcement learning training. For example, it enables UAVs to perform rapid maneuvers with acceleration of more than 3G and successfully break free from the initial missile lock in 95% of simulated combat scenarios. This rapid response mechanism improves the survivability of UAVs in complex battlefield environments.

[0023] 2) This invention adopts a hierarchical reward function. When the UAV performs a reconnaissance mission, it can plan a path under 60% visibility conditions to avoid the enemy's air defense coverage area, while maintaining an observation coverage rate of more than 80% of the target area. Compared with the traditional control strategy, the mission completion time is shortened by 25% and energy consumption is reduced by 15%, thereby improving the execution efficiency and mission stability of the reconnaissance mission.

[0024] 3) After intensive training in a simulated environment, the UAV can autonomously perform 7G maneuvers to evade enemy fighters during high-G turns, and maintain its ability to track and engage targets continuously in 80% of cases. Furthermore, relying on an online strategy update and optimization mechanism, the UAV can iterate and upgrade its strategy within 5 minutes based on real-time threat data during actual combat deployment, thereby effectively responding to new battlefield threats and ensuring mission continuity and its own safety.

[0025] 4) This invention addresses the high dynamic characteristics of air combat environments (such as enemy fighters maneuvering at 1500km / h and performing ±9G overload maneuvers) and complex interference factors (such as electromagnetic interference and sudden thunderstorms). This invention enables UAVs to have flexible and efficient autonomous learning and decision-making capabilities, thereby significantly improving their battlefield autonomy and environmental adaptability.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0028] Figure 1 This is a flowchart illustrating a drone air combat confrontation method based on hierarchical reinforcement learning, according to an exemplary embodiment. Figure 2 This is one of the partial structural flowcharts of a hierarchical reinforcement learning-based UAV air combat confrontation method according to an exemplary embodiment; Figure 3 This is a partial structural flowchart of a hierarchical reinforcement learning-based UAV air combat confrontation method, according to an exemplary embodiment. Figure 4 This is a block diagram illustrating the principle of a hierarchical reinforcement learning-based unmanned aerial vehicle (UAV) air combat system according to an exemplary embodiment. Figure 5 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment; Figure 6 This is a partial block diagram illustrating the principle of a hierarchical reinforcement learning-based unmanned aerial vehicle (UAV) air combat system according to an exemplary embodiment. Detailed Implementation

[0029] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some portions and features of certain embodiments may be included in or replace portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims and all available equivalents thereof. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0030] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0031] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0032] Figures 1-3 An embodiment of a UAV air combat countermeasure method based on hierarchical reinforcement learning according to the present invention is shown.

[0033] In this optional embodiment, the method includes: Step S101: Construct a three-dimensional terrain model using geographic information system data to simulate electromagnetic interference conditions; construct a three-dimensional dynamic wind field model based on computational fluid dynamics principles to simulate meteorological conditions; obtain the dynamic battlefield environment based on the simulation results. Step S102: Using a preset sensor attenuation model and dynamic battlefield environment, noise correction and feature enhancement are performed on the multi-source sensing data in sequence to obtain a multi-source sensing dataset. Step S103: Quantize the multi-source sensing dataset and multi-source state feature data into feature vectors, and use a nonlinear transformation network to perform weighted fusion of the feature vectors to obtain a comprehensive situational representation vector; Step S104: Based on the comprehensive situational representation vector and the task objective, construct multi-dimensional training target parameters and assign weights, and calculate the weighted total reward based on the weighted results to obtain a hierarchical reward function system; Step S105: Using the upper-level reinforcement learning agent, evaluate the comprehensive situational representation vector and the hierarchical reward function system to obtain the optimal path instruction; and send it to the lower-level flight control terminal for execution, propagate forward for sudden events during the execution process and generate actions to obtain the online optimized decision action; Step S106: Based on decision-making actions, comprehensive situation vectors, and hierarchical reward functions, construct multi-source temporal interaction data and store it in the experience replay buffer. Update the value network and policy network according to the experience replay data, maintain the target network and iteratively optimize it to obtain the UAV autonomous decision-making and control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

[0034] In this optional embodiment, a three-dimensional terrain model is constructed using geographic information system (GIS) data to simulate electromagnetic interference conditions; a three-dimensional dynamic wind field model is constructed based on computational fluid dynamics (CFD) principles to simulate meteorological conditions; based on the simulation results, the dynamic battlefield environment is obtained, including: constructing a three-dimensional terrain model based on the terrain features of mountains, plains, and oceans in the GIS data; obtaining a spatiotemporally correlated wind speed vector field based on CFD principles and calculating wind direction data to obtain a three-dimensional dynamic wind field model; and using the three-dimensional terrain model and the three-dimensional dynamic wind field model, simulating meteorological conditions in a dynamic weather terminal and electromagnetic interference on the battlefield to obtain the dynamic battlefield environment.

[0035] Specifically, this invention provides UAVs with a near-realistic training platform through a highly realistic and configurable simulated air combat environment. In this environment, a three-dimensional terrain model is generated using Geographic Information System (GIS) data, meticulously reproducing terrain features such as mountains, plains, and oceans. Furthermore, enemy units are designed as fighter jet models with realistic flight performance parameters, capable of maneuvering at a maximum speed of 1500 km / h and an overload range from +9G to -3G, ensuring the challenging and realistic nature of the training environment. To further enhance the realism of the simulated environment, it can simulate strong winds reaching 25 m / s, changes in wind direction, thunderstorms, and extreme weather conditions such as rain and fog.

[0036] The simulation of this terminal is achieved through the following steps: Construction and data generation of a three-dimensional dynamic wind field model. This three-dimensional dynamic wind field model is constructed based on computational fluid dynamics (CFD) principles or empirical wind field models, such as logarithmic wind profiles and power-law wind profiles, which can reflect the continuous changes of wind speed and direction in space (e.g., horizontal and vertical directions) and time. The model can simulate atmospheric turbulence and gust effects by introducing random noise algorithms (e.g., turbulence spectrum models, stochastic Fourier synthesis, or random walk processes), generating a wind speed vector field Vwind(x,y,z,t) with spatiotemporal correlation. Wind direction data is calculated from the horizontal component of the wind speed vector, ensuring its continuous differentiability in three-dimensional space. The three-dimensional terrain model and the three-dimensional dynamic wind field model are collectively referred to as the dynamic battlefield environment model. Meteorological conditions are quantified into physical parameters, including wind speed vector Vwind, air density ρair, and visibility VISVIS. These parameters, as environmental inputs, directly affect the dynamics and sensor performance of the UAV.

[0037] In this optional embodiment, using a preset sensor attenuation model and dynamic battlefield environment, noise correction and feature enhancement are sequentially performed on multi-source sensor data to obtain a multi-source perception dataset. This includes: obtaining environmental state parameters under the current battlefield scenario based on the dynamic battlefield environment and using a UAV six-degree-of-freedom dynamic model, wherein the environmental state parameters include meteorological condition parameters and electromagnetic interference parameters; using the sensor attenuation model and combined with the environmental state parameters to determine the attenuation coefficient and measurement noise covariance of the visual sensor and lidar under the current battlefield situation; converting the attenuation coefficient and measurement noise covariance into correction parameters, and inputting the correction parameters into the visual sensor imaging data and lidar point cloud data, and sequentially identifying and separating noise components; performing noise correction processing on the multi-source sensor data with separated noise components to obtain benchmark perception data, and extracting features from the benchmark perception data to obtain multi-dimensional key feature information; performing feature enhancement processing on the multi-dimensional key feature information to enhance the feature's discriminability and effectiveness, thereby obtaining a multi-source perception dataset.

[0038] In this optional embodiment, determining the attenuation coefficients and measurement noise covariance of the visual sensor and lidar under the current battlefield situation using a sensor attenuation model combined with environmental state parameters includes: matching the atmospheric transmittance model of the visual sensor and the scattering model of the lidar using the sensor attenuation model; inputting meteorological condition parameters from the environmental state parameters into the attenuation model of the visual sensor to calculate the attenuation coefficient of the visual sensor; inputting meteorological and electromagnetic interference parameters from the environmental state parameters into the attenuation model of the lidar to calculate the attenuation coefficient of the lidar; and based on the calculated attenuation coefficients of the visual sensor and the lidar, sequentially quantifying the measurement noise covariance to obtain the attenuation coefficients and measurement noise covariance.

[0039] Specifically, the construction and calculation process of the UAV's six-degree-of-freedom dynamics model is as follows: This model is based on the Newton-Euler equations, including the UAV's translational equations and rotational equations about its center of mass. The model inputs include the UAV's state (i.e., position, velocity, attitude, and angular velocity), control variables (i.e., rotor speed or control surface deflection), and environmental parameters (Vwind, ρair). During the calculation, aerodynamic forces and aerodynamic torques are calculated based on the relative wind speed (i.e., airspeed). Combined with the propulsion system output and gravity, the linear and angular motion states of the UAV are obtained through integration. The wind disturbance torque is correlated with wind speed changes through aerodynamic derivatives, realizing the dynamic influence of the wind field on flight attitude and trajectory. The sensor attenuation model is constructed and calculated based on physical or empirical relationships. For example, visibility (VISVIS) affects the imaging sharpness and detection range of visual sensors (such as cameras), and image contrast attenuation can be quantified through atmospheric transmittance models (such as Koschmieder's law). Precipitation or fog droplets can also affect the point cloud density and ranging error of lidar, which can be addressed by modeling signal attenuation using Mie scattering theory or experimental calibration curves. During calculation, the attenuation coefficient is interpolated based on the current VISVIS parameters or calculated to correct the sensor's measurement noise covariance or effective detection range.

[0040] In this optional embodiment, the multi-source sensing dataset and multi-source state feature data are quantized into feature vectors, and the feature vectors are weighted and fused using a nonlinear transformation network to obtain a comprehensive situational representation vector. This includes: normalizing the multi-source sensing dataset and multi-source state feature data to obtain a standardized feature data set; performing dimensionality regularization and vectorization transformation on the feature data set to uniformly map discrete and continuous features into feature vector form to obtain a feature vector set; and using a nonlinear transformation network to perform nonlinear mapping and weighted fusion on the feature vector set to construct a comprehensive correlation relationship of multi-source features to obtain a comprehensive situational representation vector.

[0041] Specifically, the construction and decision-making of the reinforcement learning state space involves inputting the original weather parameters and their resulting flight state effects, such as airspeed differences, attitude deviations, and sensor attenuation results (i.e., multi-source state feature data), as part of the state space into the upper-level reinforcement learning agent. The agent then fuses weather information with tactical situational awareness through a policy network, learning to output robust decision commands adapted to complex weather conditions. These weather conditions simultaneously affect the UAV's perception and flight performance, forcing the agent to learn adaptive strategies in different environments during training.

[0042] Simultaneously, electromagnetic interference in the battlefield is simulated, affecting the UAV's communication and radar detection capabilities, increasing the complexity of training. The simulation environment of this invention also has the ability to dynamically adjust its difficulty and complexity to adapt to the needs of different training stages. The number, attack frequency, and accuracy of enemy units can be adjusted according to the training plan, thereby gradually increasing the challenge of training. The battlefield environment layout is also updated regularly, including the redistribution of obstacles and changes in terrain, to adapt to the needs of different training scenarios. Furthermore, various mission types are designed, such as reconnaissance, strike, and escort, each with its specific environment and enemy unit configuration. The simulated adversarial scenarios include different attack modes and tactics of enemy fighters, such as dogfights and long-range missile attacks, allowing UAVs to experience realistic air combat scenarios in a simulated environment.

[0043] In this optional embodiment, based on the comprehensive situational representation vector and the mission objective, multi-dimensional training objective parameters are constructed and weights are assigned. The weighted total reward is calculated using the weighted weighting results to obtain a hierarchical reward function system. This system includes: extracting four core dimensions—mission completion, survivability, energy efficiency, and tactical effectiveness—based on the comprehensive situational representation vector and the mission objective, and constructing multi-dimensional training objective parameters; assigning weight coefficients to the multi-dimensional training objective parameters according to the value principles of each weight coefficient, and obtaining weight allocation results; performing weighted calculations on the training objective parameters based on the weight allocation results and real-time quantization values, and summing the weighted results to calculate the weighted total reward; and integrating the multi-dimensional training objective parameters, weight allocation results, weighted total reward calculation logic, and quantization standards to obtain the hierarchical reward function system.

[0044] Specifically, this invention provides a clear incentive objective for UAV training in a simulated air combat environment by designing a hierarchical reward function. The reward metrics are defined as follows: This invention employs a comprehensive reward function to guide and train the reinforcement learning agent. This function consists of multiple well-defined evaluation metrics, aiming to balance task completion, survivability, energy efficiency, and tactical effectiveness. The metrics are defined as follows: Task Completion (D_task): Quantifies the progress of core tasks. For example, each destroyed verified enemy target adds +10 points to D_task. Survival Time (T_survival): Encourages sustained combat capability. Each successful time step a drone survives on the battlefield adds +1 point to T_survival. Energy Consumption (E_energy): Assesses energy usage efficiency. To encourage energy conservation, a penalty is set at each time step, for example, E_energy adds -0.1 × current power points. Tactical Effectiveness (E_tactics): Assesses the quality of tactical actions. For example, each successful evasion of an attack adds +5 points to E_tactics; being locked on by the enemy adds -3 points. The agent's immediate total reward (R_total) is calculated as follows: Where w1 to w4 are preset weighting coefficients used to precisely calibrate the relative importance of different objectives. Design reward function: Task completion reward function: Survival time reward function: (Assuming) (in seconds).

[0045] Energy efficiency reward R_energy: R_energy = -0.001 × P_total. This term is always negative; the higher the total energy consumption, the greater the penalty. Weight allocation: Assign weights to each reward function to reflect its importance in the overall training objective. Calculate the overall reward using the overall reward function: multiply each reward function by its weight and sum them to obtain the overall reward. This overall reward value will be used to evaluate every action the drone takes in a simulated air combat environment. This is the total reward value. , which is a weighting coefficient used to balance the relative importance of objectives such as mission completion, survival, energy, and tactics. The reward values ​​are calculated based on the number of mission objectives completed, survival time, energy consumption, and the effectiveness of tactical actions (such as successful evasion). Reward feedback is applied for strategy adjustment: During training, the UAV's policy network adjusts based on the overall reward. A positive reward indicates a beneficial action; a negative reward requires strategy adjustment to avoid similar actions. Real-time optimization: After performing actions in the simulated environment, the UAV immediately receives reward feedback, which is used to optimize the policy network parameters in real time. Through this hierarchical reward function design, this invention ensures that every action the UAV takes in simulated training is aimed at optimizing the four key indicators (i.e., multi-dimensional training target parameters), thereby improving the UAV's performance in actual air combat environments.

[0046] In this optional embodiment, an upper-layer reinforcement learning agent is used to evaluate the comprehensive situational representation vector and the hierarchical reward function system to obtain the optimal path instruction. This instruction is then sent to the lower-layer flight control terminal for execution. During execution, unexpected events are propagated forward to generate actions, resulting in an online optimized decision action. This includes: inputting the comprehensive situational representation vector and the hierarchical reward function system into the upper-layer reinforcement learning agent for collaborative evaluation to obtain an evaluation result; using the upper-layer reinforcement learning agent to perform value assessment and strategy selection on the evaluation result to obtain the optimal path instruction; sending the optimal path instruction to the lower-layer flight control terminal, which then executes the optimal path instruction to obtain the path instruction execution result; monitoring for unexpected events during the execution of the optimal path instruction; if an unexpected event occurs, using the upper-layer reinforcement learning agent to perform forward propagation calculations on the unexpected event to generate action instructions adapted to the new situation, thus obtaining the action instructions corresponding to the unexpected event; integrating the path instruction execution feedback with the action instructions generated by the unexpected event, and using the upper-layer reinforcement learning agent to complete the online optimization of the decision action to obtain the online optimized decision action.

[0047] Specifically, to test the adaptability and decision-making capabilities of the UAV, simulated contingency events were introduced, such as the sudden appearance of friendly units or the arrival of enemy reinforcements. These contingency events, by altering the state space of the simulated environment, were input in real-time to the upper-level reinforcement learning agent. The agent, based on its trained policy network and value network, re-evaluated the changed situation online, thereby quickly generating and executing new optimal tactical actions (i.e., online optimization of decision-making actions). Online re-evaluation specifically refers to an event-triggered instantaneous policy refresh mechanism: when a contingency event causes a sudden change in the environmental state S_t, the agent directly inputs the new state S_t' into its policy network π; this network, through a single forward propagation, immediately outputs an action instruction a_t' adapted to the new situation. Simultaneously, the value network rapidly estimates the new state-action pair, providing immediate feedback for the decision. Based on the inherent forward computation capability of neural networks, there is no need to traverse candidate solutions, thus enabling adaptive responses to contingency events within milliseconds. Essentially, it is a continuous mapping of the agent's internalized policy into the state space, rather than an external replanning process. This allows for the rapid generation and execution of new optimal tactical maneuvers, such as switching from solo attacks to coordinated operations, or from offensive maneuvers to defensive evasion. Driven by forward computation by the agent, it enables millisecond-level, model-based responses to unexpected events, rather than relying on pre-defined rules. These unexpected events require UAVs to react quickly and adjust their tactics accordingly.

[0048] In this optional embodiment, the process of using an upper-layer reinforcement learning agent to evaluate the evaluation results and select strategies to obtain the optimal path instruction includes: extracting attitude target parameters from the optimal path instruction from the evaluation results to clarify the target range of attitude adjustment; using the attitude control terminal of the upper-layer reinforcement learning agent to extract the attitude target parameters to obtain the current flight attitude data; calculating the attitude deviation between the current flight attitude data and the target attitude parameters; generating an attitude adjustment signal based on the attitude deviation result and correcting the attitude within the flight control range; and monitoring the adjusted flight attitude in real time to confirm that the attitude parameters have reached the target range, thereby obtaining the adjusted optimal path instruction.

[0049] Specifically, the layered action space provides UAVs with the ability to perform precise control and tactical decision-making in complex battlefield environments. The following are the specific implementation steps and parameters. Action space definition, speed control: The UAV's speed control action space includes the ability to continuously adjust from 0 m / s to a maximum speed of 250 m / s, enabling rapid response within an acceleration range of 0 to 100 m / s². This speed control is achieved by the upper-layer reinforcement learning agent outputting continuous acceleration commands, which are then resolved by the lower-layer flight control system into coordinated control signals for throttle and pitch attitude (i.e., control surface signals). Ultimately, continuous and rapid speed adjustment is achieved through the combined action of the engine / motor and control surface actuators.

[0050] Attitude Adjustment: In terms of attitude adjustment, the UAV can perform actions including pitch, roll, and yaw. The pitch angle ranges from -30 degrees to +30 degrees, the roll angle ranges from -60 degrees to +60 degrees, and the yaw angle ranges from continuous rotation (i.e., attitude target parameters). Flight Path Selection: The UAV can select the optimal flight path from a pre-set path library based on the battlefield environment and mission requirements. The pre-set path library is a set of parameterized flight paths pre-generated offline (e.g., based on tactical action library modeling, sampling search planning algorithms). Selecting the optimal flight path involves the upper-layer reinforcement learning agent receiving battlefield situation information in real time and rapidly evaluating the value of all candidate paths in the path library. Specifically, the value evaluation is performed by fusing the current situation with the features of each candidate path. The fused features are then batch-input into a trained value evaluation network; this network calculates an expected cumulative reward value for each path in parallel using a forward propagation method. This expected cumulative reward is a prediction of the total weighted reward that the UAV can obtain in the future after executing this path. The agent selects the path with the highest expected cumulative reward as the current optimal flight path. Tactical evasion maneuvers, such as sharp ascents, descents, rolls, and sharp turns, are executed within 1 to 5 seconds to ensure the drone can respond quickly to enemy attacks.

[0051] Action parameter optimization allows for dynamic adjustment of UAV action parameters to suit different battlefield situations and mission requirements. For example, adjusting maximum overload limits and minimum turning radius enables more precise flight control. Specifically, this dynamic adjustment is achieved by an upper-layer reinforcement learning agent: a dedicated policy network calculates and outputs temporary adjustment coefficients for parameters such as maximum overload. This dedicated policy network is a deep neural network whose calculation process involves encoding the real-time perceived enemy / ally status, mission phase, and environmental threats into a unified state vector. This vector undergoes forward propagation and feature transformation within the network's multi-layered nonlinear structure, and the network output layer generates one or more continuous adjustment coefficients α. These coefficients α are applied in real-time to the baseline flight envelope parameters; for example, dynamically adjusting the currently allowed maximum overload to: The network's parameters are optimized through co-training with the main reinforcement learning framework, enabling its output adjustment coefficients to maximize the UAV's long-term combat effectiveness. These coefficients, serving as a new flight envelope constraint, are transmitted in real-time to the underlying flight controller, allowing the UAV to break free from fixed parameter limitations and optimally balance maneuverability, structural load, and energy state based on the current situation. Continuous design: The action space is designed to be continuous, allowing the UAV to make infinitely fine adjustments, such as completing a 0.5-degree attitude adjustment within 0.1 seconds, or adjusting its speed by 2 meters per second within 1 second. Diverse implementation: Various tactical maneuver combinations are designed, such as rolls and sharp turns, rapid descents and ascents, to adapt to changing battlefield conditions and enemy attack patterns.

[0052] Action execution and feedback, real-time execution: The UAV's action execution system can respond to action commands defined in the action space in real time, quickly executing flight control and tactical maneuvers. Feedback adjustment: After executing an action, the UAV's sensor system provides real-time feedback on the execution results, such as changes in speed, attitude, and position. The system further optimizes action parameters based on the feedback results. Through the above-mentioned layered action space design, this invention enables UAVs to perform precise flight control and tactical decision-making in complex battlefield environments, improving the UAV's survivability and mission execution efficiency.

[0053] In this optional embodiment, multi-source temporal interaction data is constructed and stored in an experience replay buffer based on decision actions, comprehensive situational vectors, and hierarchical reward functions. The value network and policy network are updated according to the experience replay data, the target network is maintained, and iterative optimization is performed to obtain an autonomous decision-making and control strategy for the UAV, thereby achieving hierarchical reinforcement learning-based autonomous UAV air combat confrontation decision-making and control optimization. This includes: collecting and constructing multi-source temporal interaction data based on the decision actions executed by the UAV, comprehensive situational representation vectors, and hierarchical reward functions; storing the multi-source temporal interaction dataset in the experience replay buffer according to preset rules and performing data storage; maintaining the target network of the updated value network and policy network, and progressively updating the target network parameters using a soft update mechanism to obtain the maintained target network; repeatedly executing the network update and maintenance processes, and iteratively optimizing the value network, policy network, and target network to obtain an iteratively optimized network system; and integrating the learning results of the experience replay data with the network optimization results using the iteratively optimized network system to obtain an autonomous decision-making and control strategy for the UAV, thereby achieving hierarchical reinforcement learning-based autonomous UAV air combat confrontation decision-making and control optimization.

[0054] Specifically, hierarchical reinforcement learning algorithm training: A hierarchical reinforcement learning algorithm, particularly the Deep Deterministic Policy Gradient (DDPG) algorithm, is used to systematically train the UAV policy. The specific implementation steps and algorithm details are as follows: Initialize network parameters: Policy Network (ActorNetwork): Initialize the policy network parameters to generate actions that control the UAV. Value Network (CriticNetwork): Initialize the value network parameters to evaluate the expected rewards of the actions generated by the policy network. Collect initial data: Explore actions: The UAV performs a series of exploration actions in a simulated environment to collect initial experience data. Store experience: Store the initial experience data in an experience replay buffer, including states, actions, rewards, and new states.

[0055] The value network is trained by randomly sampling a batch of experiences from the experience replay buffer. The target Q-value is calculated from the sampled experiences using the formula: y = r + γ·CriticNetworkTarget(s', ActorNetworkTarget(s')), where r is the immediate reward returned by the environment; γ is a discount factor used to weigh long-term rewards; s' is the new state to which the environment transitions after the action is performed; CriticNetworkTarget and ActorNetworkTarget are the target networks of the value network and policy network, respectively, used for stable training.

[0056] The loss function of the value network is calculated as: L_critic = (1 / N)·Σ(y_i - Q(s_i,a_i;θ_critic))², where N is the batch size sampled from the experience replay pool; s_i and a_i are the state and action of the i-th sample in the batch, respectively; and θ_critic is the value network parameter to be trained. The value network parameter θ_critic is updated using gradient descent. y_i is the target Q-value of the i-th sample, calculated according to the formula y_i = r_i + γ·Q_target(s'_i,π_target(s'_i)), where r_i is the immediate reward and γ is the discount factor.

[0057] Training the policy network and calculating the policy gradient: The gradient of the policy network is calculated based on the value network, using the following formula: ;in, Let be the policy gradient, N be the batch size, si be the state, π be the policy network, θπ be its parameters, and Q be the value network. Output the gradient of actions to the value network. Output the gradients of the policy network with respect to its parameters. Update the policy network: using gradient ascent, based on the calculated policy gradients. Update the policy network parameters θπ'.

[0058] Target network soft update: Periodically update the target network parameters using a soft update mechanism, the formula is: , where τ is a small positive number, such as 0.001. θ represents the current parameters of the online network (i.e., the value network or policy network). θ_target represents the corresponding target network parameters. τ is a small positive soft update coefficient, for example, 0.005.

[0059] This mechanism ensures the stability of the target value by slowly mixing parameters, thereby promoting the convergence of the overall training. Iterative optimization and training are repeated: data collection, value network training, and policy network training steps are repeatedly executed to continuously optimize the UAV's policy. Through this hierarchical reinforcement learning algorithm training, this invention enables the UAV to improve its autonomous decision-making and control capabilities in complex battlefield environments through continuous learning and optimization in a simulated environment. This training method not only improves the training efficiency of the UAV but also ensures its effectiveness and safety in practical applications. Experience replay technology plays a crucial role, enabling the UAV to learn from historical interaction experiences in simulated air combat. Every step of the UAV's operation during mission execution, including its state, actions performed, rewards obtained, and subsequent state, is recorded as a data point and stored in the experience replay buffer. This buffer acts as a large memory bank, preserving the UAV's actions and results under different combat scenarios. To train the UAV's policy network, data points are randomly sampled from this buffer to avoid temporal correlation between samples, thereby enhancing the robustness of learning. Using these data points, the UAV's value network and policy network are continuously updated. The value network evaluates the expected returns of actions, while the policy network adjusts its action selection strategy accordingly to improve future returns. This process is iterative, allowing the UAV's decision-making strategy to gradually adapt to and optimize the simulated environment. Experience replay technology enhances the generalization capability of UAV strategies. Because the training data comes from a wide variety of historical combat scenarios, the strategies learned by the UAV are not scenario-specific but rather general strategies that perform well in various battlefield environments. As simulation training continues, the data in the experience replay buffer constantly increases, providing the UAV with rich learning materials, enabling its strategies to continuously improve and adapt, ultimately achieving higher combat effectiveness.

[0060] This invention utilizes target network technology. By maintaining a target network with the same structure as the main network but different parameters, and softly updating the target network parameters at a certain ratio τ, it effectively reduces policy fluctuations and enhances training stability during hierarchical reinforcement learning. τ is typically set around 0.001 to ensure the smoothness of the reward signal and the gradual improvement of the policy. Furthermore, this invention conducts comprehensive testing of UAVs in simulated environments, including simulating enemy fighter attack strategies in varied adversarial scenarios, evaluating the effectiveness of the UAV's strategies, and testing its robustness against randomly generated threats and obstacles, thereby accurately assessing the UAV's survivability and mission execution efficiency.

[0061] The UAV training strategy of this invention, fully validated in simulated environments, is deployed in real-time to actual UAV systems via wireless communication technology. Based on feedback data collected during actual flights, such as enemy fighter response patterns and changes in the battlefield environment, the strategy is updated and optimized online, ensuring that the UAV can quickly adapt to new threats and environmental changes, maintaining optimal performance at all times. This invention achieves real-time status awareness and rapid response for UAVs in air combat environments, optimizes mission execution processes, and enhances survivability and adaptability. These innovations not only improve the survivability and mission efficiency of UAVs in complex air combat environments but also provide new technical solutions for the application of UAVs in military and civilian fields.

[0062] This invention enables UAVs to understand their flight data in real time by accurately capturing their instantaneous flight status, including key parameters such as altitude, speed, attitude, and position. This data is obtained through high-precision sensors such as altimeters, pitot tubes, inertial measurement units (IMUs), and global positioning systems (GPS), ensuring the accuracy and real-time nature of the information. Advanced target recognition technology allows the UAV to identify and track enemy fighters, missiles, and obstacles on the battlefield. Utilizing radar, infrared sensors, and vision systems, the UAV can detect targets and analyze their characteristics, such as radar cross-section and thermal signature, to achieve accurate target classification and tracking. Beyond flight status monitoring and target recognition, this invention also includes in-depth analysis of the battlefield environment. By analyzing terrain data, environmental scan data, and geographic information system (GIS) data, the UAV can identify the distribution of obstacles and terrain features on the battlefield. Simultaneously, the location information of friendly units received through the communication system is integrated into the battlefield situation, providing the UAV with a comprehensive battlefield perspective. Multi-level state fusion technology integrates data from flight status monitoring, target recognition, and battlefield situation analysis to form a comprehensive situational awareness model (i.e., a comprehensive situational representation vector). This model not only enables UAVs to understand their own flight status at a microscopic level, but also to comprehend the entire battlefield environment at a macroscopic level, thereby achieving comprehensive situational awareness. Based on this, the decision support system provides UAV operators with real-time battlefield information and decision-making suggestions, enhancing the UAV's autonomous decision-making capabilities.

[0063] Figure 4 and Figure 6 An embodiment of a UAV air combat system based on hierarchical reinforcement learning according to the present invention is shown.

[0064] In this optional embodiment, the system includes: Battlefield environment construction module 201 is used to construct a three-dimensional terrain model using geographic information system data to simulate electromagnetic interference conditions; to construct a three-dimensional dynamic wind field model based on computational fluid dynamics principles to simulate meteorological conditions; and to obtain the dynamic battlefield environment based on the simulation results. The perception dataset generation module 202 is used to perform noise correction and feature enhancement on multi-source sensing data in sequence using a preset sensor attenuation model and dynamic battlefield environment to obtain a multi-source perception dataset. The situation fusion representation module 203 is used to quantize the multi-source sensing dataset and multi-source state feature data into feature vectors, and to use a nonlinear transformation network to perform weighted fusion of the feature vectors to obtain a comprehensive situation representation vector. The hierarchical reward construction module 204 is used to construct multi-dimensional training target parameters and assign weights based on the comprehensive situational representation vector and task objective, and to calculate the weighted total reward by calculating the weighted reward result to obtain the hierarchical reward function system. The decision instruction generation module 205 is used to evaluate the comprehensive situational representation vector and hierarchical reward function system using the upper-level reinforcement learning agent to obtain the optimal path instruction; and to send it to the lower-level flight control terminal for execution. During the execution process, it performs forward propagation of sudden events and generates actions to obtain the online optimized decision actions. The strategy learning optimization module 206 is used to construct multi-source temporal interaction data based on decision actions, comprehensive situation vectors and hierarchical reward functions, store it in the experience playback buffer, update the value network and policy network according to the experience playback data, maintain the target network and iteratively optimize it to obtain the UAV autonomous decision control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

[0065] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0066] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0068] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0070] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method and system for UAV air combat based on hierarchical reinforcement learning, characterized in that, include: A three-dimensional terrain model was constructed using geographic information system data to simulate electromagnetic interference conditions; A three-dimensional dynamic wind field model was constructed based on the principles of computational fluid dynamics to simulate meteorological conditions; Based on the simulation results, a dynamic battlefield environment is obtained; Using a pre-defined sensor attenuation model and a dynamic battlefield environment, noise correction and feature enhancement are sequentially applied to multi-source sensor data to obtain a multi-source sensing dataset. The multi-source sensing dataset and multi-source state feature data are quantized into feature vectors, and the feature vectors are weighted and fused using a nonlinear transformation network to obtain a comprehensive situational representation vector. Based on the comprehensive situational representation vector and the task objective, multi-dimensional training objective parameters are constructed and weights are assigned. The weighted total reward is calculated using the weighted weights to obtain a hierarchical reward function system. By utilizing an upper-level reinforcement learning agent, the comprehensive situational representation vector and hierarchical reward function system are evaluated to obtain the optimal path instruction; this instruction is then sent to the lower-level flight control terminal for execution, and unexpected events during the execution process are propagated forward to generate actions, resulting in online optimized decision actions. Based on decision-making actions, comprehensive situation vectors, and hierarchical reward functions, multi-source temporal interaction data is constructed and stored in an experience replay buffer. The value network and policy network are updated according to the experience replay data, the target network is maintained and iteratively optimized to obtain the UAV autonomous decision-making and control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

2. The UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The three-dimensional terrain model is constructed using geographic information system data to simulate electromagnetic interference conditions; A three-dimensional dynamic wind field model was constructed based on the principles of computational fluid dynamics to simulate meteorological conditions; Based on the simulation results, the dynamic battlefield environment includes: Based on the topographic features of mountains, plains, and oceans in the geographic information system data, a three-dimensional terrain model is constructed; Based on the principles of computational fluid dynamics, the spatiotemporally correlated wind speed vector field is obtained, and the wind direction data is calculated to obtain a three-dimensional dynamic wind field model. By using a three-dimensional terrain model and a three-dimensional dynamic wind field model, meteorological conditions and electromagnetic interference on the battlefield are simulated in a dynamic weather terminal to obtain the dynamic battlefield environment.

3. The UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The process involves using a pre-defined sensor attenuation model and a dynamic battlefield environment to sequentially perform noise correction and feature enhancement on multi-source sensor data, resulting in a multi-source perception dataset including: Based on the dynamic battlefield environment, and using the UAV six-degree-of-freedom dynamic model to obtain the environmental state parameters under the current battlefield scenario, wherein the environmental state parameters include meteorological condition parameters and electromagnetic interference parameters; By using a sensor attenuation model and combining environmental state parameters, the attenuation coefficients and measurement noise covariance of visual sensors and lidar under the current battlefield situation are determined. The attenuation coefficient and the measurement noise covariance are converted into correction parameters, and the correction parameters are input into the visual sensor imaging data and the lidar point cloud data to identify and separate the noise components in turn. Noise correction processing is performed on the multi-source sensing data from which noise components are separated to obtain benchmark sensing data, and feature extraction is performed on the benchmark sensing data to obtain multi-dimensional key feature information. Feature enhancement processing is performed on multi-dimensional key feature information to enhance the recognizability and effectiveness of features, resulting in a multi-source perception dataset.

4. The UAV air combat method based on hierarchical reinforcement learning according to claim 3, characterized in that, The process of using a sensor attenuation model and combining environmental state parameters to determine the attenuation coefficient and measurement noise covariance of visual sensors and lidar under the current battlefield situation includes: Using the sensor attenuation model, we matched the atmospheric transmittance model of the visual sensor and the scattering model of the lidar, respectively. The meteorological condition parameters from the environmental state parameters are input into the attenuation model of the visual sensor to calculate the attenuation coefficient of the visual sensor. The meteorological and electromagnetic interference parameters in the environmental state parameters are input into the attenuation model of the lidar to calculate the attenuation coefficient of the lidar. Based on the calculated attenuation coefficients of the visual sensor and the lidar, the measurement noise covariance is quantified sequentially to obtain the attenuation coefficient and the measurement noise covariance.

5. The UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The process of quantizing multi-source sensing datasets and multi-source state feature data into feature vectors, and then using a nonlinear transformation network to weight and fuse these feature vectors to obtain a comprehensive situational representation vector includes: The multi-source sensing dataset and multi-source state feature data are normalized to obtain a standardized feature data set. The feature data set is dimensionality normalized and vectorized to map discrete and continuous features into feature vector form, thus obtaining a feature vector set. By using a nonlinear transformation network, the feature vector set is nonlinearly mapped and weighted to construct a comprehensive correlation relationship of multi-source features, thereby obtaining a comprehensive situational representation vector.

6. The UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The process involves constructing multi-dimensional training target parameters and assigning weights based on the comprehensive situational representation vector and task objective, and then calculating the weighted total reward using the weighted reward results to obtain a hierarchical reward function system, including: Based on the comprehensive situational representation vector and mission objectives, four core dimensions are extracted: mission completion, survivability, energy efficiency, and tactical effectiveness, and multi-dimensional training objective parameters are constructed. Based on the principle of selecting the values ​​of each weight coefficient, weight coefficients are assigned to the multi-dimensional training target parameters to obtain the weight allocation results; Based on the weight allocation results and real-time quantization values, the training target parameters are weighted and the weighted results are accumulated and summed to calculate the weighted total reward. By integrating multi-dimensional training target parameters, weight allocation results, the calculation logic of weighted total reward, and quantification standards, a hierarchical reward function system is obtained.

7. The UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The upper-layer reinforcement learning agent is used to evaluate the comprehensive situational representation vector and hierarchical reward function system to obtain the optimal path command; this command is then sent to the lower-layer flight control terminal for execution. During execution, unexpected events are propagated forward and actions are generated, resulting in online optimized decision actions, including: The integrated situational representation vector and hierarchical reward function system are input into the upper-layer reinforcement learning agent for collaborative evaluation to obtain the evaluation results. The upper-level reinforcement learning agent is used to evaluate the evaluation results and select strategies to obtain the optimal path instructions. The optimal path command is sent to the lower-level flight control terminal, which then executes the optimal path command to obtain the path command execution result. During the execution of the optimal path instruction, monitoring for sudden events is performed. If a sudden event occurs, the upper-layer reinforcement learning agent is used to perform forward propagation calculations on the sudden event to generate action instructions that adapt to the new situation and obtain the action instructions corresponding to the sudden event. The path instruction execution feedback is integrated with the action instructions generated by the sudden event, and the upper-level reinforcement learning agent is used to complete the online optimization of the decision action, resulting in the online optimized decision action.

8. The UAV air combat method based on hierarchical reinforcement learning according to claim 7, characterized in that, The step of using an upper-layer reinforcement learning agent to evaluate the assessment results and select strategies to obtain the optimal path instruction includes: Extract the attitude target parameters from the optimal path command based on the evaluation results, and clarify the target range of attitude adjustment; The attitude control terminal of the upper-layer reinforcement learning agent extracts the attitude target parameters to obtain the current flight attitude data; and calculates the attitude deviation between the current flight attitude data and the target attitude parameters. Based on the attitude deviation results, an attitude adjustment signal is generated, and the attitude is corrected within the flight control range. The adjusted flight attitude is monitored in real time to confirm that the attitude parameters have reached the target range, and the optimal path command after adjustment is obtained.

9. A UAV air combat method based on hierarchical reinforcement learning according to claim 1, characterized in that, The process involves constructing multi-source temporal interaction data based on decision-making actions, comprehensive situation vectors, and hierarchical reward functions, storing it in an experience replay buffer, updating the value network and policy network based on the experience replay data, maintaining the target network, and iteratively optimizing it to obtain an autonomous decision-making and control strategy for the UAV. This achieves hierarchical reinforcement learning-based autonomous UAV air combat confrontation decision-making and control optimization, including: Based on the decision-making actions performed by UAVs, the comprehensive situational representation vector, and the hierarchical reward function, multi-source time-series interactive data is collected and constructed. The multi-source time-series interactive dataset is stored in the experience playback buffer and stored according to preset rules. The target network of the updated value network and policy network is maintained, and the parameters of the target network are incrementally updated using a soft update mechanism to obtain the maintained target network. Repeat the network update and maintenance process, and iteratively optimize the value network, policy network, and target network to obtain the iteratively optimized network system; By integrating the learning results from the experience playback data with the network optimization results using the iteratively optimized network architecture, an autonomous decision-making and control strategy for UAVs is obtained, thereby achieving hierarchical reinforcement learning-based autonomous air combat decision-making and control optimization for UAVs.

10. A method for UAV air combat based on hierarchical reinforcement learning, characterized in that, include: The battlefield environment construction module is used to build a three-dimensional terrain model using geographic information system data to simulate electromagnetic interference conditions. A three-dimensional dynamic wind field model was constructed based on the principles of computational fluid dynamics to simulate meteorological conditions; based on the simulation results, the dynamic battlefield environment was obtained. The perception dataset generation module is used to perform noise correction and feature enhancement on multi-source sensing data in sequence using a preset sensor attenuation model and dynamic battlefield environment to obtain a multi-source perception dataset. The situation fusion representation module is used to quantize multi-source sensing datasets and multi-source state feature data into feature vectors, and to use a nonlinear transformation network to perform weighted fusion of the feature vectors to obtain a comprehensive situation representation vector. The hierarchical reward construction module is used to construct multi-dimensional training target parameters and assign weights based on the comprehensive situational representation vector and task objective, and calculate the weighted total reward based on the weighted weighting results to obtain the hierarchical reward function system. The decision command generation module is used to evaluate the comprehensive situational representation vector and hierarchical reward function system using the upper-level reinforcement learning agent to obtain the optimal path command; and then send it to the lower-level flight control terminal for execution. During the execution process, it performs forward propagation of unexpected events and generates actions to obtain the online optimized decision actions. The strategy learning optimization module is used to construct multi-source temporal interaction data based on decision actions, comprehensive situation vectors and hierarchical reward functions, and store it in the experience replay buffer. It updates the value network and policy network based on the experience replay data, maintains the target network and iteratively optimizes it to obtain the UAV autonomous decision control strategy, so as to realize the UAV autonomous air combat confrontation decision and control optimization through hierarchical reinforcement learning.

Images