Reinforcement learning hierarchical training method and system based on behavior tree architecture

Abstract

This invention relates to the field of computer decision-making systems, specifically to a hierarchical training method for reinforcement learning based on a behavior tree architecture. The method includes steps S1 (constructing a behavior tree architecture for decision scheduling of an air combat agent), S2 (constructing reinforcement learning modules), S3 (achieving corresponding pairing of the behavior tree architecture and reinforcement learning modules through a proxy interface), S4 (setting activation / lock status identifiers in the behavior tree architecture and performing hierarchical training in a bottom-up order based on these identifiers), and S5 (collaborative fine-tuning). This invention, by constructing a hierarchical correspondence between the behavior tree architecture and reinforcement learning modules, introduces a bottom-up hierarchical training and collaborative fine-tuning mechanism, decoupling the complex air combat decision-making task in terms of function and training phase. This allows the agent's strategies at each level to converge in an orderly and independent manner, ultimately forming a high-performance decision-making system that meets the requirements of beyond-visual-range air combat simulation for decision accuracy, training efficiency, and verifiability.

Description

Technical Field

[0001] This invention belongs to the field of computer decision systems, specifically relating to a hierarchical training method and system for reinforcement learning based on a behavior tree architecture. Background Technology

[0002] Behavior trees (BTs), as a modular and highly interpretable decision logic modeling tool, are widely used in complex decision-making systems such as robot control, game AI, and military simulation to construct the behavioral logic of intelligent agents. Reinforcement learning (RL) can autonomously optimize policies through interaction with the environment. Combining the two, leveraging the structured scheduling capabilities of behavior trees and the adaptive optimization capabilities of reinforcement learning, has become an important direction for improving the decision-making performance of intelligent agents in dynamic environments. However, existing solutions for combining behavior trees and reinforcement learning suffer from problems such as unreasonable architectural design and inefficient training mechanisms, making it difficult to meet the stringent requirements of high-dynamic, tightly coupled scenarios such as AFSIM (Advanced Simulation, Integration, and Modeling Framework) beyond-visual-range air combat simulation for the accuracy and efficiency of agent decision-making.

[0003] Existing technical solutions mostly adopt relatively fixed combination modes, failing to fully leverage the synergistic potential of hierarchical training and structured scheduling. Chinese invention patent application CN202410825029.7 discloses a fighter jet flight control method based on adversarial skill embedding and hierarchical reinforcement learning. Although it employs a hierarchical reinforcement learning framework, utilizing high-level policies to output macro-maneuver skills and low-level policies to generate low-level control commands, its architecture lacks an explicit, modular logical scheduling structure like a behavior tree. In complex scenarios like AFSIM, which involve multiple intertwined tasks such as radar locking, missile launch, and maneuver evasion, this solution struggles to clearly define and dynamically adjust the execution logic, priorities, and temporal dependencies between sub-tasks. This results in poor interpretability of the agent's decision-making process, and high costs for strategy adjustment and verification when tactical scenarios change.

[0004] Chinese invention patent application CN202210831674.0 discloses a multi-agent hierarchical reinforcement learning-based multi-UAV air combat decision-making method. This method abstracts the decision-making process of a single UAV into two agents, a high-level agent and a low-level agent, and trains them using a semi-Markov game model. While this approach emphasizes collaboration among multiple agents, its decision-making logic is embedded in a neural network, lacking the structured and visual characteristics of behavior trees. In simulation tasks requiring strict adherence to military doctrine or rapid tactical logic iteration, this black-box decision-making model struggles to meet the engineering requirements for traceable and verifiable decision-making processes.

[0005] Furthermore, Chinese invention patent application CN2024109600298 discloses a method, apparatus, electronic device, and storage medium for fusing reinforcement learning with behavior trees. This application proposes a method for directly embedding learning nodes and reward nodes into behavior trees, enabling behavior tree nodes to possess autonomous learning capabilities. However, this approach disperses learning capabilities across individual nodes, lacking a systematic, hierarchical training strategy. In tasks like AFSIM beyond-visual-range air combat, which involve vast action spaces and extremely high state dimensions, this flat learning method leads to an explosion in the exploration space. The agent needs to engage in massive interactions with the environment to converge, resulting in extremely low training efficiency and failing to meet the real-time training requirements of simulation systems.

[0006] In summary, existing technologies generally suffer from core problems such as loose integration of behavior trees and reinforcement learning, lack of ordered hierarchical control in the training process, excessively large exploration space leading to slow convergence, and difficulty in balancing interpretability and training efficiency. Therefore, overcoming these technical problems and shortcomings is a key issue that needs to be addressed. Summary of the Invention

[0007] The purpose of this invention is to overcome the defects described in the background art, thereby realizing a reinforcement learning hierarchical training method based on behavior tree architecture, so as to solve the problems in the prior art such as fixed combination mode of behavior tree and reinforcement learning, disordered training process, excessive exploration space leading to low training efficiency, poor policy interpretability due to lack of structured scheduling of decision logic, and difficulty in meeting the requirements of real-time training and decision verifiability in complex air combat simulation.

[0008] To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0009] A hierarchical training method for reinforcement learning based on a behavior tree architecture includes the following steps:

[0010] Step S1: Construct a behavior tree architecture for decision-making and scheduling of air combat intelligent agents. The behavior tree architecture includes a high-level decision-making layer, a middle-level coordination layer, and a low-level execution layer, which are divided according to functional levels.

[0011] Step S2: Construct a reinforcement learning module, which is used to optimize policies through interaction with the environment, and divide the reinforcement learning module into high-level reinforcement learning units, mid-level reinforcement learning units and low-level reinforcement learning units according to the three functional levels of the behavior tree architecture.

[0012] Step S3: Implement the corresponding combination of the high-level decision-making layer, middle-level coordination layer, and low-level execution layer of the behavior tree architecture with the high-level reinforcement learning unit, middle-level reinforcement learning unit, and low-level reinforcement learning unit of the reinforcement learning module through the proxy interface.

[0013] The configuration includes: when the behavior tree architecture reaches the control class node, the control class node obtains the global state matrix through the proxy interface. After dimensionality reduction and extraction, the vector is used as the input to the corresponding level of the reinforcement learning unit; the reinforcement learning unit outputs the action instruction vector. The behavior tree architecture receives action instruction vectors through condition monitoring slots. And according to the action command vector Enables dynamic adjustment of the execution order or triggering conditions of child nodes in the behavior tree architecture.

[0014] Step S4: The nodes of the high-level decision layer, the middle-level coordination layer and the low-level execution layer of the behavior tree architecture are equipped with activation / lock status indicators, and based on the activation / lock status indicators, hierarchical training is performed in a bottom-up order. Hierarchical training includes low-level switching, middle-level switching and high-level switching.

[0015] Step S5: Collaborative Fine-tuning: Release the locked state of nodes in the high-level decision-making layer, the middle-level coordination layer, and the bottom-level execution layer, link all layers of reinforcement learning units and behavior tree architecture nodes, eliminate policy conflicts between layers, and verify policy generalization.

[0016] Specifically, in step S1, the air combat agent is an agent that makes tactical decisions and executes actions in a beyond-visual-range air combat simulation environment.

[0017] In step S4, activation means that the reinforcement learning network corresponding to a certain level node in the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer is in training mode, and the model parameters participate in training and updating. The air combat agent adopts the ε-greedy exploration strategy to explore and utilize the environment and update the network weights.

[0018] Locking refers to the reinforcement learning network corresponding to a node in one of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer reaching the convergence criterion. It switches to inference mode and freezes all weight parameters of the network, stops gradient updates, and provides a stable environment with extremely low non-stationarity for the training of other layers. The inference mode is a mode that freezes the network weight parameters and only performs forward computation.

[0019] In locked-down state, the air combat agent ceases exploration and only proceeds based on the current highest [value / status]. The output value represents a deterministic action.

[0020] The bottom-level switching in step S4 is as follows: fix and lock the node execution logic of the high-level decision-making layer and the middle-level coordination layer, activate the bottom-level reinforcement learning unit, perform reinforcement learning training on each leaf action node of the bottom-level execution layer until the bottom-level action execution error is less than 5% and the training is successful, and lock the node state of the bottom-level execution layer.

[0021] The middle-layer switching in step S4 is as follows: fix and lock the execution logic of the high-level decision-making layer node, activate the middle-layer reinforcement learning unit, link the locked low-level execution layer node, and perform reinforcement learning training on the control-type node of the middle-layer coordination layer. The reinforcement learning training aims to optimize the timing coordination of subtasks. When the timing coordination deviation is lower than a preset threshold for 10 consecutive simulation rounds, the training is successful, and the state of the middle-layer coordination layer node is locked. The preset threshold is the timing deviation success value, and the default value is 0.5 seconds.

[0022] The high-level switching in step S4 involves: activating the high-level reinforcement learning unit, linking the locked mid-level coordination layer and low-level execution layer nodes, and performing reinforcement learning training on the high-level decision-making layer nodes. This process continues until the global task completion rate is within a certain range. Training is successful when the fluctuation amplitude is less than 5% in the simulation.

[0023] The global task completion rate is the weighted average of the kill rate and the survival rate.

[0024] The collaborative fine-tuning in step S5 includes: adjusting the weights of the reward functions at each level through a weight search algorithm, and / or eliminating policy conflicts between levels through an adversarial training mechanism.

[0025] The adversarial training mechanism includes alternating adversarial training between trained air combat agents and enemy air combat agents employing random strategies, until the strategy's task completion rate in dynamic scenarios is not lower than a preset proportion in the baseline scenario; the preset proportion is a benchmark value for judging the generalization of the strategy, and is set to 85%.

[0026] Training is terminated when the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer are detected to converge, and the global task completion rate is not lower than the preset proportion for 10 consecutive times.

[0027] In the behavior tree architecture, the high-level decision-making layer includes the root node and the top-level control node, which is responsible for global combat decisions and core optimization of combat mode selection and target priority ranking.

[0028] The middle coordination layer includes intermediate control nodes, which are responsible for the timing coordination of subtasks and optimize the timing connection and collaboration logic between subtasks.

[0029] The underlying execution layer includes leaf action nodes, which are responsible for the execution of specific actions and optimize tactical maneuver accuracy, missile launch timing, and radar lock-on accuracy.

[0030] Furthermore, the division of the behavior tree architecture into high-level decision-making layer, mid-level coordination layer, and low-level execution layer adopts a node importance evaluation algorithm based on Monte Carlo sensitivity analysis. Specifically, it calculates the node importance by obtaining the change in global win rate after the node's action output is disturbed. Importance for: .

[0031] in, The global win rate is the change in global win rate after perturbing the action output of this node. The global win rate is the statistical probability that the air combat agent wins in simulated combat. The applied disturbance amplitude is normalized to the standard deviation of the node's action output. Normalized sensitivity of node actions to global win rate; based on importance. The nodes are divided into a high-level decision-making layer, a middle-level coordination layer, and a low-level execution layer: The nodes were assigned to the higher-level decision-making level. The nodes are assigned to the middle coordination layer. The nodes are assigned to the underlying execution layer.

[0032] The high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units of the reinforcement learning module are combined using differentiated algorithms:

[0033] The high-level reinforcement learning unit is paired with the high-level decision-making layer and uses a semi-Markov decision process-Option framework for high-level decision training. By utilizing its temporal abstraction capabilities, it allows high-level nodes to maintain macro-actions for a period of time until the termination condition is met.

[0034] The intermediate reinforcement learning unit is paired with the intermediate coordination layer and uses a temporal differential learning algorithm to be responsible for the reinforcement learning of subtask temporal coordination, so as to handle the sequential decision problem with strict temporal dependencies based on the step-by-step update feature.

[0035] The underlying reinforcement learning unit is paired with the underlying execution layer and uses the DQN algorithm in deep reinforcement learning to be responsible for reinforcement learning of specific action execution. It fits the Q-value function through a deep neural network, processes continuous dynamic parameters in the simulation, and outputs accurate discrete instructions.

[0036] The parameters used by the high-level reinforcement learning unit, mid-level reinforcement learning unit, and low-level reinforcement learning unit to control the reinforcement learning training process include learning rate, discount factor, exploration rate, and time window, which are set as follows:

[0037] The learning rate of the underlying reinforcement learning units using the DQN algorithm The value ranges from 0.0001 to 0.01, discount factor. The value ranges from 0.95 to 0.999, and the exploration rate is... The value decays from 1.0 to a preset value between 0.01 and 0.1, and can be dynamically adjusted during training.

[0038] The learning rate of the mid-level reinforcement learning unit using the temporal difference learning algorithm The value ranges from 0.001 to 0.01, and it is used to evaluate the timing coordination of subtasks. The timeout is set to 0.3 to 1.0 seconds based on the requirements of air combat tactics.

[0039] Learning rate of high-level reinforcement learning units using a semi-Markov decision process-Option framework The value ranges from 0.0001 to 0.001.

[0040] It also includes a tiered reward mechanism, which provides differentiated reward signals to high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units based on air combat tactical priorities.

[0041] The reward signal set for the underlying reinforcement learning unit adopts a penalty function composed of attitude angle error and energy loss rate to guide the air combat agent to fit the optimal envelope.

[0042] The reward signal set for the mid-level reinforcement learning unit adopts a time-window-based collaborative reward mechanism: setting a tactical time window. ,in The moment when the radar stably locks onto the target, The optimal launch window duration is preset.

[0043] If the missile launch time satisfy: Then a positive gain is given. ,in The reward signal value received by the middle-layer reinforcement learning unit; otherwise, a penalty is imposed. This is to guide the middle-level units to optimize the timing of radar lock-on and missile launch.

[0044] The reward signal set for the high-level reinforcement learning unit adopts a sparse reward based on situational advantage, which is triggered only when the enemy aircraft is finally killed or the interception mission is successfully completed, ensuring strategic consistency in decision-making.

[0045] During the underlying training process, an action masking mechanism is used to block illegal actions; the illegal actions are determined based on the current state constraints, so that illegal actions do not participate in the action selection.

[0046] In addition, this application also discloses a reinforcement learning hierarchical training system based on behavior tree architecture, used to implement the reinforcement learning hierarchical training method based on behavior tree architecture as described above, including:

[0047] The behavior tree architecture building module is used to build a behavior tree architecture adapted to AFSIM beyond-visual-range air combat simulation, generate node configuration XML files that can be imported into AFSIM, specify the node type, hierarchical relationship and execution logic, and support drag-and-drop configuration of nodes.

[0048] The hierarchical partitioning module is used to complete the hierarchical partitioning of the behavior tree architecture based on air combat tactical priorities and using a node importance evaluation algorithm, outputting the node composition and functional definition of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer.

[0049] The parameter initialization module is used to implement the unified configuration and dynamic adjustment of training parameters for high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units.

[0050] The state awareness module is used to connect to the simulation platform through the AFSIM platform API interface to collect simulation data on environmental state, aircraft state, and node execution state in real time. After data preprocessing, it provides data support for the reinforcement learning module.

[0051] The reward calculation module is used to generate differentiated reward signals for high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units based on the air combat tactical priority using a hierarchical reward mechanism.

[0052] The hierarchical training module is used to control the switching between the reinforcement learning units and the corresponding hierarchical nodes of the behavior tree architecture, execute the hierarchical training strategy, and support the switching and adaptation of various reinforcement learning algorithms.

[0053] The collaborative fine-tuning module is used to link high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units with behavior tree architecture nodes after training at all levels. It eliminates policy conflicts between levels by adjusting the weights of reward functions at each level and using adversarial training mechanisms, and verifies policy generalization.

[0054] The training termination judgment module is used to detect the convergence of the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer, and whether the global task completion rate in the AFSIM simulation scenario is not lower than the preset ratio for 10 consecutive times.

[0055] The storage module is used to store the behavior tree architecture configuration, training parameters, training logs, and the final policy model.

[0056] The beneficial effects of this invention are:

[0057] 1. The reinforcement learning hierarchical training method based on behavior tree architecture of the present invention systematically overcomes the shortcomings of existing technologies in combining behavior trees and reinforcement learning, such as loose architecture, disordered training, redundant exploration space, and difficulty in balancing interpretability and training efficiency, by constructing a hierarchical correspondence between behavior tree architecture and reinforcement learning modules, and introducing a bottom-up hierarchical training and subsequent collaborative fine-tuning mechanism based on activation and locked state identifiers. This invention decouples the complex air combat decision-making task in both the vertical functional and horizontal training stages, enabling the strategies of each level of the agent to converge in an orderly and independent manner. Ultimately, through global collaboration, a high-performance decision-making system with internal logical self-consistency and strong external adaptability is formed. From the perspectives of architecture design and training mechanism, this invention simultaneously meets the engineering requirements of beyond-visual-range air combat simulation for decision-making accuracy, training efficiency, and process verifiability.

[0058] 2. This invention divides the reinforcement learning module into a corresponding high-level decision-making layer, a mid-level coordination layer, and a low-level execution layer according to a behavior tree architecture. It employs a bottom-up hierarchical training strategy, compressing the massive exploration that would otherwise require a combined state and action space into sequential optimization within each level's constrained decision subspace. During training at each level, the policy of its lower layer is locked and a stable execution environment is provided, significantly reducing the non-stationarity of state transitions during training. This avoids policy oscillations and mutual interference caused by multi-level synchronous training, thereby significantly reducing the number of simulation interaction rounds required for the agent to converge, effectively compressing the training cycle and hardware resource overhead.

[0059] 3. This invention matches differentiated reinforcement learning algorithms and hierarchical reward mechanisms to different functional levels, enabling the bottom layer to excel in action execution, the middle layer to be sensitive to timing coordination, and the top layer to specialize in global planning. The optimization objectives of each level are clear and do not conflict with each other. The collaborative fine-tuning stage introduced after hierarchical training effectively eliminates inter-level policy conflicts that may arise from training decoupling by adjusting the weights of the reward functions of each level and introducing an adversarial training mechanism. This allows the final generated strategy to maintain stable task completion capabilities in dynamically changing simulation scenarios such as enemy aircraft type switching and electronic jamming intensity adjustments, without the need for repeated retraining for each scenario, thus fundamentally improving generalization and robustness.

[0060] 4. This invention encapsulates the adaptive optimization capabilities of reinforcement learning within a structured framework of behavior trees. The functional responsibilities of each node at each level of the behavior tree architecture are clearly defined, and the execution logic is traceable. Any decision made by the agent can be replayed and analyzed through node jumps and conditional branches within the behavior tree architecture, fully meeting the stringent requirements of military simulation for verifiable and debuggable decision-making processes. Simultaneously, the hierarchical training and node activation / locking switching mechanism effectively decouples the training process from simulation execution, balancing training efficiency and decision interpretability, thus resolving this long-standing core contradiction. Furthermore, the system's modular design allows for flexible adaptation to different air combat mission scenarios and simulation platforms, providing a directly applicable and complete solution for engineering deployment and tactical verification. Attached Figure Description

[0061] Figure 1 This is a system block diagram of the reinforcement learning hierarchical training method and system based on behavior tree architecture of the present invention;

[0062] Figure 2 This is a schematic diagram of the behavior tree and reinforcement learning hierarchical combination architecture of the reinforcement learning hierarchical training method and system based on behavior tree architecture of the present invention;

[0063] Figure 3 This is a flowchart of the node switching process of the reinforcement learning hierarchical training method and system based on behavior tree architecture of the present invention.

[0064] Figure 4 This is a flowchart of the hierarchical training process of the reinforcement learning hierarchical training method and system based on behavior tree architecture of the present invention. Detailed Implementation

[0065] The present invention will now be described in more detail through specific embodiments.

[0066] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship shown, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0067] Example 1

[0068] This embodiment discloses a hierarchical training method for reinforcement learning based on a behavior tree architecture. By constructing a three-layer behavior tree architecture including a high-level decision-making layer, a mid-level coordination layer, and a low-level execution layer, and dividing and assembling reinforcement learning modules according to the corresponding layers, and relying on the activation / locking status indicators of behavior tree nodes, a bottom-up hierarchical training strategy (low-level switching, mid-level switching, high-level switching) and subsequent collaborative fine-tuning are adopted to solve the problems in the existing technology, such as fixed combination mode of behavior tree and reinforcement learning, disordered training process, excessive exploration space leading to low training efficiency, poor policy interpretability due to lack of structured scheduling of decision logic, and difficulty in meeting the requirements of real-time training and decision verifiability in complex air combat simulation.

[0069] To achieve the above objectives, the main steps include:

[0070] Step S1: Construct a behavior tree architecture for decision-making and scheduling of air combat agents. This architecture comprises a high-level decision-making layer, a mid-level coordination layer, and a low-level execution layer, divided according to functional hierarchy. In this step, the air combat agent refers to an agent that makes tactical decisions and executes actions in a beyond-visual-range air combat simulation environment.

[0071] In this step, based on the abstraction level and time scale of the decision-making task, the behavior tree architecture is functionally divided vertically into three distinct levels: the high-level decision-making layer focuses on formulating macro-tactical objectives on a longer time scale (such as choosing offensive, defensive, or evasive behavior patterns); the middle-level coordination layer is responsible for decomposing and coordinating high-level objectives into a series of ordered atomic sub-task sequences (such as planning search, lock-on, and launch processes); and the bottom-level execution layer directly calls and controls specific action execution units in the simulation environment (such as outputting throttle, control surface, and other control commands or specific instructions for launching missiles).

[0072] The above steps provide a clear framework for the subsequent integration and training of reinforcement learning, enabling the separation of high-level strategic decision-making, mid-level task sequence planning, and low-level precise control. This reduces the complexity of a single decision-making model and preserves the inherent modularity and readability of behavior trees, laying the foundation for building an air combat intelligent agent decision-making system with both strong learning capabilities and good interpretability.

[0073] Step S2: Construct a reinforcement learning module, which is used to optimize policies through interaction with the environment, and divide the reinforcement learning module into high-level reinforcement learning units, mid-level reinforcement learning units and low-level reinforcement learning units according to the three functional levels of the behavior tree architecture.

[0074] In this step, based on the three functional layers established in step S1—the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer—the reinforcement learning module is synchronously and correspondingly instantiated into three independent policy learners: the high-level reinforcement learning unit, the middle-level reinforcement learning unit, and the low-level reinforcement learning unit.

[0075] Each unit is specifically responsible for optimizing decision-making strategies within its corresponding functional level. High-level units learn to select macro-level behavioral patterns, mid-level units learn to prioritize and coordinate sub-tasks, and low-level units learn to generate precise environmental control instructions.

[0076] Through the above steps, a clear separation and specialized optimization of decision-making responsibilities are achieved. Each reinforcement learning unit only needs to explore and learn within its defined, relatively simple and restricted decision subspace. This greatly reduces the complexity of the overall exploration and avoids the problem of slow or even difficult convergence faced by a single agent when training in a huge joint state-action space. This lays the necessary component foundation for the subsequent implementation of an efficient and stable hierarchical training process.

[0077] Step S3: Implement the corresponding combination of the high-level decision-making layer, middle-level coordination layer, and low-level execution layer of the behavior tree architecture with the high-level reinforcement learning unit, middle-level reinforcement learning unit, and low-level reinforcement learning unit of the reinforcement learning module through the proxy interface.

[0078] In this step, the proxy interface serves as the communication and data exchange hub between the two. The interface establishes a one-to-one binding and calling relationship between each functional node (high-level decision-making layer, middle-level coordination layer, and low-level execution layer) of the behavior tree architecture and its corresponding high-level reinforcement learning unit, middle-level reinforcement learning unit, and low-level reinforcement learning unit.

[0079] Through the above steps, a loosely coupled but functionally tight system integration method is created, which ensures that the behavior tree can make dynamic and intelligent decision branch selections using the output of reinforcement learning units, and enables each reinforcement learning unit to be clearly encapsulated and integrated within the structured framework of the behavior tree. This provides a precise and programmable control entry point for subsequent implementation of controllable hierarchical training based on node state identification.

[0080] See Figure 2 The configuration includes: when the behavior tree architecture reaches the control class node, the control class node obtains the global state matrix through the proxy interface. After dimensionality reduction and extraction, the vector is used as the input to the corresponding level of the reinforcement learning unit; the reinforcement learning unit outputs the action instruction vector. The behavior tree architecture receives action instruction vectors through condition monitoring slots. And according to the action command vector Enables dynamic adjustment of the execution order or triggering conditions of child nodes in the behavior tree architecture.

[0081] Specifically, the condition monitoring slot is a predefined data structure used to receive action instruction vectors output by the reinforcement learning unit. ; Control node parsing of behavior tree architecture The classification index value or continuous value in the tree is used to dynamically modify the running priority of its child nodes, or to directly assign the RUNNING state bit to a specific behavior subtree, thereby transforming the continuous numerical decision results of reinforcement learning into discrete logical jumps of the behavior tree and realizing the dynamic adjustment of the behavior process.

[0082] When the behavior tree reaches a control node that requires intelligent decision-making, the control node obtains the global state matrix representing the current battlefield situation through the established proxy interface. The system then performs dimensionality reduction and extraction to select the core features suitable for the current decision level, and then inputs the processed state into the reinforcement learning unit of the corresponding level.

[0083] After the policy network inside the reinforcement learning unit performs calculations, it outputs specific action instruction vectors. Action command vector The specific data structure of the condition monitoring slot is received by the behavior tree architecture. Subsequently, the core logic engine of the behavior tree architecture parses the action instruction vector. The instructions contained therein are used to dynamically change the execution order or triggering conditions of the child nodes of the behavior tree architecture in real time, thereby realizing dynamic behavior planning based on real-time perception and computation.

[0084] The above steps create a seamless closed loop from environmental perception and intelligent decision-making to logical execution. This not only clarifies the data (global state matrix)... and action command vector The specific paths and forms of flow between the two major modules are also designed through condition monitoring slots, which enable the output of reinforcement learning units to be precisely injected into and change the established control flow of the behavior tree architecture in a structured and interpretable manner. This not only preserves the clarity of the behavior tree architecture framework, but also endows the behavior tree architecture with a dynamically variable intelligent decision-making kernel, effectively solving the fundamental problem that static behavior tree architecture cannot adapt to complex dynamic environments.

[0085] Step S4: The nodes of the high-level decision layer, the middle-level coordination layer and the low-level execution layer of the behavior tree architecture are equipped with activation / lock status indicators, and based on the activation / lock status indicators, hierarchical training is performed in a bottom-up order. Hierarchical training includes low-level switching, middle-level switching and high-level switching.

[0086] See Figure 4In this step, activation means that the reinforcement learning network corresponding to a certain level node in the high-level decision-making layer, the middle-level coordination layer, and the bottom-level execution layer is in training mode, and the model parameters participate in training and updating. The air combat agent adopts the ε-greedy exploration strategy to explore and utilize the environment and update the network weights.

[0087] Locking refers to the reinforcement learning network corresponding to a node in one of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer reaching the convergence criterion. It switches to inference mode and freezes all weight parameters of the network, stops gradient updates, and provides a stable environment with extremely low non-stationarity for the training of other layers. The inference mode is a mode that freezes the network weight parameters and only performs forward computation.

[0088] In locked-down state, the air combat agent ceases exploration and only proceeds based on the current highest [value / status]. The output value represents a deterministic action.

[0089] In this embodiment, the behavior tree architecture established in the aforementioned steps and assembled through the proxy interface, along with the reinforcement learning module, are treated as a whole, and a controlled, phased training strategy is implemented. Each node in the behavior tree architecture (high-level decision-making layer, mid-level coordination layer, and low-level execution layer) is assigned a dynamic activation / locking state identifier. Activation means that the reinforcement learning unit corresponding to the node is in a trainable state, and its network parameters are updated through experience collected by ε-greedy exploration; locking indicates that the reinforcement learning unit has reached the preset convergence criteria (such as low-level action execution error less than 5%, mid-level temporal coordination deviation less than 0.5 seconds for 10 consecutive rounds, and high-level global task completion rate fluctuation less than 5%), and its network parameters are frozen, providing only deterministic decision output.

[0090] Specifically, see Figure 3 In this step, the bottom layer is switched to: fixing and locking the node execution logic of the high-level decision-making layer and the middle-level coordination layer, activating the bottom layer reinforcement learning unit, and performing reinforcement learning training on each leaf action node of the bottom layer until the bottom layer action execution error is less than 5% and the training is successful, and locking the node state of the bottom layer execution layer.

[0091] The middle layer switching is as follows: the execution logic of the high-level decision-making layer nodes is fixed and locked, the middle layer reinforcement learning unit is activated, and the locked lower-level execution layer nodes are linked to perform reinforcement learning training on the control-type nodes of the middle layer coordination layer. The reinforcement learning training aims to optimize the timing coordination of subtasks. When the timing coordination deviation is lower than a preset threshold for 10 consecutive simulation rounds, the training is successful, and the state of the middle layer coordination layer nodes is locked. The preset threshold is the timing deviation success value, and the default value is 0.5 seconds.

[0092] The high-level switch proceeds as follows: Activate the high-level reinforcement learning unit, link the locked mid-level coordination layer and low-level execution layer nodes, and perform reinforcement learning training on the high-level decision-making layer nodes. This is done when the global task completion rate is continuously... Training is considered successful when the fluctuation range is less than 5% in the simulation; the global task completion rate is the weighted average of the kill rate and the survival rate.

[0093] Based on the above state control mechanism, training follows a bottom-up sequence, sequentially switching between lower layers, middle layers, and higher layers:

[0094] First, during the bottom-level switching phase, the high-level decision-making layer and the mid-level coordination layer are locked, and only the bottom-level reinforcement learning units are activated and trained to refine their basic action control. During the mid-level switching phase, the high-level decision-making layer remains locked, and the mid-level reinforcement learning units are activated, learning to coordinate sub-task sequences with the support of the locked, stable bottom-level execution layer. Finally, during the high-level switching phase, the high-level reinforcement learning units are activated, learning to formulate global tactics based on the reliable "skill library" formed by the locked mid-level coordination layer and the bottom-level execution layer.

[0095] Through the above steps, the complex multi-level reinforcement learning problem is decomposed into a series of serial, simpler sub-problems by activating / locking state flags. The training of each level is based on the environment built by the converged and stable policy of its lower level, which can greatly reduce the non-stationarity of state transitions during training, effectively avoid policy oscillations and interference, and thus significantly improve the overall training efficiency, stability and reliability of the final policy.

[0096] Step S5: Collaborative Fine-tuning: Release the locked state of nodes in the high-level decision-making layer, the middle-level coordination layer, and the bottom-level execution layer, link all layers of reinforcement learning units and behavior tree architecture nodes, eliminate policy conflicts between layers, and verify policy generalization.

[0097] In this step, collaborative fine-tuning includes adjusting the weights of the reward functions at each level using a weight search algorithm, and / or eliminating inter-level policy conflicts through an adversarial training mechanism. The weight search algorithm employs a grid search combined with Bayesian optimization, searching within a preset weight space for the combination of reward weights at each level that maximizes the global task completion rate. The adversarial training mechanism involves alternating between a trained air combat agent and an enemy air combat agent employing a random strategy, conducting 200 rounds of adversarial training to eliminate inter-level policy conflicts.

[0098] The adversarial training mechanism includes alternating adversarial training between trained air combat agents and enemy air combat agents employing random strategies, until the strategy's task completion rate in dynamic scenarios is not lower than a preset proportion in the baseline scenario; the preset proportion is a benchmark value for judging the generalization of the strategy, and is set to 85%.

[0099] Training is terminated when the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer are detected to converge, and the global task completion rate is not lower than the preset proportion for 10 consecutive times.

[0100] After completing rigorous bottom-up hierarchical training and locking each level, a global optimization phase is introduced. In this phase, collaborative fine-tuning first unlocks the nodes of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer, so that reinforcement learning units at all levels are back in a fine-tunable state.

[0101] Next, overall optimization will be carried out through two core mechanisms:

[0102] By employing a weighted search algorithm, the relative weights of multiple reward functions driving policy learning at the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer are systematically adjusted to find a better reward balance point under the global objective, thereby eliminating policy conflicts between levels.

[0103] An adversarial training mechanism is introduced, in which trained air combat agents are placed in repeated alternating adversarial situations against enemy air combat agents using random strategies. Through this highly dynamic and uncertain stress test, the agents are forced to adjust their strategies to cope with situations outside the training set, thereby verifying the generalization of the strategies.

[0104] In this embodiment, the criterion for achieving generalization is that the global task completion rate in a dynamic adversarial scenario is not less than 85% of that in the baseline scenario. The collaborative fine-tuning process continues until the strategies at each level converge again, and the global task completion rate remains stable at or above this percentage in 10 consecutive evaluations.

[0105] Through the above, a strategy of initial division followed by integration and overall optimization was achieved. While ensuring the stability of basic skills at each level (through layered training and locking), a global weight search algorithm was used to align the interests of each level in terms of optimization objectives, and an adversarial training mechanism was used to refine the strategy's robustness against environmental challenges. This effectively avoids the problem of local optima rather than global coordination that may result from layered training, ultimately producing a high-performance and highly reliable air combat agent strategy that is internally consistent and adaptable to unknown challenges externally.

[0106] To achieve an objective and quantitative functional hierarchy division of complex behavior tree nodes, and to support targeted hierarchical training, the behavior tree architecture in this embodiment includes a high-level decision-making layer consisting of a root node and a top-level control node, which is responsible for global combat decisions and core optimization of combat mode selection and target priority ranking.

[0107] The middle coordination layer includes intermediate control nodes, which are responsible for the timing coordination of subtasks and optimize the timing connection and collaboration logic between subtasks.

[0108] The underlying execution layer includes leaf action nodes, which are responsible for the execution of specific actions and optimize tactical maneuver accuracy, missile launch timing, and radar lock-on accuracy.

[0109] Furthermore, the division of the behavior tree architecture into high-level decision-making layer, mid-level coordination layer, and low-level execution layer adopts a node importance evaluation algorithm based on Monte Carlo sensitivity analysis. Specifically, it calculates the node importance by obtaining the change in global win rate after the node's action output is disturbed. Importance for: .

[0110] in, The global win rate is the change in global win rate after perturbing the action output of this node. The global win rate is the statistical probability that the air combat agent wins in simulated combat. The applied disturbance amplitude is normalized to the standard deviation of the node's action output. This represents the normalized sensitivity of node actions to the global win rate.

[0111] Based on importance The nodes are divided into a high-level decision-making layer, a middle-level coordination layer, and a low-level execution layer: The nodes were assigned to the higher-level decision-making level. The nodes are assigned to the middle coordination layer. The nodes are assigned to the underlying execution layer.

[0112] This formula originates from Monte Carlo sensitivity analysis. It quantifies the importance of nodes by observing their impact on the final global win rate through experiments that perturb node behavior in a simulation environment. Both sides of the formula are dimensionless quantities with consistent dimensions. The aforementioned threshold... and Pre-experimental statistics on multiple typical AFSIM beyond-visual-range air combat simulation scenarios determined that: In the node, perturbing the node's action output causes the global task failure rate to increase by more than 30%, indicating that it has a significant strategic global impact and is suitable as a training target for high-level decision-making. The nodes have a relatively small impact on the global win rate but are executed frequently, making them suitable as training targets for fine-grained control at the lower level.

[0113] In this embodiment, the core responsibilities of each level are first clearly defined: the high-level decision-making layer (including root nodes and top-level control nodes) focuses on macro-level combat mode selection and target priority ranking; the mid-level coordination layer (including intermediate-level control nodes) is responsible for the micro-level timing connection and coordination logic between sub-tasks; and the bottom-level execution layer (including leaf action nodes) optimizes specific tactical maneuver accuracy, missile launch timing, and radar lock-on accuracy.

[0114] To achieve a precise mapping from node functionality to the aforementioned three-layer structure, this embodiment employs a node importance evaluation algorithm based on Monte Carlo sensitivity analysis. The core of this algorithm is to quantify the impact of each node's action output on the success or failure of the overall task. The specific measurement method is as follows:

[0115] Through extensive random simulations, calculations are performed for a given node. The importance of a node is defined as the change in global win rate caused by applying a standardized perturbation (in units of the standard deviation of the node's action output) to its action output. The calculation formula is as follows: Importance Essentially, it's a normalized sensitivity that measures the average impact of small changes in node actions on the final win rate. During the partitioning, it will... The key points, that is, those that have a significant impact on the overall victory or defeat, are assigned to the high-level decision-making level; Nodes with moderate influence are assigned to the middle-level coordination layer; Nodes that affect relatively local areas are assigned to the lower-level execution layer.

[0116] The above steps provide a data-driven, objective, and repeatable hierarchical classification standard to replace subjective, experience-based classification. This is achieved through Monte Carlo simulation and importance... The calculation transforms the functional importance of nodes into measurable numerical indicators, thereby ensuring that the division of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer is strictly aligned with the actual contribution of nodes in real tasks. This provides a solid logical basis and structural foundation for implementing a bottom-up sequential training order (training the low-level execution layer that affects the local area first, and then training the high-level decision-making layer that affects the overall situation), thus fundamentally guaranteeing the effectiveness of the hierarchical training strategy.

[0117] Example 2

[0118] The similarities to the above embodiments will not be repeated, the differences are as follows:

[0119] To achieve accurate matching between the decision characteristics of different functional levels in the behavior tree architecture and the reinforcement learning training objectives, and to avoid training instability, decision lag, or control inaccuracy caused by using a single algorithm, in this embodiment, the high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units of the reinforcement learning module are combined using differentiated algorithms:

[0120] The high-level reinforcement learning unit is paired with the high-level decision-making layer. Since the single-step decision-making algorithm such as DQN in deep reinforcement learning is directly used, the high-level decision will switch at each simulation step, resulting in frequent jitter of tactical intentions, which does not conform to the time span characteristics of macro decision-making in air combat. Therefore, this embodiment adopts the semi-Markov decision process - Option framework for high-level decision training, which utilizes its time abstraction capability to allow high-level nodes to maintain macro actions for a period of time until the termination condition is met.

[0121] This embodiment combines a high-level reinforcement learning unit with a high-level decision-making layer and trains it using a semi-Markov decision process-Option framework. The semi-Markov decision process-Option framework has temporal abstraction capabilities, which can encapsulate a series of low-level actions into macro actions with clear termination conditions. This allows high-level decision-making nodes to maintain the macro actions over a longer time scale after making decisions such as combat mode selection or target priority ranking, without having to recalculate for each simulation step. This naturally conforms to the inherent time persistence characteristics of macro-tactical decision-making in air combat.

[0122] Through the above steps, the semi-Markov decision process-Option framework enables high-level reinforcement learning units to trigger re-decision only at key tactical turning points. While compressing the high-level exploration space, it ensures the stability and logical consistency of the overall combat mode from a mechanistic perspective.

[0123] The intermediate reinforcement learning unit is paired with the intermediate coordination layer and uses a temporal differential learning algorithm to be responsible for the reinforcement learning of subtask temporal coordination, so as to handle the sequential decision problem with strict temporal dependencies based on the step-by-step update feature.

[0124] For the combination of the mid-level reinforcement learning unit and the mid-level coordination layer, a temporal difference learning algorithm is adopted. The temporal difference learning algorithm has the ability to update the value estimate immediately based on the single-step state transition, and can realize error backpropagation and policy improvement at each step without waiting for the end of a complete air combat round.

[0125] In the coordination of sub-tasks with strict temporal dependencies, such as radar detection, target lock, missile launch, and guidance illumination, action feedback often has delays and cross-step credit allocation problems. If Monte Carlo-like methods are used, updates can only be made at the end of the round, which will prevent the middle coordination layer's strategy from adjusting timing parameters in a timely manner when encountering dynamically changing enemy countermeasures or electromagnetic interference.

[0126] The temporal difference learning algorithm, through its step-by-step update characteristic, can capture temporal deviations in real time and distribute reward signals to relevant decision nodes in stages, thereby rapidly optimizing the connection logic of subtasks. The mid-level reinforcement learning unit can agilely fine-tune the optimal time window for radar locking and missile launch in dynamic confrontation, greatly improving the coherence of the task flow and real-time response capability.

[0127] The underlying reinforcement learning unit is paired with the underlying execution layer and uses the DQN algorithm in deep reinforcement learning to be responsible for reinforcement learning of specific action execution. It fits the Q-value function through a deep neural network, processes continuous dynamic parameters in the simulation, and outputs accurate discrete instructions.

[0128] The combination of the bottom reinforcement learning unit and the bottom execution layer adopts the DQN algorithm in deep reinforcement learning. The DQN algorithm in deep reinforcement learning uses a deep neural network to directly fit the state-action value function, extract high-dimensional features from continuous dynamic parameters composed of aircraft attitude angle, relative speed, radar signal-to-noise ratio, etc., and outputs the accurate evaluation value of each action on a finite discrete action set.

[0129] Because the underlying execution layer deals with definite and discrete actions such as tactical maneuvers, radar mode switching, and missile launch commands, the DQN algorithm in deep reinforcement learning can avoid the smoothness constraints imposed by continuous action reinforcement learning algorithms, thereby directly outputting deterministic commands and ensuring the decisiveness and accuracy of the underlying control.

[0130] Through the above steps, the bottom-level reinforcement learning unit not only makes full use of the deep network's ability to perceive and generalize continuous simulation states, but also ensures precise control of the high-maneuverability flight of the fighter jet and the weapon delivery process through structural matching of the discrete action space, providing a solid and reliable execution foundation for the effective implementation of the upper-level strategy.

[0131] To achieve fine-grained control over the training process of each level of reinforcement learning unit and avoid training oscillations, convergence stagnation, or policy overfitting due to improper hyperparameter settings, in this embodiment, the parameters used to control the reinforcement learning training process of the high-level reinforcement learning unit, the middle-level reinforcement learning unit, and the low-level reinforcement learning unit include learning rate, discount factor, exploration rate, and time window, which are set as follows:

[0132] The learning rate of the underlying reinforcement learning units using the DQN algorithm The value ranges from 0.0001 to 0.01, discount factor. The value ranges from 0.95 to 0.999, and the exploration rate is... The value decays from 1.0 to a preset value between 0.01 and 0.1, and can be dynamically adjusted during training.

[0133] The underlying reinforcement learning unit, which employs the DQN algorithm in deep reinforcement learning, needs to learn precise discrete action values ​​from a high-dimensional state space containing continuous dynamic parameters such as aircraft attitude angle, relative speed, and radar signal-to-noise ratio. The learning rate α is limited to the range of 0.0001 to 0.01 because the weight updates of deep neural networks are sensitive to this. A lower lower bound ensures that the value network avoids gradient divergence when approaching non-stationary targets, while an upper bound of 0.01 provides sufficient convergence driving force in the early stages of training, enabling the action accuracy of each leaf action node in the underlying execution layer to steadily improve.

[0134] The discount factor γ is set between 0.95 and 0.999. This high range, close to 1, ensures that long-term tactical gains (such as survival status or kill results at the end of the round) are fully backpropagated to current tactical maneuvers or missile launch decisions, preventing short-sighted underlying strategies from ignoring the long-term consequences of actions due to excessive discounting; exploration rate The detection rate decreases from 1.0 to a preset value between 0.01 and 0.1, and supports dynamic adjustment during training. The purpose is to force the underlying reinforcement learning units to fully traverse the combination space of various tactical maneuvers such as level flight, climb, dive, and turn with radar mode switching in the early stage, and gradually reduce the detection rate as training progresses. This allows the agent to shift from exploration to exploitation, while the ability to dynamically adjust retains the flexibility to explore again in response to sudden changes in the enemy's strategy. Through the above methods, the underlying reinforcement learning units can robustly converge to high-precision underlying actions during training, while avoiding getting trapped in local optima and outputting deterministic discrete instructions.

[0135] The learning rate of the mid-level reinforcement learning unit using the temporal difference learning algorithm The value ranges from 0.001 to 0.01, and it is used to evaluate the timing coordination of subtasks. The timeout is set to 0.3 to 1.0 seconds based on the requirements of air combat tactics.

[0136] The learning rate α of the mid-level reinforcement learning unit employing the temporal difference learning algorithm is set between 0.001 and 0.01. This is because the temporal difference learning algorithm relies on immediate updates to the value estimate during single-step state transitions. The mid-level coordination layer's sub-task temporal coordination (such as the transition from radar detection to target lock to missile launch) requires the policy to respond agilely to temporal deviations. A learning rate slightly higher than the lower bound of the lower layers ensures that each single-step update effectively corrects the value estimate, promptly identifying and adjusting for premature or late triggering issues between radar lock-up and missile launch. A temporal window is used to evaluate the sub-task temporal coordination. Based on the tactical requirements of air combat, the timeframe is set to 0.3 to 1.0 seconds. This range corresponds to the optimal window period in beyond-visual-range air combat from obtaining stable fire control calculations to missile launch. If the window is shorter than 0.3 seconds, the launch opportunity may be missed due to insufficient reaction time; if it is longer than 1.0 seconds, the probability of situational changes caused by enemy maneuvers or electronic jamming increases sharply. By controlling the timing window... By confining the process to this range, the mid-level reinforcement learning units can be guided to find and maintain the optimal launch rhythm in the simulation. Through this method, the mid-level reinforcement learning units can accurately optimize the timing connection and collaborative logic of sub-tasks in dynamic adversarial scenarios, significantly improving the timing consistency from radar lock-on to missile launch.

[0137] Learning rate of high-level reinforcement learning units using a semi-Markov decision process-Option framework The value ranges from 0.0001 to 0.001.

[0138] The learning rate of the high-level reinforcement learning unit employing the semi-Markov decision process-Option framework is... The value range is set between 0.0001 and 0.001 because the high-level decision-making level is responsible for overall operational decisions such as the selection of combat modes and the prioritization of targets. Once these macro actions are initiated, they should remain stable for a period of time until the preset termination conditions are met.

[0139] If learning rate If the value function of the high-level reinforcement learning unit is too high, it will fluctuate drastically due to individual sudden states, resulting in frequent switching of macro actions, jittering of tactical intentions, and thus disrupting the stable execution chain established by the middle coordination layer and the bottom execution layer.

[0140] Extremely low learning rate This reduces the sensitivity of high-level reinforcement learning units to single-simulation rewards, and robustly shapes the globally optimal policy by slowly accumulating situational advantage reward signals across rounds. Through this method, high-level reinforcement learning units can be deeply matched with the temporal abstraction capabilities of the semi-Markov decision process-Option framework, compressing the high-level exploration space while ensuring end-to-end decision-making coherence and tactical consistency from the high-level decision-making layer to the mid-level coordination layer and then to the low-level execution layer.

[0141] Example 3

[0142] The similarities with the above embodiments and their combinations will not be repeated, the differences being:

[0143] To achieve precise guidance for different decision-making objectives of high-level, mid-level, and low-level reinforcement learning units within the behavior tree architecture, and to avoid ambiguity in credit allocation and policy conflicts between levels caused by a single global reward signal, this embodiment also includes a hierarchical reward mechanism that provides differentiated reward signals to high-level, mid-level, and low-level reinforcement learning units based on air combat tactical priorities.

[0144] The reward signal set for the underlying reinforcement learning unit adopts a penalty function composed of attitude angle error and energy loss rate to guide the air combat agent to fit the optimal envelope.

[0145] The underlying reinforcement learning unit directly controls the fighter jet's tactical maneuvers and weapon execution. The reward signal adopts a penalty function composed of attitude angle error and energy loss rate. In beyond-visual-range air combat, attitude angle error directly affects radar interception probability, missile launch off-axis angle, and flight drag, while energy loss rate characterizes the fighter jet's potential to maintain sustained maneuverability. By using attitude angle error and energy loss rate as negative penalties, the air combat agent is guided to minimize attitude angle error and suppress unnecessary energy loss in every action of the underlying execution layer, thereby driving the agent's flight state to gradually approach the optimal energy maneuver envelope.

[0146] The reward signal set for the mid-level reinforcement learning unit adopts a time-window-based collaborative reward mechanism: setting a tactical time window. ,in The moment when the radar stably locks onto the target, The optimal launch window duration is preset.

[0147] If the missile launch time satisfy: Then a positive gain is given. ,in The reward signal value received by the middle-layer reinforcement learning unit; otherwise, a penalty is imposed. This is to guide the middle-level units to optimize the timing of radar lock-on and missile launch.

[0148] The intermediate reinforcement learning unit is responsible for the temporal coordination between subtasks, and its reward signal adopts a collaborative reward mechanism based on time windows. A tactical timing window is set. ,in The moment when the radar stably locks onto the target, To preset the optimal launch window duration. If the missile launch time... satisfy If the missile is launched within the optimal window after locking on, a positive gain is granted. This indicates an excellent timing sequence; if If the launch falls outside the window, including launching too early or too late, a penalty will be imposed. .

[0149] This mechanism captures the rationality of the delay time between radar lock and missile launch from the root. Through the step-by-step update characteristics of the temporal differential learning algorithm adopted by the mid-level reinforcement learning unit, the mid-level reinforcement learning unit can quickly perceive the timing deviation and adjust the strategy, thereby accurately optimizing the timing connection between radar lock and missile launch, and ensuring that the weapon is delivered before the enemy can carry out effective evasion or electronic jamming.

[0150] The reward signal set for the high-level reinforcement learning unit adopts a sparse reward based on situational advantage, which is triggered only when the enemy aircraft is finally killed or the interception mission is successfully completed, ensuring strategic consistency in decision-making.

[0151] The high-level reinforcement learning unit undertakes global decisions such as combat mode selection and target priority ranking. Reward signals are sparse, based on situational advantage, and are triggered only upon the final kill of an enemy aircraft or successful interception. The ultimate victory in air combat is the highest criterion for judging the quality of global decisions; various local situational changes during the intermediate process cannot reliably reflect the correctness of macro-tactical intentions. Rewarding only the final event avoids frequent tactical mode switching or short-sighted behavior by the high-level reinforcement learning unit in pursuit of local rewards, ensuring that the evaluation of high-level strategies strictly focuses on the strategic consistency of the mission's end outcome. Through this scheme, the hierarchical reward mechanism forms a progressively focused and non-conflicting incentive system from the bottom-level action precision and mid-level timing coordination to the top-level strategic objectives. This allows each level of reinforcement learning unit to converge independently under the guidance of its respective optimization goals and more easily achieve global optimum through collaborative fine-tuning.

[0152] During the underlying training process, an action masking mechanism is used to block illegal actions; the illegal actions are determined based on the current state constraints, so that illegal actions do not participate in the action selection.

[0153] In the AFSIM beyond-visual-range air combat simulation scenario, the specific execution actions corresponding to each leaf action node of the bottom execution layer, such as missile launch, electronic jamming activation, or radar mode switching, are strictly limited by dynamic state constraints such as the current moment's aircraft weapon reserves, radar equipment status, and electronic jamming consumables inventory.

[0154] For example, when the number of missiles carried has been reduced to zero, launching a missile becomes an illegal action. If the underlying reinforcement learning unit still includes this action in the candidate action space and tries to execute it, it will not only fail to obtain effective environmental feedback, but will also disrupt the data distribution of the experience replay pool due to the simulation platform refusing to execute or generating abnormal return values, leading to a bias in the fitting of the value function.

[0155] The action masking mechanism inserts a real-time mask vector during the action selection phase of the underlying reinforcement learning unit. Each element of the mask vector corresponds to a candidate action. A value of 1 indicates that the action is legal and allowed to participate in action selection, while a value of 0 indicates that the action is illegal. When calculating the action probability or action value function, the evaluation value of the corresponding illegal action is forced to be set to a very small penalty value or directly excluded at the policy layer, so that illegal actions do not participate in action selection.

[0156] Using the above method, the underlying reinforcement learning unit explores and utilizes only within the strictly legalized action subspace at each step of training, which greatly reduces the number of invalid action attempts. This ensures that the state transition samples stored in the experience replay pool are all physically meaningful and valid experiences, thereby accelerating the convergence of the underlying policy. At the same time, it ensures smooth interaction and safe operation with the AFSIM simulation platform during training.

[0157] Example 4

[0158] The similarities with the above embodiments and their combinations will not be repeated, the differences being:

[0159] See Figure 1 This embodiment discloses a reinforcement learning hierarchical training system based on behavior tree architecture, used to implement the reinforcement learning hierarchical training method based on behavior tree architecture as described in the foregoing embodiments, including:

[0160] The Behavior Tree Architecture Builder module is used to construct a behavior tree architecture adapted to AFSIM beyond-visual-range air combat simulation. It generates node configuration XML files that can be imported into AFSIM, clearly defining node types, hierarchical relationships, and execution logic, and supports drag-and-drop node configuration. This module directly addresses the node configuration specifications of the AFSIM beyond-visual-range air combat simulation platform, solidifying the node types, hierarchical relationships, and execution logic of the high-level decision-making layer, mid-level coordination layer, and low-level execution layer of the behavior tree architecture into a structured node configuration XML file that can be imported into AFSIM. It also provides drag-and-drop node configuration capabilities, transforming the behavior tree architecture design process from code writing to visual logical orchestration.

[0161] The behavior tree architecture building module decouples the topology of the behavior tree architecture from the loading interface of the simulation platform. This not only lowers the threshold for designers to build complex behavior tree architectures, but also ensures the consistency and reusability of the generated behavior tree architecture configuration file with the AFSIM simulation environment in terms of format and logic.

[0162] The hierarchical partitioning module, based on air combat tactical priorities, employs a node importance evaluation algorithm based on Monte Carlo sensitivity analysis to complete the hierarchical partitioning of the behavior tree architecture, outputting the composition and functional definitions of nodes in the high-level decision-making layer, the mid-level coordination layer, and the low-level execution layer. This module replaces the subjectivity of manual experience-based partitioning with data-driven, objective quantitative standards, ensuring that the partitioning results are strictly aligned with the actual contributions of each node in beyond-visual-range air combat. This provides a reproducible partitioning basis for subsequent bottom-up hierarchical training.

[0163] The parameter initialization module sets the learning rates for high-level, mid-level, and low-level reinforcement learning units. Discount Factor Exploration rate Timing window The system supports unified configuration and dynamic adjustment of training parameters, enabling differentiated parameters at each level (including the learning rate of the underlying reinforcement learning units using the DQN algorithm in deep reinforcement learning). Range of values to The learning rate of the intermediate reinforcement learning unit using the temporal difference learning algorithm. Range of values to The learning rate of high-level reinforcement learning units employing a semi-Markov decision process-Option framework Range of values to (etc.) can complete the initialization settings once before training starts and modify them in real time according to the convergence status during training. Through the above modules, the consistency and traceability of parameters at the start of stratified training can be effectively guaranteed, while providing a control interface for dynamic tuning of hyperparameters during training, making debugging and iteration efficient and controllable.

[0164] The state awareness module establishes a real-time data channel with the simulation platform through the AFSIM platform API interface, collects simulation data such as environmental state, aircraft state, and node execution state, and preprocesses the collected raw data to transform continuous dynamic parameters (such as relative distance, relative velocity, attitude angle, etc.) into structured inputs that can be directly used by high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units.

[0165] The state awareness module standardizes the data flow between simulation execution and policy learning, shielding the heterogeneity of data formats within the AFSIM simulation platform. This allows each reinforcement learning unit to obtain the required level of dimensionality-reduced state information through a unified proxy interface, ensuring the real-time nature and consistency of training data supply.

[0166] The reward calculation module implements a hierarchical reward mechanism based on air combat tactical priorities. It generates a penalty function consisting of attitude angle error and energy loss rate as a reward signal for the bottom-level reinforcement learning units, and generates a reward signal based on tactical timing windows for the middle-level reinforcement learning units. The coordinated reward signal, based on the missile launch time Does it meet the requirements? Apply positive gain respectively or punishment This generates sparse reward signals based on situational advantage for high-level reinforcement learning units, triggered only by the final kill of an enemy aircraft or the successful completion of an interception mission.

[0167] The reward calculation module precisely binds differentiated tactical incentives with optimization goals at each level, forming a progressively focused incentive system from bottom-level action precision and mid-level timing coordination to high-level strategic consistency, enabling reinforcement learning units at each level to converge in an orderly manner in their respective optimization directions.

[0168] The hierarchical training module, based on the activation / locking status indicators of nodes at each level in the behavior tree architecture, sequentially performs bottom-level switching, mid-level switching, and high-level switching in a bottom-up order. This achieves controlled training scheduling of reinforcement learning units and corresponding level nodes in the behavior tree architecture, and supports switching and adaptation for various reinforcement learning algorithms, such as DQN, temporal difference learning, and the semi-Markov decision process-Option framework. The hierarchical training module decouples the complex full-level joint training process into a serial sub-task training sequence. Training at each level is conducted under the condition that the lower level provides a stable environment, effectively eliminating the non-stationarity interference caused by simultaneous training at multiple levels. Furthermore, the flexibility of algorithm switching within the module allows it to adapt to air combat simulation scenarios of varying complexity.

[0169] The collaborative fine-tuning module, after completing training at all levels and locking all level nodes, unlocks the nodes of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer. It then links all level reinforcement learning units and behavior tree architecture nodes, adjusts the weights of the reward functions at each level through a weight search algorithm to find the optimal reward balance point, and / or uses an adversarial training mechanism to alternately pit the trained air combat agent against an enemy air combat agent using a random strategy. This eliminates potential inter-level policy conflicts that may be introduced during the hierarchical training process and verifies the policy generalization in dynamic scenarios until the task completion rate is not lower than the preset proportion of the baseline scenario.

[0170] The collaborative fine-tuning module optimizes the overall strategy while ensuring the stability of basic skills at each level. This enables seamless integration of high-level decision-making, mid-level coordination, and low-level execution in dynamic adversarial processes, effectively solving the problem of local optima rather than global collaboration that may result from tiered training.

[0171] The training termination judgment module continuously monitors the convergence of reinforcement learning policies corresponding to the high-level decision-making layer, the mid-level coordination layer, and the low-level execution layer, and statistically analyzes the stability of the global task completion rate in multiple consecutive simulations under the AFSIM simulation scenario. When it detects that the global task completion rate is not lower than the preset proportion for 10 consecutive simulations and the policies at each level converge, the training termination signal is automatically triggered. The training termination judgment module provides an objective and quantifiable criterion for training completion, avoiding premature termination leading to immature policies or premature termination leading to overfitting, thus ensuring the engineering reliability of the output policy.

[0172] The storage module stores the behavior tree architecture configuration, training parameters, training logs, and the final policy model. It persistently stores the behavior tree architecture configuration, training parameters at each level, training logs, and the final policy model, forming a complete training process traceability chain and reusable policy assets. This ensures the training results of the air combat intelligent agent are traceable, auditable, and portable, meeting the stringent requirements of military simulation for training process data integrity and equipment finalization verification.

[0173] Example 5

[0174] This embodiment takes a single-machine interception simulation task under the AFSIM platform as an example to provide a complete simulation task example, and details the specific implementation process, parameter configuration and training effect data of the method of the present invention, so as to meet the requirements of the pre-examination requirements for providing a calculation example and listing the calculation process.

[0175] a. Simulation scenario setup and key parameter settings

[0176] The simulation scenario configuration in this embodiment is as follows: one friendly fighter jet and one enemy fighter jet, with an initial distance of 80km and initial headings towards each other. The friendly aircraft has a speed of 450 knots and an altitude of 8000m, while the enemy aircraft has a speed of 400 knots and an altitude of 9000m. The altitude difference is 1000m. The friendly aircraft is equipped with medium-range air-to-air missiles, and the mission objective is to shoot down the enemy aircraft while maximizing its own survival rate.

[0177] The training parameters for each level of reinforcement learning unit are set as follows:

[0178] The underlying reinforcement learning unit (using the DQN algorithm in deep reinforcement learning): learning rate Discount factor Experience replay pool capacity 10000, batch size 64, exploration rate It decreases linearly from 1.0 to 0.05.

[0179] Mid-level reinforcement learning unit (using temporal difference learning algorithm): Learning rate Timing window .

[0180] High-level reinforcement learning unit (using a semi-Markov decision process - Option framework): Learning rate Option termination probability Determined through policy gradient learning.

[0181] All the above parameters were determined through preliminary experiments and optimization.

[0182] b. Architecture setup, assembly, and hierarchical division

[0183] The behavior tree architecture building module constructs a behavior tree architecture containing a root node, several control nodes (priority nodes, sequence nodes, and selection nodes), and leaf action nodes. The leaf action nodes correspond to specific actions such as fighter jet level flight, climb, radar start / stop / mode switching, missile launch / guidance, and electronic jamming activation.

[0184] By using a hierarchical partitioning module, based on air combat tactical priorities, and employing a node importance evaluation algorithm, the behavior tree is divided into three layers:

[0185] High-level decision-making layer: root node + top-level control node, responsible for overall combat decisions.

[0186] Intermediate Coordination Layer: Intermediate control nodes responsible for coordinating the timing of subtasks.

[0187] The bottom execution layer consists of leaf action nodes, which are responsible for executing specific actions.

[0188] The reinforcement learning module is divided into high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units, which are respectively combined with the high-level decision-making layer, mid-level coordination layer, and low-level execution layer of the behavior tree through proxy interfaces.

[0189] c. Layered training and node switching

[0190] Bottom-level switching: The execution logic of the nodes in the high-level decision-making layer and the middle-level coordination layer is fixed and locked. The bottom-level reinforcement learning unit is activated, and reinforcement learning training is performed on each leaf action node of the bottom-level execution layer. During training, an action masking mechanism is used to block illegal actions. The bottom-level training converges after about 1500 simulations, and the bottom-level action execution error is reduced to within 5%. At this point, the state of the bottom-level execution layer nodes is locked, and the training switches to the middle-level layer.

[0191] Mid-level switching: The execution logic of high-level decision-making nodes is fixed and locked. Mid-level reinforcement learning units are activated, linking with the locked low-level execution nodes to perform reinforcement learning training on the control nodes of the mid-level coordination layer. The training objective is to optimize the timing of radar lock-on and missile launch. The reward signal adopts a time-window-based collaborative reward mechanism: a tactical timing window is set. If the missile launch time satisfy If so, a positive reward will be given. Otherwise, a punishment will be imposed. After approximately 800 simulation rounds, the mid-level training converges. The timing deviation between radar lock-on and missile launch is less than 0.5 seconds for 10 consecutive simulation rounds. At this point, the mid-level coordination layer node status is locked, and the training switches to the high-level training.

[0192] High-level switching: Activate the high-level reinforcement learning unit, linking the locked mid-level coordination layer and low-level execution layer nodes to perform reinforcement learning training on the high-level decision-making layer nodes. The training objective is to optimize the global decision-making logic. The reward signal adopts a sparse reward based on situational advantage: it is only given upon successfully eliminating an enemy aircraft. When shot down, give The high-level training converged after approximately 500 simulations, and the global task completion rate fluctuated by less than 5% in 10 consecutive simulations, thus completing the high-level node switching adaptation.

[0193] d. Coordinated fine-tuning and training termination

[0194] After training across all levels is complete, the node locking states of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer are released for collaborative fine-tuning. First, the weights of the reward functions at each level are adjusted using a weight search algorithm (grid search combined with Bayesian optimization). The search within the interval yields the weighted combination that maximizes the overall task completion rate, ultimately determining the reward weight ratios for the bottom, middle, and top layers. Secondly, through an adversarial training mechanism, the trained air combat agents were alternately pitted against enemy air combat agents employing random strategies, undergoing a total of 200 rounds of adversarial training. Cooperative fine-tuning was completed when the global mission completion rate under dynamic scenarios such as enemy aircraft type switching and electronic jamming intensity adjustments was not lower than 85% of the baseline scenario.

[0195] Training is terminated when the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer converge, and the global task completion rate in the AFSIM simulation scenario is not less than 85% for 10 consecutive times. The final behavior tree architecture and reinforcement learning policy are then output.

[0196] e. Training effect comparison data

[0197] Through the above-described tiered training, the overall decision-making quality of the agent meets the engineering application requirements of beyond-visual-range air combat simulation. Specific training performance data is as follows:

[0198] Training convergence speed: The total number of convergence rounds for the hierarchical training strategy is approximately 2800 (1500+800+500), which is about 57% lower than the approximately 6500 rounds for the single-level training method.

[0199] Strategy generalization: When facing enemy aircraft switching scenarios, the strategy trained by this invention does not need to be retrained, and the global task completion rate can be maintained at over 78%.

[0200] Motion accuracy: After the training of the underlying reinforcement learning units reached the target, the average attitude angle error of tactical maneuvers decreased from the initial 12.3° to 3.8°, which is less than 5%.

[0201] Timing coordination accuracy: After the mid-level reinforcement learning unit training reaches the target, the average timing deviation from radar lock to missile launch is reduced from the initial 1.2 seconds to 0.35 seconds, which is less than 0.5 seconds.

[0202] The above data shows that the hierarchical training method proposed in this invention can effectively improve training efficiency and policy generalization.

[0203] The experimental data disclosed in this embodiment are experimental data of parameter values ​​near the optimized parameters. Experimental data of parameter values ​​that are too much or too little beyond the optimized parameters have relatively low reference value and will not be elaborated here.

[0204] It should be noted that, in actual implementation, the structure depicted in the accompanying drawings is not a fixed or unchanging embodiment. The components of the embodiments of the invention described and shown in these drawings can typically be arranged and designed in various different configurations. Furthermore, the accompanying drawings and abstract drawings are merely illustrative and do not represent the specific structure or actual quantity in a concrete implementation.

[0205] Unless otherwise defined, the technical or scientific terms used herein should be understood in their ordinary sense as understood by one of ordinary skill in the art to which this invention pertains. The use of terms such as "a" or "an" in this specification and claims does not necessarily indicate a limitation of quantity. Terms such as "comprising" or "including" mean that the element or component preceding the word encompasses the element or component listed following the word and its equivalents, without excluding other elements or components. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0206] The exemplary embodiments of the present invention have been described in detail above with reference to preferred embodiments. However, those skilled in the art will understand that various modifications and alterations can be made to the above specific embodiments without departing from the concept of the present invention, and various combinations can be made to the various technical features and structures proposed in the present invention without exceeding the protection scope of the present invention.

Claims

1. A reinforcement learning hierarchical training method based on a behavior tree architecture, characterized in that, Includes the following steps: Step S1: Construct a behavior tree architecture for decision-making and scheduling of air combat intelligent agents. The behavior tree architecture includes a high-level decision-making layer, a middle-level coordination layer, and a low-level execution layer, which are divided according to functional levels. Step S2: Construct a reinforcement learning module, which is used to optimize policies through interaction with the environment, and divide the reinforcement learning module into high-level reinforcement learning units, mid-level reinforcement learning units and low-level reinforcement learning units according to the three functional levels of the behavior tree architecture. Step S3: Implement the corresponding combination of the high-level decision-making layer, middle-level coordination layer, and low-level execution layer of the behavior tree architecture with the high-level reinforcement learning unit, middle-level reinforcement learning unit, and low-level reinforcement learning unit of the reinforcement learning module through the proxy interface; The configuration includes: when the behavior tree architecture reaches the control class node, the control class node obtains the global state matrix through the proxy interface. After dimensionality reduction and extraction, the vector is used as the input to the corresponding level of the reinforcement learning unit; the reinforcement learning unit outputs the action instruction vector. The behavior tree architecture receives action instruction vectors through condition monitoring slots. And according to the action command vector Enables dynamic adjustment of the execution order or triggering conditions of child nodes in the behavior tree architecture; Step S4: The nodes of the high-level decision layer, the middle-level coordination layer and the low-level execution layer of the behavior tree architecture are equipped with activation / locking status indicators, and based on the activation / locking status indicators, hierarchical training is performed in a bottom-up order. Hierarchical training includes low-level switching, middle-level switching and high-level switching. Step S5: Collaborative fine-tuning, unlocking the nodes of the high-level decision-making layer, the middle-level coordination layer, and the bottom-level execution layer, linking all levels of reinforcement learning units and behavior tree architecture nodes, eliminating policy conflicts between levels, and verifying policy generalization. The collaborative fine-tuning in step S5 includes: adjusting the weights of the reward functions at each level through a weight search algorithm, and / or eliminating policy conflicts between levels through an adversarial training mechanism; The adversarial training mechanism involves alternating between trained air combat agents and enemy air combat agents employing random strategies until the task completion rate of the strategy in dynamic scenarios is not lower than the preset proportion in the baseline scenario.

2. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, In step S1, the air combat agent is an agent that makes tactical decisions and executes actions in a beyond-visual-range air combat simulation environment. In step S4, activation means that the reinforcement learning network corresponding to a certain level node in the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer is in training mode, and the model parameters participate in training and updating. The air combat agent adopts the ε-greedy exploration strategy to explore and utilize the environment and update the network weights. Locking refers to the reinforcement learning network corresponding to a node in one of the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer reaching the convergence criterion. It switches to inference mode and freezes all weight parameters of the network, stops gradient updates, and provides a stable environment with extremely low non-stationarity for the training of other layers. The inference mode is a mode that freezes the network weight parameters and only performs forward computation. In locked state, the air combat agent ceases exploration and only proceeds based on the current highest [value / status]. The output value represents a deterministic action.

3. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, The bottom-level switching in step S4 is as follows: fix and lock the node execution logic of the high-level decision-making layer and the middle-level coordination layer, activate the bottom-level reinforcement learning unit, perform reinforcement learning training on each leaf action node of the bottom-level execution layer until the training is successful when the bottom-level action execution error is less than 5%, and lock the node state of the bottom-level execution layer. The middle layer switching in step S4 is as follows: fix and lock the execution logic of the high-level decision layer node, activate the middle layer reinforcement learning unit, link the locked low-level execution layer node, perform reinforcement learning training on the control type node of the middle layer coordination layer, the reinforcement learning training aims to optimize the timing coordination of subtasks, and the training is successful when the timing coordination deviation is lower than the preset threshold for 10 consecutive simulation rounds, and the state of the middle layer coordination layer node is locked. The preset threshold is the time deviation compliance value, with a default value of 0.5 seconds; The high-level switching in step S4 involves: activating the high-level reinforcement learning unit, linking the locked mid-level coordination layer and low-level execution layer nodes, and performing reinforcement learning training on the high-level decision-making layer nodes. This process continues until the global task completion rate is within a certain range. Training is successful when the fluctuation amplitude is less than 5% in the simulation. The global task completion rate is the weighted average of the kill rate and the survival rate.

4. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, The preset ratio is the benchmark value for judging whether the generalization of the strategy meets the standard, and is set to 85%. Training is terminated when the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer, and the low-level execution layer are detected to converge, and the global task completion rate is not lower than the preset proportion for 10 consecutive times.

5. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, In the behavior tree architecture, the high-level decision-making layer includes the root node and the top-level control node, which is responsible for global combat decisions and core optimization of combat mode selection and target priority ranking. The middle coordination layer includes intermediate control nodes, which are responsible for the timing coordination of subtasks and optimize the timing connection and collaboration logic between subtasks. The underlying execution layer includes leaf action nodes, which are responsible for the execution of specific actions, and the core optimizations include tactical maneuver accuracy, missile launch timing, and radar lock-on accuracy. Furthermore, the division of the behavior tree architecture into high-level decision-making layer, mid-level coordination layer, and low-level execution layer adopts a node importance evaluation algorithm based on Monte Carlo sensitivity analysis. Specifically, it calculates the node importance by obtaining the change in global win rate after the node's action output is disturbed. Importance for: , in, The global win rate is the change in global win rate after perturbing the action output of this node. The global win rate is the statistical probability that the air combat agent wins in simulated combat. The applied disturbance amplitude is normalized to the standard deviation of the node's action output. Normalized sensitivity of node actions to global win rate; based on importance. The nodes are divided into a high-level decision-making layer, a middle-level coordination layer, and a low-level execution layer: The nodes were assigned to the higher-level decision-making level. The nodes are assigned to the middle coordination layer. The nodes are assigned to the underlying execution layer.

6. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, The high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units of the reinforcement learning module are combined using differentiated algorithms: The high-level reinforcement learning unit is paired with the high-level decision-making layer and uses a semi-Markov decision process-Option framework for high-level decision training. By utilizing its temporal abstraction capability, it allows high-level nodes to maintain macro actions for a period of time until the termination condition is met. The intermediate reinforcement learning unit is paired with the intermediate coordination layer and uses a temporal difference learning algorithm to be responsible for the reinforcement learning of the temporal coordination of subtasks, so as to handle the sequential decision problem with strict temporal dependencies based on the step-by-step update feature. The underlying reinforcement learning unit is paired with the underlying execution layer and uses the DQN algorithm in deep reinforcement learning to be responsible for reinforcement learning of specific action execution. It fits the Q-value function through a deep neural network, processes continuous dynamic parameters in the simulation, and outputs accurate discrete instructions.

7. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 6, characterized in that, The parameters used by the high-level reinforcement learning unit, mid-level reinforcement learning unit, and low-level reinforcement learning unit to control the reinforcement learning training process include learning rate, discount factor, exploration rate, and time window, which are set as follows: The learning rate of the underlying reinforcement learning units using the DQN algorithm The value ranges from 0.0001 to 0.01, discount factor. The value ranges from 0.95 to 0.999, and the exploration rate is... The value decays from 1.0 to a preset value between 0.01 and 0.1, and supports dynamic adjustment during training; Learning rate of mid-layer reinforcement learning units using the temporal difference learning algorithm The value ranges from 0.001 to 0.01, and it is used to evaluate the timing coordination of subtasks. The timeout is set to 0.3 to 1.0 seconds based on the requirements of air combat tactics. Learning rate of high-level reinforcement learning units using a semi-Markov decision process-Option framework The value ranges from 0.0001 to 0.

001.

8. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, It also includes a tiered reward mechanism, which provides differentiated reward signals to high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units based on air combat tactical priorities. The reward signal set for the underlying reinforcement learning unit adopts a penalty function composed of attitude angle error and energy loss rate to guide the air combat agent to fit the optimal envelope; The reward signal set for the mid-level reinforcement learning unit adopts a time-window-based collaborative reward mechanism: setting a tactical time window. ,in The moment when the radar stably locks onto the target, To preset the optimal launch window duration; If the missile launch time satisfy: Then a positive gain is given. ,in The reward signal value received by the middle-layer reinforcement learning unit; otherwise, a penalty is imposed. This is to guide the middle-level units to optimize the timing of radar locking and missile launch; The reward signal set for the high-level reinforcement learning unit adopts a sparse reward based on the situational advantage degree, which is triggered only when the enemy aircraft is finally killed or the interception mission is successfully completed.

9. The reinforcement learning hierarchical training method based on behavior tree architecture according to claim 1, characterized in that, During the underlying training process, an action masking mechanism is used to block illegal actions; the illegal actions are determined based on the current state constraints, so that illegal actions do not participate in the action selection.

10. A hierarchical training system for reinforcement learning based on a behavior tree architecture, characterized in that, A method for implementing a hierarchical training method for reinforcement learning based on a behavior tree architecture as described in any one of claims 1-9 includes: The behavior tree architecture building module is used to build a behavior tree architecture adapted to AFSIM beyond visual range air combat simulation, generate node configuration XML files that can be imported into AFSIM, specify the node type, hierarchical relationship and execution logic, and support drag-and-drop configuration of nodes; The hierarchical partitioning module is used to complete the hierarchical partitioning of the behavior tree architecture based on the priority of air combat tactics and the node importance evaluation algorithm, and outputs the node composition and functional definition of the high-level decision-making layer, the middle-level coordination layer and the low-level execution layer. The parameter initialization module is used to implement the unified configuration and dynamic adjustment of training parameters for high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units. The state awareness module is used to connect to the simulation platform through the AFSIM platform API interface, collect simulation data of environmental state, fighter jet state, and node execution state in real time, and provide data support for the reinforcement learning module after data preprocessing. The reward calculation module is used to generate differentiated reward signals for high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units based on the air combat tactical priority using a hierarchical reward mechanism. The hierarchical training module is used to control the switching between the reinforcement learning units and the corresponding hierarchical nodes of the behavior tree architecture, execute the hierarchical training strategy, and support the switching and adaptation of various reinforcement learning algorithms. The collaborative fine-tuning module is used to link high-level reinforcement learning units, mid-level reinforcement learning units, and low-level reinforcement learning units with behavior tree architecture nodes after all levels of training are completed. It eliminates policy conflicts between levels by adjusting the weights of reward functions at each level and adversarial training mechanisms, and verifies policy generalization. The training termination judgment module is used to detect the convergence of the reinforcement learning policies corresponding to the high-level decision-making layer, the middle-level coordination layer and the low-level execution layer, and whether the global task completion rate in the AFSIM simulation scenario is not lower than the preset proportion for 10 consecutive times. The storage module is used to store the behavior tree architecture configuration, training parameters, training logs, and the final policy model.

Images