Intelligent agent action sequence generation method based on hierarchical decision-making architecture
By integrating decision trees, state machines, and behavior trees through a hierarchical decision architecture, the problem of insufficient flexibility in modeling agent behavior in complex environments is solved, efficient generation of behavior sequences is achieved, and the decision accuracy and adaptability of agents in complex scenarios are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-04-07
AI Technical Summary
Existing single-agent behavior modeling methods struggle to balance coarse-grained policy adjustment and fine-grained action control in complex environments, resulting in insufficient agent performance and an inability to meet the needs of behavior sequence generation in complex scenarios.
A hierarchical decision-making architecture is adopted, combining decision trees, state machines, and behavior trees to integrate environmental situation information. The decision tree is used to make overall situation judgments, the state machine is used to make state transitions, and the behavior tree generates specific action sequences, forming an efficient decision-making system from global situation to specific actions.
It improves the scene adaptability and behavioral decision-making accuracy of intelligent agents in complex and highly adversarial dynamic scenarios, and realizes efficient generation from global situation to specific actions.
Smart Images
Figure CN121809727A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of artificial intelligence, and relates to an agent action sequence generation method based on a hierarchical decision architecture. BACKGROUND
[0002] In the generation of an agent action sequence in a complex environment, an agent constructed by a single method often cannot balance the difficulty of coarse-grained strategy adjustment and fine-grained action control, thereby limiting the efficiency of the agent. Therefore, it is an urgent problem to propose an action sequence generation method that can integrate the advantages of multiple technologies.
[0003] Traditional agent action sequence generation includes rule decision tree modeling, behavior tree modeling, state machine modeling, etc. A decision tree is a machine learning method that simulates the human decision-making process, and generates a tree structure through a series of questions to guide decision-making. Although the decision tree is easy to understand and implement, it is difficult to handle the dynamic switching and long-term dependence between behaviors, the tree structure is prone to expansion in a complex scenario, is difficult to maintain, lacks behavior flexibility, and is difficult to adapt to a complex and variable environment.
[0004] A behavior tree is a graphical model used to describe the behavior of an agent, and is a hierarchical tree structure used to plan and execute complex behavior tasks. It has been widely used in game AI and autonomous driving. The behavior tree realizes flexible behavior decision and execution by combining different behavior nodes and control nodes. When constructing a device agent, the behavior tree can generate an ordered action sequence according to the current state, target and task of the agent. The behavior tree supports hierarchical modeling and flexible behavior organization, but often cannot handle the global situation, and the design requirements for behavior order and priority are high, which can easily cause unexpected behavior.
[0005] A state machine is a mathematical model used to describe the transition between different states of a system, and can be used to manage the behavior logic and state transition of an agent. A state machine includes four basic elements: state, event, transition and action. The development and implementation of a state machine are relatively simple, and it is suitable for systems with clear state transition logic. However, the state machine method is difficult to deal with scenarios with a large number of states and complex transitions, and has poor scalability
[0006] In summary, the limitations of existing single modeling methods make the action sequence generation of an agent in a complex battlefield environment not flexible and intelligent, and cannot meet the efficient generation requirements of the behavior sequence in an actual complex scenario. Therefore, it is necessary to design an action sequence generation technology that integrates multiple behavior decision modeling methods. SUMMARY
[0007] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose an intelligent agent action sequence generation method based on a hierarchical decision architecture, which solves the problem of single intelligent agent behavior modeling method from understanding the global situation to generating specific actions in complex adversarial environments.
[0008] The solution to the technical problem of this invention is: a method for generating action sequences of intelligent agents based on a hierarchical decision-making architecture, comprising the following steps:
[0009] A behavior sequence generation model with a three-layer structure including a decision tree layer, a state machine layer, and a behavior tree layer is established. The current environmental situation information is integrated, including detected enemy information, target information, and environmental data collected by sensors in real time. The collected current environmental situation information is formatted and used as input data for the behavior sequence generation model.
[0010] At the decision tree level, judgment conditions are set. The decision tree compares and judges the current environmental situation information input in real time with the judgment conditions to complete the overall situation judgment. Different strategy adjustment instructions are generated according to different overall situation judgment results and input into the corresponding state machine.
[0011] At the state machine layer, based on the agent's tasks and behaviors, a core state set S = {s1, s2, ..., s...} is defined. m The state machine receives policy adjustment instructions from the decision tree and performs state transitions according to pre-defined state transition rules. The output of the state machine layer is used to select the behavior tree model under different states.
[0012] Based on the state transition of the state machine, the corresponding behavior tree model in the behavior tree layer is activated. The activated behavior tree model executes according to its own structure and rules to generate a specific action sequence of the agent.
[0013] Furthermore, the detected enemy information includes communication spectrum, jamming patterns, enemy aircraft position, speed, and heading.
[0014] Furthermore, the target information includes the target location, type, and motion state.
[0015] Furthermore, the sensor collects environmental data in real time, including the frequency, power, and waveform of active radar, communication, and navigation signals.
[0016] Furthermore, the decision tree layer specifically comprises:
[0017] Decision trees guide decision-making through a series of feature selections, generating a tree structure for classification or regression tasks, and determining the next action strategy based on the device's status and environmental information.
[0018] The root node of the decision tree is the input environmental situation information, which is used to encode and quantify the environmental situation information; internal nodes represent one or more environmental coding features; branches represent the output results of the judgment conditions; and leaf nodes represent the final decision results.
[0019] Furthermore, the state machine layer specifically comprises:
[0020] A state machine structure includes states, events, transitions, and actions. A state has specific behaviors and conditions. When a condition is met, the state machine triggers a transition. An event is an external or internal signal that triggers a state transition. A transition is the shift from one state to another. An action is an operation performed when transitioning to or leaving a state.
[0021] Furthermore, the behavior tree layer specifically comprises:
[0022] The behavior tree model consists of two main categories: root nodes and behavior nodes. The root node is the starting node for decision-making, while behavior nodes are leaf nodes, including logical nodes, action nodes, and condition nodes.
[0023] Logical nodes: control and organize the execution order or logic of other nodes, and generate their own execution results based on the execution results of their child nodes, ultimately affecting the output of the behavior tree; they include sequence nodes, selection nodes, and decorator nodes;
[0024] Action node: A node that executes a specific action or task;
[0025] Condition node: Returns a success or failure result based on specific environmental and state factors;
[0026] The behavior tree model provides a data sharing and communication mechanism between nodes.
[0027] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for generating an intelligent agent action sequence based on a hierarchical decision-making architecture.
[0028] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for generating an intelligent agent action sequence based on a hierarchical decision architecture.
[0029] A computer program product includes a computer program that, when executed by a processor, implements the steps of the method for generating intelligent agent action sequences based on a hierarchical decision architecture.
[0030] The advantages of this invention compared to the prior art are:
[0031] This invention addresses the challenges of single-agent behavior modeling in complex adversarial environments, from understanding the global situation to generating specific actions. By integrating behavior trees, state machines, and decision trees to generate behavior sequences, it improves the scenario adaptability and accuracy of behavioral decisions for agents in complex, highly adversarial, and dynamic scenarios. Attached Figure Description
[0032] Figure 1 This is a flowchart of an intelligent agent action sequence generation method based on a hierarchical decision-making architecture according to the present invention.
[0033] Figure 2 This is a schematic diagram illustrating the working principle of the decision tree layer in an embodiment of the present invention.
[0034] Figure 3 This is an example of a radar vehicle decision tree in an embodiment of the present invention;
[0035] Figure 4 This is an example of state switching within each level of the radar anti-jamming state machine in an embodiment of the present invention;
[0036] Figure 5 This is an example of a state machine model in the state machine layer of this invention;
[0037] Figure 6 This is a schematic diagram of a behavior tree model according to an embodiment of the present invention. Detailed Implementation
[0038] In complex and ever-changing environments and with diverse task requirements, intelligent agent development faces a vast strategy space and extremely long action sequences. Single intelligent agent behavior modeling methods struggle to simultaneously grasp the overall situational awareness and achieve precise action control. This invention addresses these issues by proposing an intelligent agent action sequence generation method based on a hierarchical decision-making architecture. By integrating hierarchical state machines, decision trees, and behavior trees, it achieves a rapid and efficient decision-making system from understanding the global situation to generating specific actions. This, in turn, generates accurate intelligent agent behavior sequences, improving the scenario adaptability and accuracy of behavioral decisions in complex, highly adversarial, and dynamic scenarios.
[0039] like Figure 1 As shown, the method of the present invention includes the following steps:
[0040] A behavior sequence generation model with a three-layer structure including a decision tree layer, a state machine layer, and a behavior tree layer is established. The current environmental situation information is integrated, including detected enemy information, target information, and environmental data collected by sensors in real time. The collected current environmental situation information is formatted and used as input data for the behavior sequence generation model.
[0041] At the decision tree level, judgment conditions are set. The decision tree compares and judges the current environmental situation information input in real time with the judgment conditions to complete the overall situation judgment. Different strategy adjustment instructions are generated according to different overall situation judgment results and input into the corresponding state machine.
[0042] At the state machine layer, based on the agent's tasks and behaviors, a core state set S = {s1, s2, ..., s...} is defined. m The state machine receives policy adjustment instructions from the decision tree and performs state transitions according to pre-defined state transition rules. The output of the state machine layer is used to select the behavior tree model under different states.
[0043] Based on the state transition of the state machine, the corresponding behavior tree model in the behavior tree layer is activated. The activated behavior tree model executes according to its own structure and rules to generate a specific action sequence of the agent.
[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0045] Example 1
[0046] like Figure 1 The method for generating agent action sequences based on a hierarchical decision-making architecture in this embodiment includes the following steps:
[0047] Step 1: Establish a behavior sequence generation model with a three-layer structure including a decision tree layer, a state machine layer, and a behavior tree layer. Integrate current environmental situation information, including detected enemy information, target information, and environmental data collected by sensors in real time. Organize the collected current environmental situation information into a format and use it as input data for the behavior sequence generation model.
[0048] The target information includes the target location, type, and motion state;
[0049] The detected enemy information includes communication spectrum, jamming patterns, enemy aircraft position, speed, and heading.
[0050] The sensors collect environmental data in real time, including the frequency, power, and waveform of active radar, communication, and navigation signals.
[0051] Step 2: At the decision tree level, set the judgment conditions. The decision tree compares and judges the current environmental situation information input in real time with the judgment conditions to complete the overall situation judgment. Based on different overall situation judgment results, different strategy adjustment instructions are generated and input into the corresponding state machine.
[0052] In this invention, the decision tree layer specifically comprises:
[0053] Decision trees guide decision-making through a series of feature selections, generating a tree-like structure for classification or regression tasks. Based on factors such as device status and environmental information, they determine the next course of action.
[0054] In this invention, the root node of the decision tree is the input environmental situation information, which is used to encode and quantify the environmental situation information; the internal nodes represent one or more environmental coding features (e.g., the simultaneous presence of multiple interference signals such as deception interference, pulse interference, frequency sweep interference, and agile interference); the branches represent the output results of the judgment conditions (e.g., whether the conditions for executing the cooperative anti-suppression strategy are met); and the leaf nodes represent the final decision result (e.g., executing the cooperative anti-suppression strategy).
[0055] The decision tree compares and judges the real-time input of the current environmental situation information with the judgment conditions to complete the overall situation judgment. The specific method is as follows:
[0056] The current environmental situation information is compared with the aforementioned judgment conditions. For example, if multiple interference signals such as deception jamming, pulse jamming, frequency sweeping jamming, and frequency agility jamming are detected simultaneously, a coordinated anti-suppression strategy is first executed; if enemy communication signals are detected and our direction-finding equipment has not yet been activated, and both aiming jamming and frequency sweeping jamming signals are detected simultaneously, a multi-aircraft coordinated detection strategy is executed. Figure 2 , Figure 3 As shown.
[0057] Step 3: At the state machine layer, based on the agent's task and behavior, define the core state set S = {s1, s2, ..., s3}. The state machine receives the policy adjustment instructions output by the decision tree and performs state transitions according to the pre-set state transition rules. The output of the state machine layer is used to select the behavior tree model under different states.
[0058] In this invention, the state machine layer specifically comprises:
[0059] A state machine's main structure includes states, events, transitions, and actions. Each state has specific behaviors and conditions; when these conditions are met, the state machine can trigger a transition. An event is an external or internal signal that triggers a state transition; for example, radar detecting the electronic signature of enemy equipment. A transition is the shift from one state to another. An action is an operation performed during a transition or when entering / leaving a state; for example, initiating anti-jamming measures when entering a threatened and jammed state.
[0060] This embodiment takes a radar anti-jamming state machine as an example, which is divided into three levels: under jamming threat, no jamming threat, and communication failure. State switching is performed within each level, such as... Figure 4 As shown.
[0061] The core state set S = {s1, s2, ..., s}m For example, in an electromagnetic scenario, an agent's "reconnaissance state s1", "interference state s2", "avoidance state s3", etc.
[0062] The pre-defined state transition rule is in the form of: Let the state transition function be f. s (s i ,I D ), where s i As the current state, I D The policy adjustment instructions output by the decision tree.
[0063] The difference between different state machine models in the state machine layer lies in the fact that the states, state transition conditions, and actions to be performed for each type of agent are different. For example, the states of a radar-jamming agent include being jammed, counter-jamming, signal search, and tracking, while the states of a communication agent include covert communication, communication under jamming, normal communication, and communication failure. Figure 5 As shown.
[0064] Step 4: Based on the state after the state machine transition, activate the corresponding behavior tree model in the behavior tree layer. The activated behavior tree model executes according to its own structure and rules to generate a specific action sequence of the intelligent agent.
[0065] In this invention, the behavior tree layer specifically comprises:
[0066] like Figure 6 As shown, the behavior tree model includes two main categories: root nodes and behavior nodes. The root node is the decision-making starting node, and the behavior nodes are leaf nodes, including logical nodes, action nodes, and condition nodes.
[0067] Logical nodes: control and organize the execution order or logic of other nodes, and generate their own execution results based on the execution results of their child nodes, ultimately affecting the output of the behavior tree; they include sequence nodes, selection nodes, and decorator nodes;
[0068] Action node: A node that executes a specific action or task;
[0069] Condition node: Returns a success or failure result based on specific environmental and state factors;
[0070] The behavior tree model provides a data sharing and communication mechanism between nodes.
[0071] The generation of specific agent action sequences: For example, in the "interference state", the behavior tree performs actions such as setting interference device parameters and transmitting interference signals, forming an action sequence of "setting interference frequency → adjusting interference power → transmitting interference signals".
[0072] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform... Figure 1 The method described.
[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes The steps of the function specified in one or more boxes.
[0077] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
[0078] The contents not described in detail in this specification are common knowledge to those skilled in the art.
Claims
1. A method for generating agent action sequences based on a hierarchical decision-making architecture, characterized in that, Includes the following steps: A behavior sequence generation model with a three-layer structure including a decision tree layer, a state machine layer, and a behavior tree layer is established. The current environmental situation information is integrated, including detected enemy information, target information, and environmental data collected by sensors in real time. The collected current environmental situation information is formatted and used as input data for the behavior sequence generation model. At the decision tree level, judgment conditions are set. The decision tree compares and judges the current environmental situation information input in real time with the judgment conditions to complete the overall situation judgment. Different strategy adjustment instructions are generated according to different overall situation judgment results and input into the corresponding state machine. At the state machine layer, based on the agent's tasks and behaviors, a core state set S = {s1, s2, ..., s...} is defined. m The state machine receives policy adjustment instructions from the decision tree and performs state transitions according to pre-defined state transition rules. The output of the state machine layer is used to select the behavior tree model under different states. Based on the state transition of the state machine, the corresponding behavior tree model in the behavior tree layer is activated. The activated behavior tree model executes according to its own structure and rules to generate a specific action sequence of the agent.
2. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The detected enemy information includes communication spectrum, jamming patterns, enemy aircraft position, speed, and heading.
3. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The target information includes the target's location, type, and motion state.
4. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The sensors collect environmental data in real time, including the frequency, power, and waveform of active radar, communication, and navigation signals.
5. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The decision tree layer is specifically as follows: Decision trees guide decision-making through a series of feature selections, generating a tree structure for classification or regression tasks, and determining the next action strategy based on the device's status and environmental information. The root node of the decision tree is the input environmental situation information, which is used to encode and quantify the environmental situation information; internal nodes represent one or more environmental coding features; branches represent the output results of the judgment conditions; and leaf nodes represent the final decision results.
6. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The state machine layer is specifically: A state machine structure includes states, events, transitions, and actions. A state has specific behaviors and conditions. When a condition is met, the state machine triggers a transition. An event is an external or internal signal that triggers a state transition. A transition is the shift from one state to another. An action is an operation performed when transitioning to, entering, or leaving a state.
7. The method for generating agent action sequences based on a hierarchical decision-making architecture according to claim 1, characterized in that, The behavior tree layer is specifically as follows: The behavior tree model consists of two main categories: root nodes and behavior nodes. The root node is the starting node for decision-making, while behavior nodes are leaf nodes, including logical nodes, action nodes, and condition nodes. Logical nodes: control and organize the execution order or logic of other nodes, and generate their own execution results based on the execution results of their child nodes, ultimately affecting the output of the behavior tree; they include sequence nodes, selection nodes, and decorator nodes; Action node: A node that executes a specific action or task; Condition node: Returns a success or failure result based on specific environmental and state factors; The behavior tree model provides a data sharing and communication mechanism between nodes.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.