Behavior tree importance evaluation and optimization method and system for unmanned aerial vehicle maintenance
By integrating behavior tree and reinforcement learning modules in the drone maintenance agent, combined with Shapley value evaluation and decision tree model training, the problem of high design cost and difficulty in modification of the drone maintenance behavior tree is solved, and the optimization of the behavior tree logic and automatic adjustment of the execution order are realized.
Patent Information
- Application Number
- CN202411394510.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-08
- Publication Date
- 2025-04-04
AI Technical Summary
When designing a drone maintenance behavior tree, it is costly and difficult to modify, and the behavior tree cannot independently adjust the subtask weight, resulting in the inability to effectively optimize the maintenance process.
By integrating the behavior tree and reinforcement learning module in the agent, the agent's communication mechanism is used to send signals to the reinforcement learning module before the behavior tree triggers the action node, collect the behavior tree running data and train the decision tree model. The Shapley value is used to evaluate the importance of the decision tree feature, calculate the weight of the behavior tree nodes, and adjust the execution order of the nodes to optimize the maintenance logic.
It reduces the design cost of the behavior tree under UAV maintenance, improves design efficiency and modification flexibility, ensures the logical optimization of the behavior tree during execution, and avoids losses caused by wrong operations.
Smart Images
Figure CN119762035A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and task planning, and in particular relates to a behavior tree importance evaluation and optimization method for unmanned aerial vehicle maintenance. Background Art
[0002] A behavior tree is a graphical representation method for planning and controlling the behavior of an agent. It uses a tree structure to show the decision-making process of an agent and decomposes complex tasks into a series of simple behavior nodes.
[0003] Although behavior trees have significant advantages in agent decision-making, designing a reasonable and excellent drone maintenance behavior tree is a very challenging task. This process requires coordination among various drone design manufacturers, and also requires comprehensive consideration of factors such as mission requirements, environmental constraints, and system performance, which results in extremely high design costs.
[0004] In addition, in the drone maintenance mission, the maintenance process of each component has its strict requirements. For example, if the power supply of the body is insufficient, the power module is not allowed to be powered off privately to save electricity; if there is a short circuit inside some components, powering it on privately may cause a system short circuit. Therefore, the behavior tree cannot be completely allowed to operate autonomously to prevent the drone from falling or expensive components from being burned due to incorrect operations, avoiding major losses.
[0005] Therefore, in the design and optimization process of behavior trees, the method of introducing decision trees to analyze historical maintenance data can optimize the execution logic of behavior trees to the greatest extent while being fully interpretable, thus overcoming the limitations of traditional manual design methods. Summary of the invention
[0006] The purpose of the present invention is to provide a behavior tree importance evaluation and optimization method and system for drone maintenance, which solves the problems of excessive cost and high modification difficulty caused by designing behavior trees when performing intelligent maintenance of drones.
[0007] In order to achieve the above object, the present invention adopts the following technical solution: a behavior tree importance evaluation and optimization method and system for drone maintenance, comprising the following steps:
[0008] S1. Build a behavior tree for performing drone maintenance tasks and integrate it into the agent. At the same time, configure a reinforcement learning module in the agent.
[0009] S2. Using the communication mechanism of the intelligent agent, the behavior tree can send a signal to notify the reinforcement learning module before triggering the action node, and the reinforcement learning module selects actions for the behavior tree to collect the operation data of the behavior tree;
[0010] S3, preprocessing the operation data, and using it as a training set to train a decision tree algorithm, generating a decision model for predicting and analyzing maintenance results;
[0011] S4. Use Shapley value to evaluate the importance of each feature in the decision tree, thereby evaluating the importance of the conditional nodes in the behavior tree;
[0012] S5. Set weight calculation modules in the parent nodes of all conditional nodes in turn, and use these modules to gradually transfer weight values from the conditional nodes to each node of the entire behavior tree;
[0013] S6. The behavior tree determines the execution order of subtasks based on the weight of each node.
[0014] 2. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, characterized in that S1 comprises:
[0015] S1.1. By integrating the behavior tree and reinforcement learning modules within the agent, and utilizing the communication mechanism of the agent, a signal can be sent to the reinforcement learning module before the behavior tree triggers an action node, thereby achieving communication between the two modules.
[0016] S1.2, Reinforcement learning algorithms can use reinforcement learning methods such as Q-Learning and Sarsa;
[0017] 3. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S2 comprises:
[0018] S2.1, the communication mechanism of the intelligent agent can use TCP / IP communication, meta-object system and other communication methods;
[0019] S2.2. After using the communication mechanism of the agent, a signal transmitter can be installed in the control node. When the control node is triggered, it will send the current node information to the reinforcement learning module in the form of a signal according to the state of the agent. The reinforcement learning module will make subsequent action selections and hand them over to the behavior tree for execution;
[0020] S2.3. The data taken in the behavior tree is the data feature in the conditional node. When the conditional node in the behavior tree is triggered, the current data will be recorded.
[0021] 4. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S3 comprises:
[0022] S3.1. Collect and preprocess the data during the operation of the behavior tree, including input conditions, execution process and output results, and then use these data to train the decision tree algorithm to build a decision model that can predict the success or failure of the maintenance task;
[0023] S3.2, data preprocessing includes a series of preprocessing tasks such as missing value processing, one-hot encoding, and removing invalid features of running data;
[0024] S3.3. The data features in the decision tree are mapped to the conditional nodes in the behavior tree, that is, the data judgment logic in the behavior tree can find the corresponding logic in the decision tree.
[0025] 5. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S4 comprises:
[0026] S4.1. After normalizing the values finally calculated by using Shapley values in the decision tree, the weights of each feature in the decision tree are obtained. Since there is a mapping relationship between the features in the decision tree and the conditional nodes in the behavior tree, the weights of the conditional nodes in the behavior tree can be further determined by these feature weights.
[0027] 6. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S5 comprises:
[0028] S5.1. The weight calculation module will calculate its own weight before being triggered. The specific calculation method is to sum up the weights of all its child nodes and then take the average value;
[0029] S5.2. The weight allocation method is to take advantage of the separation of structure and node behavior in the data-driven behavior tree. By building a data port in the data-driven behavior tree, the weight can be transferred to the behavior tree as structural data.
[0030] 7. A behavior tree importance evaluation and optimization system for drone maintenance according to any one of claims 1-6, characterized in that it comprises: a behavior tree module, a reinforcement learning module, a data acquisition module, a decision tree module, an importance analysis module, and a node weight allocation module;
[0031] The behavior tree module is used to build a behavior tree for performing maintenance tasks, control the behavior of the agent, and send the agent state and action sequence to the reinforcement learning module;
[0032] The reinforcement learning module is used to select actions for the behavior tree module, making the decision of the behavior tree dynamic, thereby enriching the operation data of the behavior tree;
[0033] The data collection module is used to collect the operation data of the behavior tree. When the condition node of the relevant subtask in the behavior tree is triggered, it records the current environment status, maintenance results and other information;
[0034] The decision tree module is used to perform preprocessing work such as one-hot encoding and missing value processing on the operating data, and use the processed data as a training set to train a prediction model for evaluating maintenance results;
[0035] The importance analysis module is used to evaluate the importance of data features in the model using the Shapley value method, and then evaluate the importance of conditional nodes in the behavior tree;
[0036] The node weight assignment module is used to calculate the node weight according to the importance of the conditional node, and pass the weight together with the behavior tree structure into the behavior tree using the XML file, and the behavior tree will perform the task based on this.
[0037] Beneficial effects of the present invention:
[0038] The present invention sets up a behavior tree importance evaluation and optimization method for UAV maintenance, designs a data collection method, and trains the behavior tree by using reinforcement learning so that the behavior tree runs while being trained, thereby collecting diverse operating data.
[0039] To solve the problem that behavior trees cannot adjust the weights of their subtasks, we designed a weighted sequential node that automatically calculates its own weight using recursive calculations and executes its subordinate weighted sequential nodes according to the weights.
[0040] By evaluating the importance of conditional nodes in the behavior tree, all nodes in the behavior tree are weighted, thereby changing the node execution order of the behavior tree and further optimizing the task execution logic of the behavior tree. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a flow chart of the behavior tree importance evaluation and optimization method for UAV maintenance disclosed in the present invention;
[0042] Figure 2 It is a structural diagram of the behavior tree importance evaluation and optimization system for drone maintenance disclosed in the present invention; DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0044] Embodiment 1
[0045] like Figure 1 As shown, this embodiment provides a behavior tree importance evaluation and optimization method for drone maintenance, and the specific steps are as follows:
[0046] S1. Build a behavior tree for performing drone maintenance tasks and integrate it into the agent. At the same time, configure a reinforcement learning module in the agent.
[0047] S2. Using the signal and slot mechanism in MOC, the behavior tree can send a signal to notify the reinforcement learning module before triggering the action node, and the reinforcement learning module selects actions for the behavior tree to collect the operation data of the behavior tree;
[0048] S3, preprocessing the operation data, and using it as a training set to train a decision tree algorithm, generating a decision model for predicting and analyzing maintenance results;
[0049] S4. Use Shapley value to evaluate the importance of each feature in the decision tree, thereby evaluating the importance of the conditional nodes in the behavior tree;
[0050] S5. Set weight calculation modules in the parent nodes of all conditional nodes in turn, and use these modules to gradually transfer weight values from the conditional nodes to each node of the entire behavior tree;
[0051] S6. The behavior tree determines the execution order of subtasks based on the weight of each node.
[0052] 2. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, characterized in that S1 comprises:
[0053] S1.1. The behavior tree and reinforcement learning modules are integrated in the intelligent agent. By converting the intelligent agent into MOC, the signal in the behavior tree can be transmitted to the reinforcement learning module to realize the communication between the two modules.
[0054] S1.2, Reinforcement learning algorithms can use reinforcement learning methods such as Q-Learning and Sarsa;
[0055] 3. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S2 comprises:
[0056] S2.1. After the agent is converted to MOC, a signal transmitter can be installed in the control node. When the control node is triggered, it will send the current node information to the reinforcement learning module in the form of a signal according to the state of the agent. The reinforcement learning module will make subsequent action selections and hand them over to the behavior tree for execution.
[0057] By setting a signal transmitter in the control node in the behavior tree, when the control node is triggered, the current state of the agent plus the current reward will be sent to the agent. The content of the transmission is shown in the following table:
[0058]
[0059]
[0060] The notification stores an action sequence in the reinforcement learning module. The reinforcement learning module can select actions based on the available actions under the node and send them to the behavior tree module. The action sequence is stored in xml:
[0061] <behaviortree>
[0062] <Node name="Node1">
[0063] <action> Action1< / action>
[0064] <action> Action2< / action>
[0065]
[0066] <Node name="Node2">
[0067] <action> Action3< / action>
[0068] <action> Action4< / action>
[0069]
[0070] < / behaviortree>
[0071] S2.2. The data taken in the behavior tree is the data feature in the conditional node. When the conditional node in the behavior tree is triggered, the current data will be recorded.
[0072] The conditional node data required for collection is {C 1 ,C 2 ,C 3 ,.....,C n-1}
[0073] Table 1 Collected agent data
[0074]
[0075] 4. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S3 comprises:
[0076] S3.1. Collect and preprocess the data during the operation of the behavior tree, including input conditions, execution process and output results, and then use these data to train the decision tree algorithm to build a decision model that can predict the success or failure of the maintenance task;
[0077] The entropy value calculation formula for the decision tree data feature constructed using the ID3 (Iterative Dichotomiser3) algorithm is as follows, where S is the sample, i.e., the conditional node, and the size of S is |S|:
[0078] The information gain is calculated as follows, where B is one of the conditional nodes of the set:
[0079]
[0080] S3.2, data preprocessing includes a series of preprocessing tasks such as missing value processing, one-hot encoding, and removing invalid features of running data;
[0081] Since the data feature "Condition Description" has no significant contribution to the decision tree training results, this feature should be removed from the data set. At the same time, for the "Result" feature, we need to perform one-hot encoding to facilitate subsequent analysis and model training.
[0082] S3.3. The data features in the decision tree are mapped to the conditional nodes in the behavior tree, that is, the data judgment logic in the behavior tree can find the corresponding logic in the decision tree.
[0083] In the experimental environment, the conditional node "PowerCheck" corresponds to "BatteryLevel" in the environment, and the conditional node "SensorCheck" corresponds to "SensorStatus" and "Temp" in the environment. When the subtask to which each conditional node belongs ends, the system will send a signal to instruct the environment to record the data of these tags and the inspection results. The corresponding relationship between the conditional node and the environment status will be recorded in XML in advance.
[0084] 5. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S4 comprises:
[0085] S4.1. Shapley value is a method to fairly distribute feature contributions. It can quantitatively analyze the contribution of each feature to the model prediction results. Through this evaluation, the impact of each feature in the decision tree on the prediction results can be determined, and then the impact of each conditional node in the behavior tree on the maintenance results can be determined.
[0086] S4.2. After normalizing the values finally calculated by using Shapley values in the decision tree, the weights of each feature in the decision tree are obtained. Since there is a mapping relationship between the features in the decision tree and the conditional nodes in the behavior tree, the weights of the conditional nodes in the behavior tree can be further determined by these feature weights.
[0087] The weight value is finally mapped to the relevant condition node through the established mapping relationship. The weight calculation method of the corresponding mapping table node "PowerCheck" and node "SensorCheck" is as follows:
[0088] Table 2 Weight distribution method
[0089]
[0090] 6. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1, wherein S5 comprises:
[0091] S5.1. The weight calculation module will calculate its own weight before being triggered. The specific calculation method is to sum up the weights of all its child nodes and then take the average value;
[0092] S5.2. The weight allocation method is to take advantage of the separation of structure and node behavior in the data-driven behavior tree. By building a data port in the data-driven behavior tree, the weight can be transferred to the behavior tree as structural data.
[0093] Embodiment 2
[0094] like Figure 2 As shown, this embodiment provides a behavior tree importance evaluation and optimization system for drone maintenance, which is characterized by comprising: a behavior tree module, a reinforcement learning module, a data acquisition module, a decision tree module, an importance analysis module, and a node weight allocation module;
[0095] The behavior tree module is used to build a behavior tree for performing maintenance tasks, control the behavior of the agent, and send the agent state and action sequence to the reinforcement learning module;
[0096] The reinforcement learning module is used to select actions for the behavior tree module, making the decision of the behavior tree dynamic, thereby enriching the operation data of the behavior tree;
[0097] The data collection module is used to collect the operation data of the behavior tree. When the condition node of the relevant subtask in the behavior tree is triggered, it records the current environment status, maintenance results and other information;
[0098] The decision tree module is used to perform preprocessing work such as one-hot encoding and missing value processing on the operating data, and use the processed data as a training set to train a prediction model for evaluating maintenance results;
[0099] The importance analysis module is used to evaluate the importance of data features in the model using the Shapley value method, and then evaluate the importance of conditional nodes in the behavior tree;
[0100] The node weight assignment module is used to calculate the node weight according to the importance of the conditional node, and pass the weight together with the behavior tree structure into the behavior tree using the XML file, and the behavior tree will perform the task based on this.
[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A behavior tree importance evaluation and optimization method for drone maintenance, characterized in that: The steps include: S1. Build a behavior tree for performing drone maintenance tasks and integrate it into the agent. At the same time, configure a reinforcement learning module in the agent. S2. Using the communication mechanism of the intelligent agent, the behavior tree can send a signal to notify the reinforcement learning module before triggering the action node, and the reinforcement learning module selects actions for the behavior tree to collect the operation data of the behavior tree; S3, preprocessing the operation data, and using it as a training set to train a decision tree algorithm, generating a decision model for predicting and analyzing maintenance results; S4. Use Shapley value to evaluate the importance of each feature in the decision tree, thereby evaluating the importance of the conditional nodes in the behavior tree; S5. Set weight calculation modules in the parent nodes of all conditional nodes in turn, and use these modules to gradually transfer weight values from the conditional nodes to each node of the entire behavior tree; S6. The behavior tree determines the execution order of subtasks based on the weight of each node.
2. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1 is characterized in that: The S1 includes: S1.
1. By integrating the behavior tree and reinforcement learning modules within the agent, and utilizing the communication mechanism of the agent, a signal can be sent to the reinforcement learning module before the behavior tree triggers the action node, thus realizing communication between the two modules. S1.
2. Reinforcement learning algorithms can use reinforcement learning methods such as Q-Learning and Sarsa.
3. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1 is characterized in that: The S2 includes: S2.1, the communication mechanism of the intelligent agent can use TCP / IP communication, meta-object system and other point-to-point communication methods; S2.2, through the communication mechanism of the agent, a signal transmitter is installed in the control node. When the node is triggered, the current node information and agent status are sent to the reinforcement learning module. The reinforcement learning module selects subsequent actions based on this information and sends them to the behavior tree for execution; S2.
3. The data taken in the behavior tree is the data feature in the conditional node. When the conditional node in the behavior tree is triggered, the current data will be recorded.
4. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1 is characterized in that: The S3 includes: S3.
1. Collect and preprocess the data during the operation of the behavior tree, including input conditions, execution process and output results, and use these data to train the decision tree algorithm to build a decision model for predicting the success or failure of the maintenance task; S3.2, data preprocessing includes a series of steps such as missing value processing, one-hot encoding and removal of invalid features; S3.
3. The data features in the decision tree are mapped to the conditional nodes in the behavior tree, that is, the data judgment logic in the behavior tree can find the corresponding logic in the decision tree.
5. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1 is characterized in that: The S4 includes: S4.
1. The result of Shapley value calculation of the decision tree is normalized to obtain the weight of each feature in the decision tree. Since there is a mapping relationship between the features of the decision tree and the conditional nodes of the behavior tree, these feature weights can be further used to determine the weights of the conditional nodes in the behavior tree.
6. The behavior tree importance evaluation and optimization method for drone maintenance according to claim 1 is characterized in that: The S5 includes: S5.
1. The weight calculation module will calculate its own weight before being triggered. The specific calculation method is to sum up the weights of all its child nodes and then take the average value; S5.
2. The weight allocation method is to take advantage of the separation of structure and node behavior in the data-driven behavior tree. By building a data port in the data-driven behavior tree, the weight can be transferred to the behavior tree as structural data.
7. A behavior tree importance evaluation and optimization system for drone maintenance according to any one of claims 1-6, characterized in that: include: Behavior tree module, reinforcement learning module, data acquisition module, decision tree module, importance analysis module, node weight allocation module; The behavior tree module is used to build a behavior tree for performing maintenance tasks, control the behavior of the agent, and send the agent state and action sequence to the reinforcement learning module; The reinforcement learning module is used to select actions for the behavior tree module, making the decision of the behavior tree dynamic, thereby enriching the operation data of the behavior tree; The data collection module is used to collect the operation data of the behavior tree. When the condition node of the relevant subtask in the behavior tree is triggered, it records the current environment status, maintenance results and other information; The decision tree module is used to perform preprocessing work such as one-hot encoding and missing value processing on the operating data, and use the processed data as a training set to train a prediction model for evaluating maintenance results; The importance analysis module is used to evaluate the importance of data features in the model using the Shapley value method, and then evaluate the importance of conditional nodes in the behavior tree; The node weight assignment module is used to calculate the node weight according to the importance of the conditional node, and pass the weight together with the behavior tree structure into the behavior tree using the XML file, and the behavior tree will perform the task based on this.