Reinforcement learning and behavior tree fusion method and device, electronic equipment and storage medium

Abstract

The invention relates to a reinforcement learning and behavior tree fusion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an environment observation result at a current moment under the condition that a target task is executed in an environment; the reward node determines a first reward value for the environment state at the current moment according to the environment observation result; the learning node obtains a prediction result according to the environment observation result and the first reward value and updates a first parameter corresponding to the learning node, and the prediction result is used for determining a first action executed by the agent at the next moment; and taking the prediction result determined at the last time as the final output result of the learning node until the first action meets the first condition. According to the method disclosed by the invention, the node of the behavior tree can automatically determine the action to be executed for completing the target task, so that the dependence on expert experience is reduced, the manual adjustment proportion is reduced, and the efficiency of establishing the behavior tree is improved on the premise of meeting the accuracy requirement.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and device for integrating reinforcement learning with behavior trees, an electronic device, and a storage medium. Background Art

[0002] With the development of technology, the robotics industry continues to move towards automation and intelligence, and intelligent agents have been widely studied in industries such as gaming, robotics, and combat simulation. Behavior trees, due to their strong interpretability, modularity, and high responsiveness, have garnered increasing attention in the field of intelligent agent research.

[0003] Behavior trees, as a method for modeling intelligent strategies, play a vital role in fields such as robotic control and game AI. Their modularity not only improves the efficiency of developing complex strategies but also greatly simplifies their management and maintenance. However, in practical applications, behavior trees are often constructed based on expert experience, requiring extensive manual tuning and lacking a learning mechanism. Summary of the Invention

[0004] In view of this, the present disclosure proposes an intelligent agent control solution.

[0005] According to one aspect of the present disclosure, there is provided an agent control method, which is applied to an agent equipped with a behavior tree, the behavior tree including: a learning node and a reward node, the method including: an agent control method, characterized in that it is applied to an agent equipped with a behavior tree, the behavior tree including: a learning node and a reward node, the method including: when performing a target task in an environment, obtaining an environmental observation result at a current moment; the reward node determining a first reward value for the environmental state at the current moment based on the environmental observation result; the learning node obtaining a prediction result based on the environmental observation result and the first reward value and updating a first parameter corresponding to the learning node, the prediction result being used to determine a first action to be performed by the agent at the next moment; until the first action satisfies a first condition, the last determined prediction result is used as the final output result of the learning node, and the final output result is used to determine a target action to be performed by the agent when performing the target task in the environment.

[0006] In one possible implementation, the learning node includes: a learning action node, the prediction result is the first action, and the learning node corresponds to multiple preset second actions; the learning node obtains the prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: determining the first execution probability of each second action based on the environmental observation result; determining the first action based on each first execution probability; updating the first parameter based on the environmental observation result, the first reward value and each first execution probability.

[0007] In one possible implementation, the learning node includes: a learning switching node, and the prediction result is: a target action node; the behavior tree includes: an action node, and the child nodes of the learning switching node include: at least two first action nodes, and the first action node corresponds to at least one preset third action; the learning node obtains a prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: determining the second execution probability of each of the third actions based on the environmental observation result; determining the target action node based on each of the second execution probabilities; the first action is the action output by the target action node; and updating the first parameter based on the environmental observation result, the first reward value, and each second execution probability.

[0008] In one possible implementation, the learning node includes: a learning parameter node, the learning parameter node corresponds to at least one action node or condition node; the learning node obtains a prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: obtaining the motion parameter of the next moment motion based on the environmental observation result; the first action is: the action output by the action node or condition node corresponding to the learning parameter node based on the motion parameter; updating the first parameter based on the environmental observation result and the first reward value; the method also includes: the learning parameter node inputs the motion parameter to the at least one action node or condition node; when the output of the at least one action node or condition node meets the first condition, the motion parameter determined for the last time is used as the parameter of the at least one action node or condition node.

[0009] In one possible implementation, the behavior tree includes: a reward pool, the reward pool includes at least one message queue, and the message queues correspond to tasks. The method further includes: the reward node associating the first reward value with a target message queue corresponding to the target task; the learning node sequentially obtaining the first reward associated with the target message queue based on the target task; when the environment terminates, the learning node obtains the remaining first reward in the target queue based on the target task, and uses the remaining first reward to update the first parameter.

[0010] In one possible implementation, the second execution probability of each of the third actions is determined based on the environmental observation results, including: the learning switching node obtains a score of the action corresponding to each first action node based on the environmental observation results; each score is normalized to obtain each second execution probability; the method includes: based on the environmental observation results, determining some of the actions corresponding to each first action node as illegal actions; and setting the score corresponding to the illegal action to a preset value, which preset value prevents the illegal action from being executed.

[0011] In one possible implementation, the status returned by the learning switching node receiving node includes: running and failure, and the learning switching node includes: a first learning switching node, a second learning switching node, and a third learning switching node; when the status returned by the child node of the first learning switching node is running, the first learning switching node selects the target action node at the previous moment as the target action node at the current moment; the second learning switching node presets a judgment condition, and determines whether to continue to determine the target action node based on the judgment result; when the status returned by the child node of the third learning switching node is failure, the third learning switching node selects the target action node at the previous moment as the target action node at the current moment.

[0012] According to another aspect of the present disclosure, an intelligent agent control device is provided. The device is provided in an intelligent agent, wherein a behavior tree is provided in the device. The behavior tree includes a learning node and a reward node. The device includes:

[0013] An environment observation result acquisition unit, used to acquire the environment observation result at the current moment when executing the target task in the environment;

[0014] The reward node is used to determine a first reward value for the current environmental state based on the environmental observation result;

[0015] The learning node is configured to obtain a prediction result based on the environmental observation result and the first reward value and update a first parameter corresponding to the learning node, wherein the prediction result is used to determine a first action to be performed by the agent at a next moment;

[0016] The final output result determination unit is used to use the last determined prediction result as the final output result of the learning node until the first action satisfies the first condition, and the final output result is used to determine the target action performed by the intelligent agent when performing the target task in the environment.

[0017] In a possible implementation, the learning node includes: a learning action node, the prediction result is the first action, and the learning node corresponds to a plurality of preset second actions;

[0018] The learning action node is used to:

[0019] determining a first execution probability of each of the second actions according to the environmental observation result;

[0020] determining the first action based on each of the first execution probabilities;

[0021] The first parameter is updated according to the environmental observation result, the first reward value, and each first execution probability.

[0022] In one possible implementation, the learning node includes a learning switch node, the prediction result is a target action node, the behavior tree includes an action node, and the child nodes of the learning switch node include at least two first action nodes, each of which corresponds to at least one preset third action.

[0023] The learning switch node is used to:

[0024] determining, based on the environmental observation result, a second execution probability of each of the third actions;

[0025] determining a target action node based on each of the second execution probabilities;

[0026] The first action is an action output by the target action node;

[0027] The first parameter is updated according to the environmental observation result, the first reward value, and each second execution probability.

[0028] In a possible implementation, the learning node includes: a learning parameter node, the learning parameter node corresponding to at least one action node or condition node;

[0029] The learning parameter node is used to:

[0030] Obtaining motion parameters for the next moment of motion based on the environmental observation results;

[0031] The first action is: an action output by the action node or condition node corresponding to the learning parameter node according to the motion parameter;

[0032] updating the first parameter according to the environmental observation result and the first reward value;

[0033] The learning parameter node is also used to:

[0034] inputting the motion parameters into the at least one action node or condition node;

[0035] When the output of the at least one action node or condition node satisfies the first condition, the motion parameter determined last time is used as a parameter of the at least one action node or condition node.

[0036] In a possible implementation, the behavior tree includes: a reward pool, the reward pool includes at least one message queue, and the message queue corresponds to the task.

[0037] The reward nodes are used to:

[0038] Associating the first reward value with a target message queue corresponding to the target task;

[0039] The learning node is also used to:

[0040] Based on the target task, sequentially obtain the first reward associated with the target message queue;

[0041] When the environment is terminated, a remaining first reward in the target queue is obtained based on the target task, and the first parameter is updated using the remaining first reward.

[0042] In a possible implementation, determining the second execution probability of each of the third actions according to the environmental observation result includes:

[0043] The learning switching node obtains scores of actions corresponding to the first action nodes according to the environment observation results;

[0044] Normalizing the scores to obtain the second execution probabilities;

[0045] The device comprises: an illegal action determination unit,

[0046] The illegal action determination unit is used for:

[0047] According to the environmental observation result, determining some actions among the actions corresponding to each of the first action nodes as illegal actions;

[0048] The score corresponding to the illegal action is set to a preset value, which prevents the illegal action from being executed.

[0049] In a possible implementation, the state returned by the learning switching node receiving node includes: running and failure, and the learning switching node includes: a first learning switching node, a second learning switching node, and a third learning switching node;

[0050] The first learning switch node is configured to select the target action node at the previous moment as the target action node at the current moment when the state returned by the child node of the first learning switch node is running;

[0051] The second learning switching node is used to make a judgment according to a preset judgment condition, and determine whether to continue to determine the target action node according to the judgment result;

[0052] The third learning switch node is used to select the target action node at the previous moment as the target action node at the current moment when the status returned by the child node of the third learning switch node is failure.

[0053] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0054] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0055] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0056] In the disclosed embodiment, the behavior tree includes a reward node, allowing the agent, after executing an action, to determine a first reward value based on the environment's feedback on that action and the observed results of the environment. Furthermore, the behavior tree includes a learning node, which can be trained based on the observed results and the first reward value until a first condition is met, at which point the final output of the learning node is determined. This allows the nodes in the behavior tree to automatically determine the actions required to complete the target task, reducing reliance on expert experience and the need for manual adjustments. This improves the efficiency of establishing the behavior tree while still meeting accuracy requirements.

[0057] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0059] Figure 1 A flowchart of the intelligent agent control method provided in an embodiment of the present disclosure.

[0060] Figure 2 A schematic diagram of the process of self-learning of an intelligent agent provided in an embodiment of the present disclosure.

[0061] Figure 3 A schematic diagram of the process of self-learning of an intelligent agent provided in an embodiment of the present disclosure.

[0062] Figure 4 A schematic diagram of the structure of an electronic device for body-controlled intelligent body provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0064] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0065] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0066] Figure 1 This is a flow chart of the intelligent agent control method provided by the embodiment of the present disclosure. Figure 1 As shown, the method is applied to an agent loaded with a behavior tree, wherein the behavior tree includes a learning node and a reward node. The method includes:

[0067] S11, when executing the target task in the environment, obtaining the current environment observation result.

[0068] In the disclosed embodiments, the environment refers to the physical environment in which the agent resides. The target task refers to the task performed by the agent; for example, in an autonomous driving scenario, the target task may be turning, parking, etc. Another example: in a production line scenario, the target task may be placing an object in a specified location, etc. The environmental observation result may be data reflecting the state of the environment acquired using the image acquisition device carried by the agent. This data may include RGB images, infrared images, radar signals, sound signals, etc. The disclosed embodiments do not limit the specific form of the observation results.

[0069] S12: The reward node determines a first reward value for the current environmental state based on the environmental observation result.

[0070] In the disclosed embodiments, a behavior tree may include a reward node. Reward rules may be preset in the reward node; the reward node determines a first reward value corresponding to the current environmental state based on the current environmental observation results. For example, a first reward value of 2 may be obtained if the vehicle successfully avoids an obstacle while driving, while a first reward value of -1 may be obtained if the vehicle collides with an obstacle while driving.

[0071] S13, the learning node obtains a prediction result based on the environmental observation result and the first reward value and updates the first parameter corresponding to the learning node, and the prediction result is used to determine the first action to be performed by the intelligent agent at the next moment.

[0072] The behavior tree also includes a learning node, and a neural network can be set in the learning node. For example, the learning node can include a value network and a policy network. For ease of description, the parameters of the neural network set in the learning node are named the first parameters. The learning node can obtain a prediction result based on the environmental observation results at the current moment and the first reward value. The prediction result can be the action performed by the agent at the next moment, or a parameter indicating the execution of the action, or a target subnode among the subnodes of the current learning node (the target subnode will eventually perform the action). In short, the prediction result can indicate the first action performed by the agent at the next moment. In addition, the learning node can adjust the first parameter based on the environmental observation results and the first reward value to perform self-learning of the learning node.

[0073] S14, until the first action satisfies the first condition, the last determined prediction result is used as the final output result of the learning node, and the final output result is used to determine the target action performed by the agent when performing the target task in the environment.

[0074] Repeat the above steps S11-S13 until the first action satisfies the pre-set first condition, and the last determined prediction result can be used as the final output result. The final output result can indicate the target action that the agent should perform when performing the target task in the current environment. The first condition can be set according to the scenario. For example: in the scenario of autonomous driving, the first condition can be set to: smoothly passing the road section without hitting the roadblock; the first condition can also be set to: the vehicle successfully parks in the parking space, etc. For another example: in the scenario of a production line, the first condition can be set to: the distance between the object placed by the agent and the surrounding of the placement table is not less than the distance threshold.

[0075] In the disclosed embodiment, the behavior tree includes a reward node, allowing the agent, after executing an action, to determine a first reward value based on the environment's feedback on that action and the observed results of the environment. Furthermore, the behavior tree includes a learning node, which can be trained based on the observed results and the first reward value until a first condition is met, at which point the final output of the learning node is determined. This allows the nodes in the behavior tree to automatically determine the actions required to complete the target task, reducing reliance on expert experience and the need for manual adjustments. This improves the efficiency of establishing the behavior tree while still meeting accuracy requirements.

[0076] In one possible implementation, the learning node includes: a learning action node, the prediction result is the first action, and the learning node corresponds to multiple preset second actions; the learning node obtains the prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: determining the first execution probability of each second action based on the environmental observation result; determining the first action based on each first execution probability; updating the first parameter based on the environmental observation result, the first reward value and each first execution probability.

[0077] In the disclosed embodiments, a learning node includes a learning action node. Several reinforcement learning actions with clear objectives and predefined goals can be abstracted into a learning action node. For ease of description, these reinforcement learning actions are named second actions. A learning action node can be a leaf node. If the learning node is a learning action node, the learning action node can predict the action to be output by the learning action node at the next moment based on the environmental observation results and the first reward value.

[0078] The learning action node can determine the first execution probability of each second action based on the environmental observation results. The second action corresponding to the highest first execution probability is then output as the first action by the learning action node. Alternatively, sampling can be performed based on the first execution probabilities to determine the first action, which is then output by the learning action node.

[0079] In an embodiment of the present disclosure, the parameters of the first neural network set at the learning action node can be adjusted based on the reinforcement learning algorithm according to the environmental observation results, the first reward value, and each first execution probability.

[0080] Exemplarily, the actor-critic algorithm is used for reinforcement learning. The first neural network includes: a first value network and a first policy network. The value network can predict the cumulative reward value obtained by completing the target task in the current state based on the environmental observation results. For the convenience of description, this cumulative reward value is named the cumulative reward value at the current moment. The first reward value at the previous moment can be obtained. And the cumulative reward value obtained by completing the target task in the state at the previous moment predicted by the value network at the previous moment. For the convenience of description, this cumulative reward value is named the cumulative reward value at the previous moment. The sum of the first reward value at the previous moment and the cumulative reward value at the current moment is used as a label, and the difference between the cumulative reward value at the previous moment and the label is determined, and the difference is used to adjust the parameters of the first value network. The parameters of the subsequent second value network and the parameters of the third value network can also be adjusted using this method.

[0081] The first policy network can predict the first execution probability of each preset second action based on environmental observations. Based on the current cumulative reward value and each first execution probability output by the first value network, the parameters of the first policy network are adjusted with the goal of maximizing the cumulative reward value at the next moment. This ensures that second actions with large first execution probabilities have higher execution probabilities at the next moment. This ensures that after the first action output by the learning action node, the environmental observation results obtained continuously approach the first condition, ultimately determining the target action for the learning action node.

[0082] In the disclosed embodiment, the learning action node can automatically determine the first action based on the environmental observation results, and then obtain the environmental observation results after the first action is performed to determine a new first action. This allows a single node in the behavior tree to automatically perform reinforcement learning. In addition, the first parameters of the neural network of the learning action node can be updated based on the environmental observation results, the first reward value, and each first execution probability. In this way, after each first action output by the learning action node, the obtained environmental observation results continue to approach the first condition to ultimately complete reinforcement learning. It is also possible to determine the target action that the learning action node should output when performing the target task.

[0083] In one possible implementation, the learning node includes: a learning switching node, and the prediction result is: a target action node; the behavior tree includes: an action node, and the child nodes of the learning switching node include: at least two first action nodes, and the first action node corresponds to at least one preset third action; the learning node obtains a prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: determining the second execution probability of each of the third actions based on the environmental observation result; determining the target action node based on each of the second execution probabilities; the first action is the action output by the target action node; and updating the first parameter based on the environmental observation result, the first reward value, and each second execution probability.

[0084] The learning action node can be a node that performs a basic action. The action node can be a node that performs an encapsulated complex action. The complex action here can be an action composed of multiple basic actions, or an action composed of a basic action and other actions. The first action node can include: an action node, and / or a learning motion node. The child nodes of the learning switching node can include at least two first action nodes. In the case where the learning node is a learning switching node, the learning switching node can predict the target action node for outputting the action at the next moment based on the environmental observation results and the first reward value. The target action node is a node selected from multiple first action nodes.

[0085] In an embodiment of the present disclosure, multiple third actions can be preset in the learning switch node, and these multiple third actions can be actions that can be executed by the child node (first action node) of the learning switch node. The third action can be a complex action or a basic action. The learning switch node can determine the second execution probability of each third action based on the environmental observation results. And the third action corresponding to the second execution probability with the highest value is used as the first action. Sampling can also be performed based on each second execution probability to determine the first action. The correspondence between the third action and the first action node can be preset in the learning switch node. Therefore, based on the correspondence, the first action node corresponding to the third action as the first action is determined, and the first action node is used as the target action node. The target action node outputs the first action.

[0086] In an embodiment of the present disclosure, the parameters of the second neural network set at the learning switching node can be adjusted based on the reinforcement learning algorithm according to the environmental observation results, the first reward value, and each second execution probability.

[0087] Exemplarily, an actor-critic algorithm is employed for reinforcement learning. The second neural network includes a second value network and a second policy network. The second value network can have the same or different structure as the first value network, and the training method for the second value network can be the same as that for the first value network, which will not be further described here.

[0088] The second policy network can have the same or different structure as the first policy network. The second policy network can predict the second execution probability of each preset third action based on the environmental observation results. Based on the current cumulative reward value and each second execution probability output by the second value network, the parameters of the second policy network are adjusted with the goal of maximizing the cumulative reward value at the next moment. The third action with a large second execution probability value can obtain a higher execution probability at the next moment. This is so that after the target action node determined by the learning switch node outputs the first action, the environmental observation results obtained continue to approach the first condition, and finally determine the target action node that should be selected to complete the target task.

[0089] In scenarios where multiple actions are available to complete a task, learning switch nodes can be used to determine the action node or learning motion node that best suits the environment. This allows the agent to autonomously determine the target action node based on the environment to execute the correct action, reducing reliance on expert experience and manual adjustments, while improving the efficiency of behavior tree building while still meeting accuracy requirements.

[0090] In one possible implementation, the learning node includes: a learning parameter node, the learning parameter node corresponds to at least one action node or condition node; the learning node obtains a prediction result and updates the first parameter corresponding to the learning node based on the environmental observation result and the first reward value, including: obtaining the motion parameter of the next moment motion based on the environmental observation result; the first action is: the action output by the action node or condition node corresponding to the learning parameter node based on the motion parameter; updating the first parameter based on the environmental observation result and the first reward value; the method also includes: the learning parameter node inputs the motion parameter to the at least one action node or condition node; when the output of the at least one action node or condition node meets the first condition, the motion parameter determined for the last time is used as the parameter of the at least one action node or condition node.

[0091] A learning parameter node can share the same parent node with at least one of its corresponding action nodes or condition nodes. For ease of description below, the action node or condition node corresponding to the learning parameter node is named a parameter execution node. The learning parameter node can output floating-point or integer motion parameters through learning, and the parameter execution node can read these motion parameters as its own parameters. In this way, an action node or condition node can be converted into a learning node. In the case where the learning node is a learning parameter node, the learning parameter node can predict the motion parameters to be output at the next moment based on the environmental observation results and the first reward value.

[0092] The learning parameter node can determine the probabilities of various preset motion parameters based on environmental observations. These preset motion parameters can be determined based on the target task. For example, if the target task is autonomous parking, the motion parameter can be the steering wheel rotation angle. Multiple rotation angles can be preset. The learning parameter node outputs the motion parameter corresponding to the highest probability. If the parameter execution node is an action node, the parameter execution node reads this motion parameter and outputs the action controlled by the motion parameter. If the parameter execution node is a conditional node, the parameter execution node determines an action node based on the motion parameter to execute the action controlled by the motion parameter.

[0093] In the embodiment of the present disclosure, the third neural network set in the learning parameter node can be adjusted based on the reinforcement learning algorithm according to the environmental observation results, the first reward value, and the probability of each preset parameter.

[0094] Exemplarily, an actor-critic algorithm is employed for reinforcement learning. The third neural network includes a third value network and a third policy network. The third value network can have the same or different structure as the first or second value network, and the training method for the third value network can be the same as for the first or second value network. Details are omitted here.

[0095] The structure of the third policy network can be the same as or different from that of the first policy network or the second policy network. The third policy network can predict the probability of each preset motion parameter based on the environmental observation results. Based on the current cumulative reward value output by the third value network and the probability of each preset motion parameter, the cumulative reward value at the next moment is maximized as the goal, and the parameters of the third policy network are adjusted. The motion parameters with large probability values ​​can obtain a higher probability at the next moment. This is so that after the parameter execution node executes the action according to the motion parameters determined by the learning parameter node, the environmental observation results obtained continue to approach the first condition. Then, the motion parameters that the parameter learning node should output to complete the target task are finally determined.

[0096] During reinforcement learning, a parameter learning node can learn one or more motion parameters. After the first condition is met, the parameter learning node can input the motion parameter(s) determined last during training into one or more parameter execution nodes. The parameter execution nodes to which the motion parameters are input can be the same as or different from the parameter execution nodes used during training. This makes it very convenient to convert an action node or condition node into a learning node. This allows traditional behavior trees to be easily converted into self-learning behavior trees, saving development time.

[0097] In one possible implementation, the behavior tree includes: a reward pool, the reward pool includes at least one message queue, and the message queues correspond to tasks. The method further includes: the reward node associating the first reward value with a target message queue corresponding to the target task; the learning node sequentially obtaining the first reward associated with the target message queue based on the target task; when the environment terminates, the learning node obtains the remaining first reward in the target queue based on the target task, and uses the remaining first reward to update the first parameter.

[0098] In an embodiment of the present disclosure, a behavior tree may include a reward pool, and the reward pool includes at least one message queue. A behavior tree may perform multiple tasks. For a single task, one or more learning nodes in the behavior tree may be trained separately. A single task may correspond to a message queue. When training a behavior tree for a single task, the obtained first reward value may be stored in the message queue corresponding to the task. The learning node may determine the target message queue based on the target task, and access the first reward value in the time sequence in which the first reward value is stored. When the environment is terminated, the learning node may obtain the unused first reward value (the remaining first reward value) in the message queue corresponding to each task executed in the environment, and use the remaining first reward value to update the first parameter of the learning node corresponding to the message queue.

[0099] If the simulation environment and the behavior tree execution environment are in different threads, the action output by the learning node may not take effect immediately, and the first reward value may not be fed back until several time periods have passed. After these time periods, the learning node may become inactive, resulting in reward loss and poor training effectiveness. To address this issue of reward loss caused by the simulation environment and the behavior tree execution environment being in different threads, a message queue corresponding to each task is set up in the reward pool. Whenever a learning node is activated, the first reward value is retrieved from the corresponding message queue in chronological order to complete reinforcement learning. Furthermore, when the environment terminates, the remaining first reward is retrieved all at once and used for reinforcement learning. This reduces the probability of wasting first reward values ​​and improves the effectiveness of behavior tree self-learning.

[0100] In one possible implementation, the second execution probability of each of the third actions is determined based on the environmental observation results, including: the learning switching node obtains a score of the action corresponding to each first action node based on the environmental observation results; each score is normalized to obtain each second execution probability; the method includes: based on the environmental observation results, determining some of the actions corresponding to each first action node as illegal actions; and setting the score corresponding to the illegal action to a preset value, which preset value prevents the illegal action from being executed.

[0101] In the disclosed embodiment, the second neural network may first determine the scores corresponding to each third action (i.e., the action corresponding to the first action node) based on the environmental observation results, and perform a normalization process on these scores, such as normalization, to obtain a second execution probability.

[0102] In certain circumstances, some child nodes (first action nodes) of a learning switch node should not output an action. For example, when executing an autonomous driving outbound mission and there is a sign prohibiting left turns in the environment, the first action node that outputs a left turn should not output an action (an illegal action). If a learning switch node selects the first action node that performs a left turn, the behavior tree's self-learning performance will be severely affected.

[0103] In view of this, illegal actions can be determined based on the environmental observation results. Then, a preset value is used as the score corresponding to the illegal action. The preset value can be a minimum value so that the second execution probability of the illegal action is close to zero during the subsequent normalization process.

[0104] For ease of understanding, the process of prohibiting the output of illegal actions is introduced below using formulas (1)-(3).

[0105]

[0106] πθ(·|s)=softmax(mask(ls)))

[0107]

[0108] in, is the policy gradient, representing the gradient of the objective function J(θ) with respect to the policy network parameters. θ is the policy network parameter. E is the expected value operator, representing the expected value of a random variable. Σ represents the sum from time steps t = 0 to t = T, where T is the total number of time steps. It is the advantage function, which represents the advantage value of the action at a certain time step t. It is the core part of the policy gradient, which represents the current policy π θThe gradient of the logarithm of π. a represents the action and s represents the environment state. θ (·|s) is the policy function, which represents the probability distribution of taking an action in a given state s. The softmax function represents the conversion of a vector into a probability distribution. mask(l(s)) represents the mask function, which is used to mask illegal actions. l(s) represents the value of all actions in state s. l(s)i represents the value of the i-th action in state s. M represents a very small value or negative infinity, which is used to represent the value of illegal actions so that their probability approaches 0 during softmax calculation. ai represents the i-th action. mask(l(s))i represents the mask function. If ai is a legal action, the mask function returns l(s)i; if ... i It is an illegal action, and the mask function returns M. Since M is a very large negative number, the probability of the corresponding action will be close to 0 after passing through the softmax function, thus achieving the effect of action masking.

[0109] In the disclosed embodiment, illegal actions can be determined based on environmental observation results and blocked to reduce the probability of outputting illegal actions, thereby improving the accuracy of automatic behavior tree training.

[0110] In one possible implementation, the status returned by the learning switching node receiving node includes: running and failure, and the learning switching node includes: a first learning switching node, a second learning switching node, and a third learning switching node; when the status returned by the child node of the first learning switching node is running, the first learning switching node selects the target action node at the previous moment as the target action node at the current moment; the second learning switching node presets a judgment condition, and determines whether to continue to determine the target action node based on the judgment result; when the status returned by the child node of the third learning switching node is failure, the third learning switching node selects the target action node at the previous moment as the target action node at the current moment.

[0111] A judgment condition can be preset in the second learning switch node. Each time the second learning switch node determines the target action node, it performs a judgment and obtains a judgment result. If the judgment result meets the judgment condition, the target action node is determined. If the judgment result does not meet the judgment condition, the target action node is not determined, and the preset action or the preset task can be executed. For example, in a drone task that determines flight actions, the judgment condition can be set in the second learning node as follows: the remaining fuel can support return; the preset task is a return task. Before determining the target action node, the second learning node first determines whether the remaining fuel can support return. If it can support return, it continues to determine the target action node; if it cannot support return, it executes the return task. This enables the behavior tree to have the function of forcibly interrupting tasks.

[0112] In the disclosed embodiment, the learning switch nodes are further subdivided to make the process of reinforcement training of the behavior tree more automated and more robust.

[0113] Figure 2 This is a flow chart of the self-learning process of the intelligent agent provided in the embodiment of the present disclosure. Figure 2 As shown, the agent obtains the environmental observation results (s1t, s2t, s3t, s4t, s5t) of the state st of the environment at the current time (time t). The behavior tree installed in the agent includes reward node A and reward node B. Reward node A determines the first reward value r1t based on the environmental observation result s1t and stores the first reward value r1t in message queue B in the reward pool. Reward node B determines the first reward value r2t based on the environmental observation result s2t and stores the first reward value r2t in message queue A in the reward pool. The behavior tree includes learning nodes A, learning node B, and learning node C. The environmental observation result s3t is input into learning node A, the environmental observation result s4t is input into learning node B, and the environmental observation result s5t is input into learning node C. Learning node A retrieves the first reward value r3t from message queue A; learning node B retrieves the first reward value r4t from message queue B. Based on the first reward value r4t and the environmental observation result s4t, the learning node B determines the action at+1, and the intelligent agent will output the action at+1 at the next moment (t+1 moment).

[0114] Furthermore, at time t, the agent outputs action at. The environment responds to action at, which is expressed as state st+1. Then, at time t+1, the agent obtains the environmental observation results of state st+1. The reward node can determine the first reward value based on the environmental observation results of state st+1 and store it in the reward pool. The learning node can determine the action, motion parameters, or target action node to be output at the next moment (time t+2) based on the environmental observation results of state st+1 and the first reward value. This process is repeated continuously until the output of the learning node meets the first condition and training stops.

[0115] Figure 3 This is a schematic diagram of the structure of the intelligent agent control device provided by the embodiment of the present disclosure. The device 20 is set in the intelligent agent, and a behavior tree is set in the device 20. The behavior tree includes: a learning node 23 and a reward node 22, such as Figure 3 As shown, the device 20 includes:

[0116] The environment observation result acquisition unit 21 is used to acquire the environment observation result at the current moment when executing the target task in the environment;

[0117] The reward node 22 is used to determine a first reward value for the current environmental state based on the environmental observation result;

[0118] The learning node 23 is configured to obtain a prediction result based on the environmental observation result and the first reward value and update a first parameter corresponding to the learning node, wherein the prediction result is used to determine a first action to be performed by the agent at a next moment;

[0119] The final output result determination unit 24 is used to use the last determined prediction result as the final output result of the learning node until the first action satisfies the first condition, and the final output result is used to determine the target action performed by the intelligent agent when performing the target task in the environment.

[0120] In a possible implementation, the learning node 23 includes: a learning action node, the prediction result is the first action, and the learning node 23 corresponds to a plurality of preset second actions;

[0121] The learning action node is used to:

[0122] determining a first execution probability of each of the second actions according to the environmental observation result;

[0123] determining the first action based on each of the first execution probabilities;

[0124] The first parameter is updated according to the environmental observation result, the first reward value, and each first execution probability.

[0125] In a possible implementation, the learning node 23 includes: a learning switch node, the prediction result is: a target action node; the behavior tree includes: an action node, and the child nodes of the learning switch node include: at least two first action nodes, each of which corresponds to at least one preset third action;

[0126] The learning switch node is used to:

[0127] determining, based on the environmental observation result, a second execution probability of each of the third actions;

[0128] determining a target action node based on each of the second execution probabilities;

[0129] The first action is an action output by the target action node;

[0130] The first parameter is updated according to the environmental observation result, the first reward value, and each second execution probability.

[0131] In a possible implementation, the learning node 23 includes: a learning parameter node, the learning parameter node corresponding to at least one action node or condition node;

[0132] The learning parameter node is used to:

[0133] Obtaining motion parameters for the next moment of motion based on the environmental observation results;

[0134] The first action is: an action output by the action node or condition node corresponding to the learning parameter node according to the motion parameter;

[0135] updating the first parameter according to the environmental observation result and the first reward value;

[0136] The learning parameter node is also used to:

[0137] inputting the motion parameters into the at least one action node or condition node;

[0138] When the output of the at least one action node or condition node satisfies the first condition, the motion parameter determined last time is used as a parameter of the at least one action node or condition node.

[0139] In a possible implementation, the behavior tree includes: a reward pool, the reward pool includes at least one message queue, and the message queue corresponds to the task.

[0140] The reward node 22 is used to:

[0141] Associating the first reward value with a target message queue corresponding to the target task;

[0142] The learning node 23 is also used for:

[0143] Based on the target task, sequentially obtain the first reward associated with the target message queue;

[0144] When the environment is terminated, a remaining first reward in the target queue is obtained based on the target task, and the first parameter is updated using the remaining first reward.

[0145] In a possible implementation, determining the second execution probability of each of the third actions according to the environmental observation result includes:

[0146] The learning switching node obtains scores of actions corresponding to the first action nodes according to the environment observation results;

[0147] Normalizing the scores to obtain the second execution probabilities;

[0148] The device 20 includes: an illegal action determination unit,

[0149] The illegal action determination unit is used for:

[0150] According to the environmental observation result, determining some actions among the actions corresponding to each of the first action nodes as illegal actions;

[0151] The score corresponding to the illegal action is set to a preset value, which prevents the illegal action from being executed.

[0152] In a possible implementation, the state returned by the learning switching node receiving node includes: running and failure, and the learning switching node includes: a first learning switching node, a second learning switching node, and a third learning switching node;

[0153] The first learning switch node is configured to select the target action node at the previous moment as the target action node at the current moment when the state returned by the child node of the first learning switch node is running;

[0154] The second learning switching node is used to make a judgment according to a preset judgment condition, and determine whether to continue to determine the target action node according to the judgment result;

[0155] The third learning switch node is used to select the target action node at the previous moment as the target action node at the current moment when the status returned by the child node of the third learning switch node is failure.

[0156] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. Its specific implementation can refer to the description of the above method embodiment. For the sake of brevity, it will not be repeated here.

[0157] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0158] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0159] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0160] Figure 4 This is a schematic diagram of the structure of an electronic device for body-controlled intelligent bodies provided by an embodiment of the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 4 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0161] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0162] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0163] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0164] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0165] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0166] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0167] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0169] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0170] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0171] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An intelligent agent control method, characterized in that: Applied to an agent loaded with a behavior tree, the behavior tree comprising: a learning node and a reward node, the method comprising: When executing the target task in the environment, obtain the current environmental observation results; The reward node determines a first reward value for the environmental state at the current moment according to the environmental observation result; The learning node obtains a prediction result according to the environmental observation result and the first reward value and updates a first parameter corresponding to the learning node, wherein the prediction result is used to determine a first action to be performed by the agent at a next moment; Until the first action satisfies the first condition, the last determined prediction result is used as the final output result of the learning node, and the final output result is used to determine the target action performed by the agent when performing the target task in the environment.

2. The method according to claim 1, characterized in that The learning node includes: a learning action node, the prediction result is the first action, and the learning node corresponds to a plurality of preset second actions; The learning node obtains a prediction result and updates a first parameter corresponding to the learning node according to the environmental observation result and the first reward value, including: Determining a first execution probability of each of the second actions according to the environmental observation result; determining the first action based on each of the first execution probabilities; The first parameter is updated according to the environmental observation result, the first reward value, and each first execution probability.

3. The method according to claim 1, characterized in that: The learning node includes: a learning switch node, and the prediction result is: a target action node; the behavior tree includes: an action node, and the child nodes of the learning switch node include: at least two first action nodes, and the first action node corresponds to at least one preset third action; The learning node obtains a prediction result and updates a first parameter corresponding to the learning node according to the environmental observation result and the first reward value, including: Determining, according to the environmental observation result, a second execution probability of each of the third actions; determining a target action node based on each of the second execution probabilities; The first action is an action output by the target action node; The first parameter is updated according to the environmental observation result, the first reward value, and each second execution probability.

4. The method according to claim 1, characterized in that: The learning node includes: a learning parameter node, the learning parameter node corresponds to at least one action node or condition node; The learning node obtains a prediction result and updates a first parameter corresponding to the learning node according to the environmental observation result and the first reward value, including: According to the environmental observation results, the motion parameters of the motion at the next moment are obtained; The first action is: an action output by an action node or a condition node corresponding to the learning parameter node according to the motion parameter; updating the first parameter according to the environmental observation result and the first reward value; The method further comprises: The learning parameter node inputs the motion parameter into the at least one action node or condition node; When the output of the at least one action node or condition node satisfies the first condition, the motion parameter determined last time is used as a parameter of the at least one action node or condition node.

5. The method according to claim 1, characterized in that The behavior tree includes: a reward pool, the reward pool includes at least one message queue, the message queue corresponds to the task, and the method further includes: The reward node associates the first reward value with a target message queue corresponding to the target task; The learning node sequentially obtains a first reward associated with the target message queue based on the target task; When the environment is terminated, the learning node obtains the remaining first reward in the target queue based on the target task, and updates the first parameter using the remaining first reward.

6. The method according to claim 3, characterized in that The determining, according to the environmental observation result, the second execution probability of each of the third actions comprises: The learning switching node obtains scores of actions corresponding to each of the first action nodes according to the environmental observation results; Performing normalization processing on each of the scores to obtain each of the second execution probabilities; The method comprises: According to the environmental observation result, determining some actions among the actions corresponding to each of the first action nodes as illegal actions; The score corresponding to the illegal action is set to a preset value, which prevents the illegal action from being executed.

7. The method according to claim 3, characterized in that The states returned by the learning switch node receiving node include: running and failure, and the learning switch node includes: a first learning switch node, a second learning switch node, and a third learning switch node; When the state returned by the child node of the first learning switch node is running, the first learning switch node selects the target action node at the previous moment as the target action node at the current moment; The second learning switching node makes a judgment according to a preset judgment condition, and determines whether to continue to determine the target action node according to the judgment result; When the status returned by the child node of the third learning switch node is failure, the third learning switch node selects the target action node at the previous moment as the target action node at the current moment.

8. An intelligent agent control device, characterized in that: A behavior tree is set in the device, and the behavior tree includes: a learning node and a reward node. The device includes: An environment observation result acquisition unit, used to acquire the environment observation result at the current moment when executing the target task in the environment; The reward node is used to determine a first reward value for the environmental state at a current moment according to the environmental observation result; The learning node is used to obtain a prediction result and update a first parameter corresponding to the learning node according to the environmental observation result and the first reward value, wherein the prediction result is used to determine a first action to be performed by the agent at a next moment; The final output result determination unit is used to use the last determined prediction result as the final output result of the learning node until the first action satisfies the first condition, and the final output result is used to determine the target action performed by the agent when performing the target task in the environment.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images