Robot navigation action decision method and device based on Monte Carlo tree search

Abstract

The present invention provides a robot navigation action decision method and device based on Monte Carlo tree search, which relates to the field of robot navigation technology. The method includes: obtaining system status information at the current moment; the system status information includes robot status information, pedestrian status information and obstacle status information of the robot's environment; using the system status information at the current moment as the root node to perform Monte Carlo tree search to obtain a constructed search tree; during the search process, the system status information corresponding to the leaf node is input into the robot strategy value model, and action selection and back propagation are performed based on the output result of the robot strategy value model; based on the number of action selections corresponding to the root node in the search tree, the robot navigation action decision result at the current moment is determined. The present invention can improve the accuracy of action decision-making and the stability of robot navigation.

Description

Technical Field

[0001] The present invention relates to the field of robot navigation technology, and in particular to a robot navigation action decision method and device based on Monte Carlo tree search. Background Art

[0002] In robot navigation tasks, reinforcement learning can simulate the control method of human navigation behavior. By extracting and modeling scene features, the robot motion planning control strategy is continuously optimized in interactive learning, and finally the navigation behavior that meets human expectations is obtained. However, the existing robot navigation technology only focuses on how to navigate in a multi-pedestrian environment. When dealing with surrounding obstacles, the usual practice is to treat obstacles as static pedestrians. However, the structural characteristics of obstacles are significantly different from those of pedestrians, which may cause instability in the navigation system. Summary of the invention

[0003] The embodiment of the present invention provides a robot navigation action decision method and device based on Monte Carlo tree search, which are used to improve the accuracy of the robot navigation action decision and ensure the stability of the navigation system.

[0004] In the first aspect, an embodiment of the present invention provides a robot navigation action decision method based on Monte Carlo tree search, comprising: obtaining system state information at the current moment; wherein the system state information includes robot state information, pedestrian state information and obstacle state information of the robot's environment; performing Monte Carlo tree search with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on system state information samples; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system state information, and output the action probability distribution and state value based on the above feature information; based on the number of action selections corresponding to the root node in the search tree, determine the robot navigation action decision result at the current moment.

[0005] In some embodiments, the Monte Carlo tree search is performed with the system state information at the current moment as the root node to obtain a constructed search tree, including: determining the i+1th starting leaf node and the i+1th action based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, and the number of action selections under the i-th leaf node; wherein i is an integer greater than or equal to 0; when i=0, the leaf node based on the i-th leaf node set is the root node; based on the system state information of the i+1th starting leaf node, determining the i+1th system state information after the robot performs the i+1th action; adding the i+1th system state information as the i+1th leaf node to the i-th search tree to obtain the i+1th search tree; wherein, when i =0, the i-th search tree is the root node; the i+1-th system state information is input into the robot strategy value model to obtain the state value of the i+1-th leaf node and the prior probability of each action to be selected under the i+1-th leaf node, and the number of selections and the average reward value of each action to be selected under the i+1-th leaf node are initialized; the number of selections corresponding to all selected actions in the path between the root node and the i+1-th leaf node is updated, and the average reward value corresponding to all selected actions is updated based on the state value of the i+1-th leaf node; the i+1-th leaf set in the i+1-th search tree is determined, i is set to i+1, and the search is continued until the preset number of searches is reached, and the i+1-th search tree is determined as the search tree.

[0006] In some embodiments, the robot strategy value model includes a robot-obstacle interaction subnetwork, a robot-pedestrian interaction subnetwork, a robot subnetwork and a fusion subnetwork; the inputting of the i+1th system state information into the robot strategy value model to obtain the state value of the i+1th leaf node and the prior probability of each action to be selected under the i+1th leaf node includes: based on the i+1th robot state information in the i+1th system state information, determining the first state information of the robot in the robot coordinate system, the second state information and the orientation state information in the world coordinate system; inputting the first state information and the i+1th obstacle state information in the i+1th system state information into the robot-obstacle interaction subnetwork to obtain the machine state value output by the robot-obstacle interaction subnetwork. The first interaction feature information between a person and an obstacle; the second state information and the i+1th pedestrian state information in the i+1th system state information are input into the robot-pedestrian interaction subnetwork to obtain the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork; the orientation state information is input into the robot subnetwork to obtain the robot's own feature information output by the robot subnetwork; the first interaction feature information, the second interaction feature information and the robot's own feature information are input into the fusion subnetwork to obtain the action probability distribution and state value output by the fusion subnetwork; based on the action probability distribution, the prior probability of each action to be selected under the i+1th leaf node is determined, and based on the state value, the state value of the i+1th leaf node is determined.

[0007] As a possible implementation, the robot-obstacle interaction subnetwork includes a first multi-layer perceptron, a pooling layer, a first cross-attention layer, a first feedforward network layer, and a second multi-layer perceptron; the first state information and the i+1th obstacle state information in the i+1th system state information are input into the robot-obstacle interaction subnetwork to obtain the first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction subnetwork, including: based on the i+1th obstacle state information, determining the boundary position information of each of the multiple obstacles; inputting the boundary position information of each of the multiple obstacles into the first multi-layer perceptron to obtain the first multi-layer perceptron The boundary feature information of each obstacle output by the multi-layer perceptron is input; the boundary feature information of each obstacle is input into the pooling layer to obtain the feature information of each obstacle output by the pooling layer; the first state information is input into the second multi-layer perceptron to obtain the first feature information of the robot output by the second multi-layer perceptron; the feature information of each obstacle and the first feature information are input into the first cross-attention layer to obtain the feature information of the obstacle that the robot pays attention to output by the first cross-attention layer; the feature information of the obstacle that the robot pays attention to is input into the first feedforward network layer to obtain the first interactive feature information output by the first feedforward network layer.

[0008] As a possible implementation, the robot-pedestrian interaction subnetwork includes a third multi-layer perceptron, a feature splicing layer, a fourth multi-layer perceptron, a second cross-attention layer, and a second feedforward network layer; the i+1th pedestrian state information includes state information of multiple pedestrians; the second state information and the i+1th pedestrian state information in the i+1th system state information are input into the robot-pedestrian interaction subnetwork to obtain the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork, including: inputting the state information of each of the multiple pedestrians into the third multi-layer perceptron to obtain the state information of each of the pedestrians output by the third multi-layer perceptron. feature information of the robot; inputting the second state information into the fourth multi-layer perceptron to obtain the second feature information of the robot output by the fourth multi-layer perceptron; inputting the state information of each of the multiple pedestrians and the second state information into the feature splicing layer to obtain the splicing feature information output by the feature splicing layer; inputting the feature information of each pedestrian, the second feature information and the splicing feature information into the second cross-attention layer to obtain the feature information of the pedestrians that the robot pays attention to output by the second cross-attention layer; inputting the feature information of the pedestrians that the robot pays attention to into the second feed-forward network layer to obtain the second interactive feature information output by the second feed-forward network layer.

[0009] In some embodiments, the fusion subnetwork includes a self-attention layer, a fifth multi-layer perceptron and a sixth multi-layer perceptron; the first interaction feature information, the second interaction feature information and the robot's own feature information are input into the fusion subnetwork to obtain the action probability distribution and state value output by the fusion subnetwork, including: inputting the first interaction feature information, the second interaction feature information and the robot's own feature information into the self-attention layer to obtain the global navigation feature information output by the self-attention layer; inputting the global navigation feature information into the fifth multi-layer perceptron to obtain the action probability distribution output by the fifth multi-layer perceptron; inputting the global navigation feature information into the sixth multi-layer perceptron to obtain the state value output by the sixth multi-layer perceptron.

[0010] In a second aspect, an embodiment of the present invention provides a robot navigation action decision device based on Monte Carlo tree search, comprising: an acquisition module, used to acquire system state information at a current moment; wherein the system state information includes robot state information, pedestrian state information and obstacle state information of the robot's environment; a search module, used to perform Monte Carlo tree search with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on system state information samples; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system state information, and output the action probability distribution and state value based on the above feature information; a determination module, used to determine the robot navigation action decision result at the current moment based on the number of action selections corresponding to the root node in the search tree.

[0011] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the program, the robot navigation action decision method based on Monte Carlo tree search described in the first aspect is implemented.

[0012] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot navigation action decision method based on Monte Carlo tree search described in the first aspect.

[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the robot navigation action decision method based on Monte Carlo tree search described in the first aspect.

[0014] According to the robot navigation action decision method and device based on Monte Carlo tree search of the embodiment of the present invention, the system status information at the current moment is obtained; wherein the system status information includes robot status information, pedestrian status information and obstacle status information of the robot's environment; Monte Carlo tree search is performed with the system status information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system status information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on system status information samples; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system status information, and output the action probability distribution and state value based on the above feature information; based on the number of action selections corresponding to the root node in the search tree, the robot navigation action decision result at the current moment is determined. The robot of the present invention performs multiple simulations and deductions by means of Monte Carlo tree search to consider various situations that may arise in the future. During the search process, the robot combines its own feature information and the interactive feature information between the robot and pedestrians and obstacles to perform action probability prediction and state value prediction. Finally, the action decision result is determined based on the search result, thereby improving the accuracy of action decisions and the stability of robot navigation, and also realizing the application of robot navigation in scenarios of human-machine-environment integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 One of the flow charts of the robot navigation action decision method based on Monte Carlo tree search provided by an embodiment of the present invention;

[0017] Figure 2 A second flow chart of a robot navigation action decision method based on Monte Carlo tree search provided in an embodiment of the present invention;

[0018] Figure 3A third flow chart of a robot navigation action decision method based on Monte Carlo tree search provided in an embodiment of the present invention;

[0019] Figure 4 This is a diagram showing an example of the structure of a robot strategy value model in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the structure of a robot navigation action decision device based on Monte Carlo tree search provided by an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In robot navigation tasks, reinforcement learning can simulate the control mode of human navigation behavior. By extracting and modeling scene features, the robot motion planning control strategy is continuously optimized in interactive learning, and finally the navigation behavior that meets human expectations is obtained. Therefore, methods based on the reinforcement learning framework have become the mainstream research direction for solving robot motion planning problems in dynamic environments. Social navigation methods based on reinforcement learning can be divided into two categories according to the type of input data: methods based on sensor data and methods based on agent information. The sensor data-based method directly maps the data of sensors (such as radars, cameras, etc.) into the robot's motion planning instructions by constructing an end-to-end motion planning reinforcement learning framework. In order to consider the motion state and interaction information of pedestrians in robot navigation planning, some methods use the state of pedestrians as input for navigation planning training and learning.

[0024] However, most of the existing methods that use pedestrian data as input only focus on how to navigate in a multi-pedestrian environment, without explicitly considering the surrounding environmental information. When dealing with surrounding obstacles, obstacles are usually treated as static pedestrians. However, there are significant differences between the structural characteristics of obstacles and pedestrians, and this simplified operation may cause instability in the navigation system. At the same time, most of these methods lack modeling of the interactive relationship between humans and the environment, making it difficult to apply them to scenarios where humans, machines, and the environment are integrated.

[0025] In order to solve the above problems, an embodiment of the present invention provides a robot navigation action decision method and device based on Monte Carlo tree search.

[0026] Figure 1 One of the flow charts of the robot navigation action decision method based on Monte Carlo tree search provided by the embodiment of the present invention. Figure 1 As shown, the method may include the following steps.

[0027] Step 101, obtaining the current system status information; wherein the system status information includes the robot status information, the pedestrian status information and the obstacle status information of the robot's environment.

[0028] In some embodiments, the robot state information may include the robot's speed information, position information, and orientation information, and may also include position relationship information between the robot's current position and the end point, etc.

[0029] Among them, the speed information, position information and orientation information of the robot can be determined based on the sensors equipped by the robot itself. The speed information of the robot can include linear velocity, angular velocity, etc. The position information of the robot can include the coordinate information of the robot at the current moment. The orientation information of the robot can include the direction of the robot. The position relationship information between the position of the robot at the current moment and the end point can be determined based on the position information of the robot at the current moment and the position information of the end point, for example, it can include the distance between the position of the robot at the current moment and the end point, the vector of the line connecting the robot and the end point, the angle, etc.

[0030] In some embodiments, pedestrian status information can also be obtained based on sensors equipped by the robot, for example, it can include the position information and speed information of all pedestrians within a certain distance from the robot. Obstacle status information can be the position information of all obstacles in the robot's environment, or the position information of obstacles within the robot's field of view.

[0031] As an implementation method, the position information of obstacles within the robot's field of view can be determined by constructing an isosceles triangle with the robot's current position as the vertex and the current velocity direction as the axis of symmetry, and the vertex angle of the isosceles triangle is and waist length L as preset values; based on the coordinate information of the isosceles triangle and the coordinate information of all obstacles in the environment, determine one or more intersection areas of the obstacle and the triangle, and the position information of all intersection areas is determined as the obstacle state information.

[0032] Step 102, performing a Monte Carlo tree search with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on system state information samples; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system state information, and output the action probability distribution and state value based on the above feature information.

[0033] Among them, Monte Carlo tree search is a heuristic algorithm that combines simulation and tree search. In an embodiment of the present invention, the robot navigation action decision is simulated multiple times based on the Monte Carlo tree search algorithm, so as to recursively construct a search tree, and finally obtain the robot's action decision result based on the statistical information of the search tree.

[0034] In some embodiments, the nodes in the constructed search tree represent the system state information in each simulation, starting from the node S i An outgoing edge represents an action chosen by the robot in this state. After taking this action, the robot transfers to node S i+1 Corresponding system status information. That is to say, in the Monte Carlo tree search process, the actions under the leaf nodes are selected, and new leaf nodes are obtained by simulating the execution of the actions until the preset number of searches is reached to obtain the constructed search tree. Among them, the leaf node refers to the node where there are still actions that have not been selected. For example, multiple actions corresponding to each node can be pre-set. If multiple actions corresponding to a node have been selected during the search tree construction process, the node will no longer participate in the selection. If multiple actions corresponding to a node have not been selected, or there is at least one action among the multiple actions corresponding to a node that has not been selected, then the node is a leaf node.

[0035] In some embodiments, each edge of the search tree stores a set of statistical data, including the number of times the action is selected in this state, the average reward value, the prior probability, etc. The process of performing a Monte Carlo tree search with the current system state information as the root node includes: (1) a selection phase, traversing from the root node based on the Monte Carlo tree search algorithm, and selecting the action corresponding to the leaf node based on a preset search strategy; (2) an expansion phase, based on the selected leaf node and action, simulating the robot to perform the action in the system state corresponding to the leaf node to obtain a new system state, and adding the new system state as a new leaf node to the search tree, and initializing the statistical data of the new leaf node; (3) back propagation, obtaining the state value of the new leaf node, and based on the state value of the new leaf node, updating the statistical information of the edges involved in all nodes in the selected path, and returning to (1) to repeat the execution until the number of searches reaches a threshold, and the final search tree is obtained.

[0036] Among them, in the selection stage, the score corresponding to each selection can be calculated based on the prior probability of the action to be selected under each leaf node, the average reward value, and the number of selections of each action under each leaf node, and the corresponding action under the leaf node with the highest score is selected for search simulation to expand to obtain a new leaf node. In the back propagation stage, the state value of the new leaf node needs to be determined, and back propagation is performed based on the state value to update the number of selections and average reward values ​​of all selected actions involved in the current simulation path.

[0037] In order to improve the accuracy of the prior probabilities of multiple actions under each leaf node, the embodiment of the present invention uses the robot strategy value model, combined with the interaction between the robot and pedestrians, the interaction between the robot and obstacles, and the robot's own state information, to obtain the probability distribution of actions under the corresponding state, and based on the probability distribution of actions under the corresponding state, determine the prior probability of each action under the node. For example, after adding a leaf node, the system state information of the newly added leaf node can be input into the robot strategy value model to obtain the action probability distribution output by the robot strategy value model, and based on the action probability distribution, determine the probability of each action, and use the probability of each action to determine the prior probability of the corresponding action to be selected under the newly added leaf node.

[0038] In some embodiments, the robot policy value model has the ability to output action probability distribution and state value based on system state information, so in the back-propagation stage, the state value information of the newly added leaf node can be input into the robot policy value model to obtain the state value input by the robot policy value model, and back-propagation can be performed based on the state value.

[0039] Among them, the robot strategy value model can be a reinforcement learning model trained based on system state information samples.

[0040] As an implementation manner, the robot strategy value model may include an obstacle feature extraction layer, a pedestrian feature extraction layer, a robot feature extraction layer, a first interactive feature extraction layer, a second interactive feature extraction layer, a fusion layer and a mapping layer. The obstacle state information in the system state information is input into the obstacle feature extraction layer to obtain the obstacle feature information output by the obstacle feature extraction layer; the pedestrian state information in the system state information is input into the pedestrian feature extraction layer to obtain the pedestrian feature information output by the pedestrian feature extraction layer; the robot state information in the system state information is input into the robot feature extraction layer to obtain the robot's own feature information output by the robot feature extraction layer; the obstacle feature information and the robot's own feature information are input into the first interactive feature extraction layer to obtain the first interactive feature information between the robot and the obstacle output by the first interactive feature extraction layer; the pedestrian feature information and the robot's own feature information are input into the second interactive feature extraction layer to obtain the second interactive feature information between the robot and the pedestrian output by the second interactive feature extraction layer; the first interactive feature information, the second interactive feature information and the robot's own feature information are input into the fusion layer to obtain the fusion feature information output by the fusion layer; the fusion feature information is input into the mapping layer to obtain the action probability distribution and state value output by the mapping layer.

[0041] Step 103, based on the number of action selections corresponding to the root node in the search tree, determine the robot navigation action decision result at the current moment.

[0042] The number of action selections corresponding to the root node includes the number of times each action is selected under the root node. In some embodiments, the action with the most selections can be determined as the robot navigation action decision result at the current moment.

[0043] According to the robot navigation action decision method based on Monte Carlo tree search of an embodiment of the present invention, the system state information at the current moment is obtained; wherein the system state information includes robot state information, pedestrian state information and obstacle state information of the robot's environment; Monte Carlo tree search is performed with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on system state information samples; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system state information, and output the action probability distribution and state value based on the above feature information; based on the number of action selections corresponding to the root node in the search tree, the robot navigation action decision result at the current moment is determined. The robot of the present invention performs multiple simulations and deductions by means of Monte Carlo tree search to consider various situations that may arise in the future. During the search process, the robot combines its own feature information and the interactive feature information between the robot and pedestrians and obstacles to perform action probability prediction and state value prediction. Finally, the action decision result is determined based on the search result, thereby improving the accuracy of action decisions and the stability of robot navigation, and also realizing the application of robot navigation in scenarios of human-machine-environment integration.

[0044] Figure 2 The second flow chart of the robot navigation action decision method based on Monte Carlo tree search provided by the embodiment of the present invention. Figure 2 As shown, based on the above embodiments, Figure 1 The implementation process of step 102 in may include the following steps.

[0045] Step 201, based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, and the number of action selections under each leaf node, determine the i+1th starting leaf node and the i+1th action; wherein i is an integer greater than or equal to 0; when i=0, the leaf node in the i-th leaf node set is the root node.

[0046] In some embodiments, the i+1th starting leaf node and the i+1th action may be selected by a UCB (Upper Confidence Bound) strategy algorithm, wherein the i+1th starting leaf node refers to the starting system state information of the i+1th simulation, and the i+1th action refers to the action performed by the robot simulation under the starting system state information.

[0047] The i-th leaf node set refers to the set of all leaf nodes in the i-th search tree obtained after i simulations. In other words, each selection phase requires selection from the corresponding pending actions under all leaf nodes. The pending actions refer to the actions that have not been selected yet.

[0048] As a possible implementation method, the score of the action to be selected under each leaf node can be determined based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, as well as the number of action selections under each leaf node, and the action to be selected with the largest score is taken as the i+1th action, and the leaf node corresponding to the i+1th action is taken as the i+1th starting leaf node.

[0049] As an example, based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, as well as the number of action selections under each leaf node, the process of determining the score of the action to be selected under each leaf node can be shown in the following formula (1).

[0050] (1);

[0051] in, For leaf nodes Next action to be selected score; For leaf nodes Next action to be selected The average reward value of For leaf nodes Next action to be selected The prior probability of For leaf nodes Next action to be selected The number of selections; For leaf nodes The number of choices for the next j-th action, is the number of actions under each node; and is a constant used for balance and The relative importance between them.

[0052] Step 202, based on the system status information of the i+1th starting leaf node, determine the i+1th system status information after the robot performs the i+1th action.

[0053] In some embodiments, the i+1th system state information may include the i+1th obstacle state information, the i+1th pedestrian state information, and the i+1th robot state information. In other words, the i+1th system state information refers to the system state information of the new system state after the robot performs the i+1th action in the system state corresponding to the i+1th starting leaf node.

[0054] In some embodiments, the pedestrian status information in the system status information of the i+1th starting leaf node can be determined first, and the pedestrian status information and the i+1th action can be input into the pedestrian trajectory prediction model to obtain the i+1th pedestrian status information output by the pedestrian trajectory prediction model. Among them, the pedestrian trajectory prediction model can be a model in the relevant technology, which is used to implement the output of updated pedestrian status information based on the pedestrian status information and the robot execution action. Based on the robot status information and the i+1th action in the system status information of the i+1th starting leaf node, the i+1th robot status information can be derived according to the kinematics of the robot. If the i+1th obstacle status information is the obstacle status information within the robot's field of view, then based on the i+1th robot status information, an isosceles triangle with the robot position as the vertex and the speed direction as the axis of symmetry can be constructed, and based on the position information of the surrounding environment obstacles, the obstacle position information of the area intersecting with the isosceles triangle is determined, and it is used as the i+1th obstacle status information.

[0055] Step 203, adding the i+1th system status information as the i+1th leaf node to the i+1th search tree to obtain the i+1th search tree; wherein, when i=0, the i+1th search tree is the root node.

[0056] Step 204, input the i+1th system state information into the robot strategy value model, obtain the state value of the i+1th leaf node, and the prior probability of each action to be selected under the i+1th leaf node, and initialize the number of selections and the average reward value of each action to be selected under the i+1th leaf node.

[0057] In some embodiments, the i+1th system state information is input into the robot strategy value model to obtain the action probability distribution and state value output by the robot strategy value model, the state value is determined as the state value of the i+1th leaf node, and the probability of each action in the action probability distribution output by the robot strategy value model is determined as the probability of the corresponding action to be selected.

[0058] The number of selections of each action to be selected under the initialized i+1th leaf node is 0. The average reward value of each action to be selected under the initialized i+1th leaf node can also be 0.

[0059] In some embodiments, the process of initializing the average reward value of each action to be selected under the i+1th leaf node may include: determining whether the initialized i+1th leaf node is a root node. If the initialized i+1th leaf node is a root node, the average reward value of each action to be selected under the initialized i+1th leaf node is 0; if the initialized i+1th leaf node is not a root node, the average reward value of each action to be selected under the initialized i+1th leaf node is the average state reward value of the parent node.

[0060] As an example, the calculation process of the average state reward value of each node can be shown as follows (2).

[0061] (2);

[0062] in, For any node in the search tree; for The average state reward value of the node; is the root node; represents the indicator function; for The parent node of the node; for The average state reward value of the node; for The average reward value of the j-th action under the node.

[0063] Step 205, update the selection times corresponding to all selected actions in the path between the root node and the i+1th leaf node, and update the average reward values ​​corresponding to all selected actions based on the state value of the i+1th leaf node.

[0064] Among them, all the selected actions in the path from the root node to the i+1th leaf node refer to the actions corresponding to the edges between all the nodes involved in the path from the root node to the i+1th leaf node. For example, all the nodes involved in the path from the root node s0 to the i+1th leaf node s3 include nodes s0, s1, s2, and s3, then the action a1 selected from s0 to s1, the action a2 selected from s1 to s2, and the action a3 selected from s2 to s3 are all the selected actions in the path from the root node to the i+1th leaf node.

[0065] It can be understood that the simulated robot executes the i+1th action to obtain the i+1th leaf node, which is equivalent to executing all the selected actions in the path between the root node and the i+1th leaf node once in this process. Therefore, to update the number of selections corresponding to all the selected actions in the path between the root node and the i+1th leaf node, the number of selections corresponding to all the selected actions in the path between the root node and the i+1th leaf node is increased by 1.

[0066] In some embodiments, the implementation process of updating the average reward value corresponding to all selected actions based on the state value of the i+1th leaf node may include: determining the cumulative discounted returns of all selected actions in the path between the root node and the i+1th leaf node based on the state value of the i+1th leaf node; obtaining the updated average reward value corresponding to all selected actions based on the cumulative discounted returns of all selected actions in the path between the root node and the i+1th leaf node, as well as the number of selections and the average reward value corresponding to all selected actions in the path between the root node and the i+1th leaf node.

[0067] As a possible implementation method, the implementation process of determining the cumulative discounted rewards of all selected actions in the path between the root node and the i+1th leaf node based on the state value of the i+1th leaf node may include: inputting the system state information of the i+1th leaf node into a predefined reward function to determine the immediate reward for executing the i+1th action, and the i+1th search tree stores the immediate rewards corresponding to the actions under each node; based on the immediate rewards corresponding to all selected actions in the path between the root node and the i+1th leaf node, and the state value of the i+1th leaf node, determine the cumulative discounted rewards of all selected actions in the path between the root node and the i+1th leaf node based on the state value of the i+1th leaf node. As an example, the i+1th leaf node is , the nodes involved in the path from the root node to the i+1th leaf node include , , …, , any node to The selected action between , the process of calculating the cumulative discounted return is shown in the following formula (3).

[0068] (3);

[0069] in, For Action The cumulative discounted return of is a value between 1 and d; is the default value; For Action Instant rewards; For Node The status value.

[0070] Based on the above example, based on the cumulative discounted returns of all selected actions in the path from the root node to the i+1th leaf node, and the number of selections and average reward values ​​corresponding to all selected actions in the path from the root node to the i+1th leaf node, the implementation process of obtaining the updated average reward values ​​corresponding to all selected actions is shown in the following formula (4).

[0071] (4);

[0072] in, for Select an action under a node The number of times here is the value before updating in step 205; for Node Action The average reward value.

[0073] Step 206, determine the i+1th leaf set in the i+1th search tree, set i to i+1, continue searching until a preset number of searches is reached, and determine the i+1th search tree as the search tree.

[0074] That is to say, if i+1 is equal to the preset number of searches, the i+1th search tree is determined as the constructed search tree. If i+1 is less than the preset number of searches, the i+1th leaf set in the i+1th search tree is determined, and the search is continued back to step 201.

[0075] According to the robot navigation action decision method based on Monte Carlo tree search of an embodiment of the present invention, based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, and the number of action selections under each leaf node, the i+1th starting leaf node and the i+1th action are determined; based on the system state information of the i+1th starting leaf node, the i+1th system state information after the robot performs the i+1th action is determined; the i+1th system state information is added to the i+1th search tree as the i+1th leaf node to obtain the i+1th search tree; the i+1th system state information is input into the robot strategy value model to obtain the state value of the i+1th leaf node and the prior probability of each action to be selected under the i+1th leaf node, and the number of selections and the average reward value of each action to be selected under the i+1th leaf node are initialized; the number of selections corresponding to all selected actions in the path between the root node and the i+1th leaf node is updated, and the average reward value corresponding to all selected actions is updated based on the state value of the i+1th leaf node; Determine the i+1th leaf set in the i+1th search tree, set i to i+1, continue searching until a preset number of searches is reached, and determine the i+1th search tree as the search tree. The present invention performs multiple simulations and deductions by means of Monte Carlo tree search to consider various situations that may occur in the future, thereby selecting the optimal action decision result.

[0076] Figure 3 The third flow chart of the robot navigation action decision method based on Monte Carlo tree search provided by the embodiment of the present invention. In some embodiments, the robot strategy value model includes a robot-obstacle interaction sub-network, a robot-pedestrian interaction sub-network, a robot sub-network and a fusion sub-network. Figure 3 As shown, based on the above embodiment, the i+1th system state information is input into the robot strategy value model to obtain the state value of the i+1th leaf node, and the implementation process of the prior probability of each action to be selected under the i+1th leaf node may include the following steps.

[0077] Step 301 : Based on the (i+1)th robot state information in the (i+1)th system state information, determine the first state information of the robot in the robot coordinate system, the second state information and the orientation state information in the world coordinate system.

[0078] In some embodiments, the robot coordinate system refers to a rectangular coordinate system based on the robot position information and speed information in the i+1th robot state information, with the origin of the robot's two-dimensional position and the speed direction as the x-axis. The first state information in the robot coordinate system may include the speed information of the robot in the i+1th system state, and the unit vector of the line between the robot's two-dimensional position and the end point in the robot coordinate system.

[0079] In some embodiments, the second state information may include the position information, speed information, and orientation information of the robot in the world coordinate system in the (i+1)th system state. For example, the second state information may include the position information, speed information, and cosine and sine values ​​of the robot's orientation in the world coordinate system in the (i+1)th system state.

[0080] In some embodiments, the robot's position information may include the distance between the robot and the end point in the (i+1) system state, the distance between the robot and the end point and the robot's orientation, the velocity vector of the robot's linear velocity in the robot-end point coordinate system, the robot's rotational angular velocity, etc. The robot-end point coordinate system refers to a rectangular coordinate system established with the line between the robot and the end point as the x-axis.

[0081] Step 302: Input the first state information and the (i+1)th obstacle state information in the (i+1)th system state information into the robot-obstacle interaction sub-network to obtain the first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction sub-network.

[0082] That is, the robot-obstacle interaction subnetwork is used to extract features from the first state information and the (i+1)th obstacle state information, and perform interaction feature extraction based on the extracted features to obtain first interaction feature information between the robot and the obstacle.

[0083] As a possible implementation method, the robot-obstacle interaction subnetwork includes a first feature extraction layer, a second feature extraction layer and a first interactive feature extraction layer. The first state information is input into the first feature extraction layer to obtain the first feature information of the robot; the i+1th obstacle state information is input into the second feature extraction layer to obtain the obstacle feature information; the first feature information and the obstacle feature information are input into the first interactive feature extraction layer to obtain the first interactive feature information between the robot and the obstacle.

[0084] In some embodiments, Figure 4 As shown, the robot-obstacle interaction subnetwork includes a first multi-layer perceptron, a pooling layer, a first cross attention layer, a first feedforward network layer, and a second multi-layer perceptron. The first state information and the i+1th obstacle state information in the i+1th system state information are input into the robot-obstacle interaction subnetwork, and the implementation process of obtaining the first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction subnetwork includes the following steps.

[0085] Step S1, based on the (i+1)th obstacle state information, determining the boundary position information of each of the plurality of obstacles.

[0086] In some embodiments, the i+1th obstacle state information may be obstacle information in the robot's environment, in which case the obstacle state information in all states is consistent. In other embodiments, the i+1th obstacle state information may be obstacle state information within the robot's field of view in the i+1th system state.

[0087] As a possible implementation method, based on the state information of the i+1th obstacle, the process of determining the boundary position information of each of the multiple obstacles includes: based on the position and speed of the robot in the i+1th system state, constructing an isosceles triangle with the position as the vertex and the direction of the speed as the axis of symmetry, and the vertex angle and waist length of the isosceles triangle are set values; based on the position information of the surrounding obstacles and the position information of the isosceles triangle of the robot, at least one intersection area of ​​the obstacle within the isosceles triangle, each intersection area is regarded as an obstacle; each intersection area can be regarded as a polygon formed by a set of vertices connected in a counterclockwise order, and the characteristic vector of each edge in the polygon in the robot coordinate system is used as the boundary position information of the corresponding obstacle. In other words, the boundary position information of each obstacle contains the characteristic vector of each edge in the polygon corresponding to the obstacle in the robot coordinate system. For example, if obstacle j is a K-gon, the boundary position information of obstacle j is ,in, is the feature vector of the kth edge of obstacle j in the robot coordinate system.

[0088] Step S2: inputting the boundary position information of each of the multiple obstacles into a first multi-layer perceptron to obtain boundary feature information of each obstacle output by the first multi-layer perceptron.

[0089] The first multi-layer perceptron is a coding layer, which is used to perform nonlinear transformation on input information through an activation function based on a linear combination of weights and biases.

[0090] In some embodiments, the boundary feature information of each obstacle may include feature information of each edge corresponding to the obstacle. For example, the boundary position information of obstacle j is: , after feature extraction, the boundary feature information of obstacle j is obtained as , is the feature information of the kth edge of obstacle j.

[0091] Step S3: input the boundary feature information of each obstacle into the pooling layer to obtain the feature information of each obstacle output by the pooling layer.

[0092] In some embodiments, the pooling layer may perform a maximum pooling operation on all feature information belonging to the same obstacle to obtain feature information of each obstacle, as specifically shown in the following formula (5).

[0093] (5);

[0094] in, is the characteristic information of obstacle j.

[0095] Step S4: input the first state information into a second multi-layer perceptron to obtain first feature information of the robot output by the second multi-layer perceptron.

[0096] The second multi-layer perceptron is used to extract features from the first state information to obtain first feature information.

[0097] Step S5, inputting the feature information of each obstacle and the first feature information into the first cross-attention layer, and obtaining the obstacle feature information that the robot pays attention to output by the first cross-attention layer.

[0098] When avoiding obstacles, the robot pays different attention to obstacles in different directions around it, which is related to its own position, orientation, speed direction and the direction of the end point. Therefore, a cross-attention mechanism is used to model the impact of different obstacles on the robot's motion planning and state evaluation.

[0099] In some embodiments, the first cross-attention layer maps the first feature information to a query vector through linear transformation, maps the feature information of the obstacle to a key vector and a value vector, obtains the robot's attention coefficient for each obstacle, and then obtains the obstacle feature information that the robot pays attention to through weighted summation of the attention coefficient and the value vector. Specifically, as shown in the following formulas (6) and (7).

[0100] (6);

[0101] (7);

[0102] in, Based on the first feature information The mapped query vector; The key vector based on the feature vector mapping of obstacle j; is the value vector mapped based on the feature vector of obstacle j; is the robot’s attention coefficient to obstacle j; The obstacle feature information that the robot is concerned about; , and is the mapping matrix in the first cross-attention layer; is the number of obstacles.

[0103] Step S6: input the obstacle feature information that the robot is concerned about into the first feedforward network layer to obtain the first interactive feature information output by the first feedforward network layer.

[0104] Among them, the first feedforward network layer can be a feedforward network with residual connection, which is used to transform the obstacle feature information that the robot pays attention to to obtain the first interactive feature information.

[0105] Step 303: input the second state information and the i+1th pedestrian state information in the i+1th system state information into the robot-pedestrian interaction subnetwork to obtain the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork.

[0106] In some embodiments, the state information of the i+1th pedestrian may include state information of multiple pedestrians. The robot-pedestrian interaction subnetwork is used to extract features from the second state information of the robot and the state information of each pedestrian, and perform interaction feature extraction based on the extracted features to obtain second interaction feature information between the robot and the pedestrian.

[0107] As a possible implementation, Figure 4 As shown, the robot-pedestrian interaction subnetwork includes a third multi-layer perceptron, a feature concatenation layer, a fourth multi-layer perceptron, a second cross attention layer, and a second feedforward network layer. The second state information and the i+1th pedestrian state information in the i+1th system state information are input into the robot-pedestrian interaction subnetwork, and the implementation process of obtaining the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork may include the following steps.

[0108] Step S7: input the state information of each of the multiple pedestrians into a third multi-layer perceptron to obtain feature information of each pedestrian output by the third multi-layer perceptron.

[0109] Step S8, inputting the second state information into a fourth multi-layer perceptron to obtain second feature information of the robot output by the fourth multi-layer perceptron.

[0110] Step S9: input the state information and the second state information of the plurality of pedestrians into the feature stitching layer to obtain the stitching feature information output by the feature stitching layer.

[0111] In some embodiments, the feature splicing layer may include a seventh multi-layer perceptron, an eighth multi-layer perceptron, a ninth multi-layer perceptron, and a splicing unit. The state information of each of the multiple pedestrians is input into the seventh multi-layer perceptron to obtain the feature information of each pedestrian output by the seventh multi-layer perceptron; the second state information is input into the eighth multi-layer perceptron to obtain the third feature information of the robot output by the eighth multi-layer perceptron; the feature information of each pedestrian and the third feature information of the robot are input into the splicing unit for feature splicing, and the spliced ​​features are input into the ninth multi-layer perceptron to obtain spliced ​​feature information.

[0112] The splicing feature information may include the relative feature values ​​between the robot and each pedestrian. If there are N pedestrians in total, the splicing feature information is { V1, V2, …, V N},V i is the relative value of the feature between the robot and pedestrian i.

[0113] Step S10, input the feature information, the second feature information and the spliced ​​feature information of each pedestrian into the second cross-attention layer, and obtain the feature information of the pedestrian that the robot pays attention to output by the second cross-attention layer.

[0114] It can be understood that when considering the interaction between pedestrians and robots, the robot's attention to each pedestrian is different, which is related to the robot's state. In the embodiment of the present invention, based on the second cross attention layer, the interaction between the robot and each pedestrian is captured to obtain the pedestrian feature information that the robot pays attention to.

[0115] In some embodiments, the second cross-attention layer can map the second feature information into a query vector, map the feature information of each pedestrian into a key vector, obtain the robot's attention coefficient for each pedestrian, and perform weighted summation based on the robot's attention coefficient for each pedestrian and the spliced ​​feature information to obtain the feature information of the pedestrians that the robot pays attention to.

[0116] Step S11, inputting the pedestrian feature information that the robot is concerned about into the second feedforward network layer to obtain the second interactive feature information output by the second feedforward network layer.

[0117] The second feed-forward network layer may be a feed-forward network with a residual connection, and the feed-forward network here may have two layers.

[0118] Step 304: input the orientation state information into the robot sub-network to obtain the robot's own feature information output by the robot sub-network.

[0119] In some embodiments, the robot sub-network can be an MLP (Multi-Layer Perceptron) with a ReLU activation function, that is, Figure 4The tenth multi-layer perceptron in the figure. By extracting features from the orientation state information, the robot's own feature information is obtained.

[0120] Step 305: input the first interaction feature information, the second interaction feature information and the robot's own feature information into the fusion sub-network to obtain the action probability distribution and state value output by the fusion sub-network.

[0121] In some embodiments, Figure 4 As shown, the fusion sub-network is used to fuse the first interaction feature information, the second interaction feature information and the robot's own feature information, and perform mapping based on the fused features to obtain the action probability distribution and state value.

[0122] As a possible implementation, the fusion sub-network includes a self-attention layer, a fifth multi-layer perceptron, and a sixth multi-layer perceptron. The first interaction feature information, the second interaction feature information, and the robot's own feature information are input into the fusion sub-network, and the implementation process of obtaining the action probability distribution and state value output by the fusion sub-network may include the following steps.

[0123] Step S12: input the first interaction feature information, the second interaction feature information and the robot's own feature information into the self-attention layer to obtain the global navigation feature information output by the self-attention layer.

[0124] It is understandable that during the robot's navigation, the environment around the robot is constantly changing, and the robot may face different scenarios at different time points. When the robot is performing state assessment and motion planning, it needs to adjust its attention allocation according to the changes in the environment. To this end, the self-attention layer can automatically adjust the relative importance of the first interactive feature information, the second interactive feature information, and the robot's own feature information according to the scenario.

[0125] In some embodiments, the self-attention layer introduces a learnable symbol W, which can capture the relationship between different features. In the self-attention layer, the symbol W is used as the query vector, and the importance coefficient of each feature information is calculated using the dot product operation, and the first interactive feature information, the second interactive feature information and the robot's own feature information are weighted and summed with the corresponding importance coefficients to obtain the global navigation feature information.

[0126] Step S13, inputting the global navigation feature information into the fifth multi-layer perceptron to obtain the action probability distribution output by the fifth multi-layer perceptron.

[0127] Step S14, inputting the global navigation feature information into the sixth multi-layer perceptron to obtain the state value output by the sixth multi-layer perceptron.

[0128] In some embodiments, the global navigation feature information may first be processed by an eleventh multi-layer perceptron, and then passed through a fifth multi-layer perceptron and a sixth multi-layer perceptron respectively to obtain an action probability distribution and a state value.

[0129] Step 306, based on the action probability distribution, determine the prior probability of each action to be selected under the i+1th leaf node, and based on the state value, determine the state value of the i+1th leaf node.

[0130] The robot strategy value model in the embodiment of the present invention realizes the capture of the interaction information between the robot and obstacles, and the interaction information between the robot and pedestrians, and can be adaptively adjusted in different scenarios, so it can be suitable for the navigation action decision of the robot in complex scenarios.

[0131] In order to implement the above embodiment, the present invention also provides a robot navigation action decision device based on Monte Carlo tree search.

[0132] Figure 5 A schematic diagram of the structure of a robot navigation action decision device based on Monte Carlo tree search provided by an embodiment of the present invention. Figure 5 As shown, the device includes: an acquisition module 510, a search module 520 and a determination module 530. The acquisition module 510 is used to acquire the system state information at the current moment; wherein the system state information includes the robot state information, the pedestrian state information of the robot's environment and the obstacle state information; the search module 520 is used to perform a Monte Carlo tree search with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained by training based on the system state information sample; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian and the robot's own feature information based on the system state information, and output the action probability distribution and state value based on the above feature information; the determination module 530 is used to determine the robot navigation action decision result at the current moment based on the number of action selections corresponding to the root node in the search tree.

[0133] In some embodiments, the search module 520 is specifically used to: determine the i+1th starting leaf node and the i+1th action based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, and the number of action selections under each leaf node; wherein i is an integer greater than or equal to 0; when i=0, the leaf node in the i-th leaf node set is the root node; based on the system status information of the i+1th starting leaf node, determine the i+1th system status information after the robot performs the i+1th action; add the i+1th system status information as the i+1th leaf node to the i-th search tree to obtain the i+1th search tree; wherein, when i=0, the i-th search The tree is the root node; the i+1th system state information is input into the robot strategy value model to obtain the state value of the i+1th leaf node and the prior probability of each action to be selected under the i+1th leaf node, and the number of selections and the average reward value of each action to be selected under the i+1th leaf node are initialized; the number of selections corresponding to all selected actions in the path between the root node and the i+1th leaf node is updated, and the average reward value corresponding to all selected actions is updated based on the state value of the i+1th leaf node; the i+1th leaf set in the i+1th search tree is determined, i is set to i+1, and the search is continued until the preset number of searches is reached, and the i+1th search tree is determined as the search tree.

[0134] In some embodiments, the robot strategy value model includes a robot-obstacle interaction subnetwork, a robot-pedestrian interaction subnetwork, a robot subnetwork and a fusion subnetwork; the search module 520 is also used to: determine the first state information of the robot in the robot coordinate system, the second state information and the orientation state information in the world coordinate system based on the i+1th robot state information in the i+1th system state information; input the first state information and the i+1th obstacle state information in the i+1th system state information into the robot-obstacle interaction subnetwork to obtain the first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction subnetwork; input the second state information The information and the i+1th pedestrian state information in the i+1th system state information are input into the robot-pedestrian interaction sub-network to obtain the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction sub-network; the orientation state information is input into the robot sub-network to obtain the robot's own feature information output by the robot sub-network; the first interaction feature information, the second interaction feature information and the robot's own feature information are input into the fusion sub-network to obtain the action probability distribution and state value output by the fusion sub-network; based on the action probability distribution, the prior probability of each action to be selected under the i+1th leaf node is determined, and based on the state value, the state value of the i+1th leaf node is determined.

[0135] As a possible implementation method, the robot-obstacle interaction subnetwork includes a first multi-layer perceptron, a pooling layer, a first cross-attention layer, a first feedforward network layer and a second multi-layer perceptron; the search module 520 is also used to: determine the boundary position information of each of the multiple obstacles based on the i+1th obstacle state information; input the boundary position information of each of the multiple obstacles into the first multi-layer perceptron to obtain the boundary feature information of each obstacle output by the first multi-layer perceptron; input the boundary feature information of each obstacle into the pooling layer to obtain the feature information of each obstacle output by the pooling layer; input the first state information into the second multi-layer perceptron to obtain the first feature information of the robot output by the second multi-layer perceptron; input the feature information and the first feature information of each obstacle into the first cross-attention layer to obtain the obstacle feature information that the robot pays attention to output by the first cross-attention layer; input the obstacle feature information that the robot pays attention to into the first feedforward network layer to obtain the first interaction feature information output by the first feedforward network layer.

[0136] As a possible implementation method, the robot-pedestrian interaction subnetwork includes a third multi-layer perceptron, a feature splicing layer, a fourth multi-layer perceptron, a second cross-attention layer and a second feedforward network layer; the i+1th pedestrian state information includes the state information of multiple pedestrians respectively; the search module 520 is also used to: input the state information of multiple pedestrians respectively into the third multi-layer perceptron, and obtain the feature information of each pedestrian output by the third multi-layer perceptron; input the second state information into the fourth multi-layer perceptron, and obtain the second feature information of the robot output by the fourth multi-layer perceptron; input the state information and the second state information of multiple pedestrians respectively into the feature splicing layer, and obtain the splicing feature information output by the feature splicing layer; input the feature information, the second feature information and the splicing feature information of each pedestrian into the second cross-attention layer, and obtain the feature information of the pedestrians that the robot pays attention to output by the second cross-attention layer; input the feature information of the pedestrians that the robot pays attention to into the second feedforward network layer, and obtain the second interaction feature information output by the second feedforward network layer.

[0137] As a possible implementation method, the fusion subnetwork includes a self-attention layer, a fifth multi-layer perceptron and a sixth multi-layer perceptron; the search module 520 is also used to: input the first interaction feature information, the second interaction feature information and the robot's own feature information into the self-attention layer to obtain the global navigation feature information output by the self-attention layer; input the global navigation feature information into the fifth multi-layer perceptron to obtain the action probability distribution output by the fifth multi-layer perceptron; input the global navigation feature information into the sixth multi-layer perceptron to obtain the state value output by the sixth multi-layer perceptron.

[0138] According to the robot navigation action decision device based on Monte Carlo tree search in an embodiment of the present invention, multiple simulations are performed by means of Monte Carlo tree search to consider various situations that may arise in the future. During the search process, the robot's own feature information and the interaction feature information between the robot and pedestrians and obstacles are combined to perform action probability prediction and state value prediction. Finally, the action decision result is determined based on the search result, thereby improving the accuracy of action decisions and the stability of robot navigation, and realizing the application of robot navigation in scenarios where humans, machines and the environment are integrated.

[0139] It should be noted that the above explanations and descriptions in the embodiment of the robot navigation action decision method based on Monte Carlo tree search can also be applied to the robot navigation action decision device based on Monte Carlo tree search in the embodiment of the present invention, and will not be repeated here.

[0140] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the computer program in the memory 630 to execute the steps of the robot navigation action decision method based on Monte Carlo tree search.

[0141] For example, the method includes: obtaining system status information at the current moment; wherein the system status information includes robot status information, pedestrian status information and obstacle status information of the robot's environment; performing Monte Carlo tree search with the system status information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system status information corresponding to the leaf node is input into the robot policy value model, and action selection and back propagation are performed based on the output result of the robot policy value model; the robot policy value model is obtained based on system status information sample training; the robot policy value model is used to determine the first interaction feature information between the robot and the obstacle, the second interaction feature information between the robot and the pedestrian, and the robot's own feature information based on the system status information, and output the action probability distribution and state value based on the above feature information; based on the number of action selections corresponding to the root node in the search tree, determine the robot navigation action decision result at the current moment.

[0142] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0143] On the other hand, an embodiment of the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the robot navigation action decision method based on Monte Carlo tree search provided in the above-mentioned embodiments.

[0144] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot navigation action decision method based on Monte Carlo tree search provided in the above-mentioned embodiments.

[0145] The non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drives (SSDs)), etc.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot navigation action decision method based on Monte Carlo tree search, characterized in that: include: Acquire the system status information at the current moment; wherein the system status information includes the robot status information, the pedestrian status information and the obstacle status information of the robot's environment; A Monte Carlo tree search is performed with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot strategy value model, and action selection and back propagation are performed based on the output result of the robot strategy value model; the robot strategy value model is obtained by training based on system state information samples; the robot strategy value model includes a robot-obstacle interaction subnetwork, a robot-pedestrian interaction subnetwork, a robot subnetwork and a fusion subnetwork; The robot-obstacle interaction subnetwork is used to output the first interaction feature information between the robot and the obstacle; the robot-pedestrian interaction subnetwork is used to output the second interaction feature information between the robot and the pedestrian; the robot subnetwork is used to output the robot's own feature information; the first interaction feature information, the second interaction feature information and the robot's own feature information are input into the fusion subnetwork to obtain the action probability distribution and state value output by the fusion subnetwork; Based on the number of action selections corresponding to the root node in the search tree, the robot navigation action decision result at the current moment is determined.

2. The method according to claim 1, characterized in that The Monte Carlo tree search is performed with the system status information at the current moment as the root node to obtain a constructed search tree, including: Based on the prior probability and average reward value of the action to be selected under each leaf node in the i-th leaf node set, and the number of action selections under each leaf node, determine the i+1-th starting leaf node and the i+1-th action; wherein i is an integer greater than or equal to 0; when i=0, the leaf node based on the i-th leaf node set is the root node; Determining, based on the system status information of the i+1th starting leaf node, the i+1th system status information after the robot performs the i+1th action; Adding the i+1th system status information as the i+1th leaf node to the i+1th search tree to obtain the i+1th search tree; wherein, when i=0, the i+1th search tree is the root node; Input the i+1th system state information into the robot strategy value model, obtain the state value of the i+1th leaf node, and the prior probability of each action to be selected under the i+1th leaf node, and initialize the number of selections and the average reward value of each action to be selected under the i+1th leaf node; Update the number of selections corresponding to all selected actions in the path from the root node to the i+1th leaf node, and based on the state value of the i+1th leaf node, update the average reward value corresponding to all selected actions; Determine the i+1th leaf set in the i+1th search tree, set i to i+1, continue searching until a preset number of searches is reached, and determine the i+1th search tree as the search tree.

3. The method according to claim 2, characterized in that The step of inputting the i+1th system state information into the robot strategy value model to obtain the state value of the i+1th leaf node and the prior probability of each action to be selected under the i+1th leaf node comprises: Based on the (i+1)th robot state information in the (i+1)th system state information, determining first state information of the robot in the robot coordinate system, second state information and orientation state information in the world coordinate system; Inputting the first state information and the (i+1)th obstacle state information in the (i+1)th system state information into the robot-obstacle interaction sub-network, and obtaining first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction sub-network; Input the second state information and the (i+1)th pedestrian state information in the (i+1)th system state information into the robot-pedestrian interaction subnetwork, and obtain the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork; Inputting the position state information into the robot sub-network to obtain the robot's own feature information output by the robot sub-network; Based on the action probability distribution, the prior probability of each action to be selected under the i+1th leaf node is determined, and based on the state value, the state value of the i+1th leaf node is determined.

4. The method according to claim 3, characterized in that The robot-obstacle interaction subnetwork includes a first multi-layer perceptron, a pooling layer, a first cross-attention layer, a first feedforward network layer, and a second multi-layer perceptron; the first state information and the i+1th obstacle state information in the i+1th system state information are input into the robot-obstacle interaction subnetwork, and the first interaction feature information between the robot and the obstacle output by the robot-obstacle interaction subnetwork is obtained, including: Based on the (i+1)th obstacle state information, determining boundary position information of each of the plurality of obstacles; Inputting the boundary position information of each of the plurality of obstacles into the first multi-layer perceptron to obtain the boundary feature information of each obstacle output by the first multi-layer perceptron; Inputting boundary feature information of each obstacle into the pooling layer to obtain feature information of each obstacle output by the pooling layer; Inputting the first state information into the second multi-layer perceptron to obtain first feature information of the robot output by the second multi-layer perceptron; Inputting the feature information of each obstacle and the first feature information into the first cross attention layer, and obtaining the obstacle feature information that the robot pays attention to output by the first cross attention layer; The obstacle feature information that the robot is concerned about is input into the first feedforward network layer to obtain the first interactive feature information output by the first feedforward network layer.

5. The method according to claim 3, characterized in that: The robot-pedestrian interaction subnetwork includes a third multi-layer perceptron, a feature concatenation layer, a fourth multi-layer perceptron, a second cross attention layer, and a second feedforward network layer; the i+1th pedestrian state information includes state information of a plurality of pedestrians; the second state information and the i+1th pedestrian state information in the i+1th system state information are input into the robot-pedestrian interaction subnetwork, and the second interaction feature information between the robot and the pedestrian output by the robot-pedestrian interaction subnetwork is obtained, including: Inputting the state information of each of the plurality of pedestrians into the third multi-layer perceptron to obtain feature information of each of the pedestrians output by the third multi-layer perceptron; Inputting the second state information into the fourth multi-layer perceptron to obtain second feature information of the robot output by the fourth multi-layer perceptron; Inputting the state information of each of the plurality of pedestrians and the second state information into the feature stitching layer to obtain stitching feature information output by the feature stitching layer; Inputting the feature information of each pedestrian, the second feature information and the spliced ​​feature information into the second cross attention layer, and obtaining the feature information of the pedestrian that the robot pays attention to output by the second cross attention layer; The pedestrian feature information that the robot is concerned about is input into the second feedforward network layer to obtain the second interactive feature information output by the second feedforward network layer.

6. The method according to claim 3, characterized in that The fusion sub-network includes a self-attention layer, a fifth multi-layer perceptron, and a sixth multi-layer perceptron; the first interaction feature information, the second interaction feature information, and the robot's own feature information are input into the fusion sub-network to obtain the action probability distribution and state value output by the fusion sub-network, including: Inputting the first interaction feature information, the second interaction feature information and the robot's own feature information into the self-attention layer to obtain the global navigation feature information output by the self-attention layer; Inputting the global navigation feature information into the fifth multi-layer perceptron to obtain the action probability distribution output by the fifth multi-layer perceptron; The global navigation feature information is input into the sixth multi-layer perceptron to obtain the state value output by the sixth multi-layer perceptron.

7. A robot navigation action decision device based on Monte Carlo tree search, characterized in that: include: An acquisition module, used to acquire the system status information at the current moment; wherein the system status information includes the robot status information, the pedestrian status information and the obstacle status information of the robot's environment; A search module is used to perform a Monte Carlo tree search with the system state information at the current moment as the root node to obtain a constructed search tree; wherein, during the search process, the system state information corresponding to the leaf node is input into the robot strategy value model, and action selection and back propagation are performed based on the output result of the robot strategy value model; the robot strategy value model is obtained by training based on system state information samples; the robot strategy value model includes a robot-obstacle interaction subnetwork, a robot-pedestrian interaction subnetwork, a robot subnetwork and a fusion subnetwork; The robot-obstacle interaction subnetwork is used to output the first interaction feature information between the robot and the obstacle; the robot-pedestrian interaction subnetwork is used to output the second interaction feature information between the robot and the pedestrian; the robot subnetwork is used to output the robot's own feature information; the first interaction feature information, the second interaction feature information and the robot's own feature information are input into the fusion subnetwork to obtain the action probability distribution and state value output by the fusion subnetwork; a determination module is used to determine the robot's navigation action decision result at the current moment based on the number of action selections corresponding to the root node in the search tree.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the robot navigation action decision method based on Monte Carlo tree search described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the robot navigation action decision method based on Monte Carlo tree search as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the robot navigation action decision method based on Monte Carlo tree search according to any one of claims 1 to 6 is implemented.

Images