Robot navigation model construction and robot navigation method, apparatus and device

Abstract

The present application relates to the field of electronic technology, disclose a robot navigation model construction and robot navigation method, device and equipment, the method provided by the present application, the local observation information of each robot is converted into the graph structure of the corresponding robot, the node features of each neighbor node of the corresponding robot are extracted through the graph structure of each robot, the node of each robot corresponding to different neighbor nodes is aggregated by using the pre-constructed graph neural network model, the neighbor node aggregation information of the corresponding robot is obtained, the local observation information of each robot and the neighbor node aggregation information are used as the first state of the corresponding robot, the preset multi-agent reinforcement learning model is optimized and trained by using the first state corresponding to different robots respectively, the preset action space and the preset reward function, until the preset condition is met, the robot navigation model of each robot is obtained. The robot navigation model constructed can accurately plan the robot navigation route.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronics, in particular to a robot navigation model construction and a robot navigation method, device and equipment. BACKGROUND

[0002] Multi-robot target navigation, as a core field of robot research, aims to plan a conflict-free path for robots from the initial position to the target position, and ensure the balance of the load while completing the navigation, so as to improve the overall work efficiency. In recent years, multi-robot target navigation has been widely applied to automatic driving, disaster rescue and logistics and other practical scenes. However, it is still a challenge to calculate a solution that balances between optimality and efficiency in real-world scenarios.

[0003] In the related art, path planning is usually divided into two categories: offline planning and online planning. Offline planning assumes that obstacles in the environment are static and fully known. In this scenario, traditional path planning algorithms directly plan the path of the robot to reach the target position, and the robot is navigated based on the planned path. These algorithms can be directly applied without complex training, but they face high computational overhead in actual deployment. In path planning, all environmental information and the position information of other moving objects need to be obtained in advance, which leads to insufficient generalization performance of these traditional path planning methods and makes it difficult to adapt to complex cooperation scenarios. In online path planning scenarios, the robot perceives dynamic obstacles near it and incomplete information about the environment, and plans a path based on the perceived dynamic obstacle information and partial environmental information. The robot is navigated based on the planning result. Since the perception range of the robot is limited, the path planned based on the perceived information is not the optimal path, and subsequent navigation based on the planned path reduces the efficiency of the robot's task execution. Therefore, there is an urgent need for a robot navigation method that has a wide range of applications and can accurately plan a path. SUMMARY

[0004] Therefore, the present application provides a robot navigation model construction and a robot navigation method, device and equipment to solve the problem that the path planning method in the related art has a small range of applications and is not accurate enough, and cannot well implement path navigation.

[0005] In a first aspect, the present application provides a robot navigation model construction method, which comprises: obtaining local observation information of a plurality of robots, the local observation information of each robot being used to represent position information of the corresponding robot, target position information of the corresponding robot, and position information of a plurality of other objects in the observation field of view, the other objects including obstacles and other robots in the observation field of view; taking the corresponding robot in the local observation information of each robot as an agent node i, taking each other object as a neighbor node j, and taking the distance between the agent node i and each neighbor node j as an edge to construct a graph structure of the corresponding robot; determining node features corresponding to different neighbor nodes j of the corresponding robot based on the graph structure of each robot, the node features being used to represent relative position information between the neighbor node j and the agent node i and relative position information between the target position of the neighbor node j and the agent node i; inputting the node features corresponding to different neighbor nodes j of each robot into a pre-constructed graph neural network for feature aggregation to obtain neighbor node aggregation information of the corresponding robot; taking the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and using the first state corresponding to different robots, a preset action space, and a preset reward function to optimize and train a preset multi-agent reinforcement learning model until a preset condition is met to obtain a robot navigation model of each robot.

[0006] The robot navigation model construction method provided by the application, by taking the corresponding robot in the local observation information of each robot as an agent node i, each other object as a neighbor node j, and the distance between the agent node i and each neighbor node j as an edge, a graph structure of the corresponding robot is constructed, the node features of each neighbor node of the corresponding robot are extracted through the graph structure of each robot, the graph neural network model constructed in advance is used to aggregate the nodes corresponding to different neighbor nodes of each robot, the neighbor node aggregation information of the corresponding robot is obtained, the local observation information and the neighbor node aggregation information of each robot are taken as the first state of the corresponding robot, the preset multi-agent reinforcement learning model is optimized and trained by using the first state corresponding to different robots, the preset action space and the preset reward function, until the preset condition is met, and the robot navigation model of each robot is obtained. The method provided by the application uses local observation information when constructing the model, reflects the perception limitation in real robot operation, improves the generalization ability of the system in different scale environments, and reduces the dimension of neural network input. The graph neural network model is used to aggregate the nodes corresponding to different neighbor nodes of each robot, so as to better capture the structure and interaction in the multi-agent system, and the local observation information and the neighbor node aggregation information of each robot are taken as the first state of the corresponding robot for subsequent multi-agent reinforcement learning model training, so that the model can dynamically adapt to the change of the number of entities in the environment, which helps to improve the path planning effect of the model. The robot navigation model constructed can realize the navigation path planning of the robot, has a wide range of applications, and the planned navigation path is more accurate.

[0007] In an optional implementation, the steps of obtaining the robot navigation model of each robot by using the first state of the corresponding robot, the preset action space, and the preset reward function to optimize and train the preset multi-robot reinforcement learning model until a preset condition is met, include: obtaining the policy network and the critic network of each robot; initializing the policy network and the critic network of each robot; inputting the first state of each robot into the corresponding policy network to make the corresponding policy network output the first action of the corresponding robot; determining the second state and the reward value of the corresponding robot after the interaction with the environment based on the first action of each robot and the preset reward function; obtaining the global state information of the corresponding robot by aggregating the first states of different robots; inputting the global state information of each robot into the critic network of the corresponding robot to obtain the first state value of the corresponding robot; storing the first state, the second state, the first action, the first state value, and the reward value of each robot as experience data into an experience pool; if t is less than the maximum time step T, taking the second state as the first state and returning to the step of setting t = t + 1 until t = T to obtain a target experience pool; training the policy network and the critic network of each robot by using the experience data in the target experience pool until the maximum training times are reached to obtain the robot navigation model of the corresponding robot.

[0008] In an optional implementation, the steps of training the policy network and the critic network of each robot by using the experience data in the target experience pool until the maximum training times are reached to obtain the robot navigation model of the corresponding robot include: determining the target action sequence and the local observation information sequence of each robot to reach the target position based on a preset path search algorithm; training the policy network and the critic network of each robot by using the target action sequence and the local observation information sequence of different robots, the experience data in the target experience pool, and the imitation learning method until the maximum training times are reached to obtain the robot navigation model of the corresponding robot.

[0009] In an optional implementation, the steps of obtaining the global state information of the corresponding robot by aggregating the first states of different robots include: splicing the local observation information in the first state of different robots to obtain global observation information; performing mean pooling processing on the neighbor node aggregation information in the first state of different robots to obtain global node features; and determining the global state information based on the global observation information and the global node features.

[0010] In a second aspect, the present application provides a robot navigation method, comprising: obtaining a first state of a target robot and a robot navigation model of the target robot, the robot navigation model being constructed by the robot navigation model construction method of the first aspect or any of the corresponding embodiments thereof, and the first state of the target robot being determined by local observation information of the target robot; inputting the first state information of the target robot into the robot navigation model of the target robot, so that the robot navigation model outputs a target action sequence; and performing a navigation operation on the target robot based on the target action sequence.

[0011] The robot navigation method provided by the present application can effectively improve the accuracy of the navigation result by obtaining the first state of the target robot, inputting the first state into the pre-constructed robot navigation model, and making the model output a target action sequence, and the navigation path obtained by the target robot through the execution of the target action sequence is the optimal navigation path.

[0012] In a third aspect, the present application provides a robot navigation model construction device, comprising: a first obtaining module configured to obtain local observation information of a plurality of robots, wherein the local observation information of each robot is used to represent position information of the corresponding robot, target position information of the corresponding robot, and position information of a plurality of other objects in an observation field of view, and the other objects include obstacles and other robots in the observation field of view; a construction module configured to construct a graph structure of each robot by taking the corresponding robot in the local observation information of each robot as an agent node i, taking each other object as a neighbor node j, and taking the distance between the agent node i and each neighbor node j as an edge; a first determining module configured to determine node features corresponding to different neighbor nodes j of each robot based on the graph structure of each robot, wherein the node features are used to represent relative position information between the neighbor node j and the agent node i, and relative position information between the target position of the neighbor node j and the agent node i; an aggregation module configured to input the node features corresponding to different neighbor nodes j of each robot into a pre-constructed graph neural network for feature aggregation, to obtain neighbor node aggregation information of the corresponding robot; and a training module configured to take the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and to optimize and train a pre-set multi-agent reinforcement learning model by using the first state corresponding to different robots, a pre-set action space, and a pre-set reward function, until a pre-set condition is met, to obtain a robot navigation model of each robot.

[0013] In a fourth aspect, the present application provides a robot navigation device, comprising: a second acquisition module, configured to acquire a first state of a target robot and a robot navigation model of the target robot, the robot navigation model being constructed by the robot navigation model construction method according to the first aspect or any one of the corresponding embodiments thereof, and the first state of the target robot being determined by local observation information of the target robot; a second determination module, configured to input the first state information of the target robot into the robot navigation model of the target robot, so as to make the robot navigation model output a target action sequence; and a navigation module, configured to perform a navigation operation on the target robot based on the target action sequence.

[0014] In a fifth aspect, the present application provides a computer device, comprising: a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions; the processor executes the computer instructions, thereby executing the robot navigation model construction method according to the first aspect or any one of the corresponding embodiments thereof, or executing the robot navigation method according to the second aspect.

[0015] In a sixth aspect, the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make a computer execute the robot navigation model construction method according to the first aspect or any one of the corresponding embodiments thereof, or execute the robot navigation method according to the second aspect.

[0016] In a seventh aspect, the present application provides a computer program product, which comprises computer instructions, and the computer instructions are used to make a computer execute the robot navigation model construction method according to the first aspect or any one of the corresponding embodiments thereof, or execute the robot navigation method according to the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a robot navigation model construction method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a specific example of local observation information of a robot in an embodiment of the present application;

[0020] Figure 3 is a flowchart of another robot navigation model construction method according to an embodiment of the present application;

[0021] Figure 4 is a specific example of a model training method combining deep reinforcement learning algorithm and imitation learning method in the embodiment of the present application;

[0022] Figure 5 is a flowchart of a robot navigation method according to an embodiment of the present application;

[0023] Figure 6 is a structural block diagram of a robot navigation model construction device according to an embodiment of the present application;

[0024] Figure 7 is a structural block diagram of a robot navigation device according to an embodiment of the present application;

[0025] Figure 8 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0027] In related technologies, path planning is generally divided into two categories: offline planning and online planning. Offline planning assumes that obstacles in the environment are static and completely known. In this scenario, traditional path planning algorithms directly plan a path for the robot to reach the target position, and the robot is navigated based on the planned path. These algorithms can be directly applied without complex training, but they face high computational overhead in actual deployment. In path planning, all environmental information and the position information of other moving objects need to be obtained in advance, which leads to insufficient generalization performance of these traditional path planning methods and makes it difficult to adapt to complex cooperative scenarios. In online path planning scenarios, the robot perceives dynamic obstacles and incomplete information about the environment in its vicinity, and plans a path based on the perceived dynamic obstacle information and partial environmental information. The robot is navigated based on the planning result. Since the perception range of the robot is limited, the path planned based on the perceived information is not the optimal path, and subsequent navigation based on the planned path reduces the efficiency of the robot's task execution.

[0028] A promising solution is to combine offline planning with online planning, in which case the offline path planner is responsible for revising or optimizing the trajectory proposed by the global planner. For example, some researchers proposed a global dynamic window approach that combines path planning with real-time obstacle avoidance, enabling robots to move at high speed, goal-oriented, and reactive in unknown and dynamic environments. However, their approach can result in highly suboptimal paths. Some researchers also proposed a multi-strategy decision to achieve autonomous navigation in dynamic social environments. However, in their work, the trajectory of the robot is selected from a closed-loop behavior with predictable utility, rather than being explicitly planned.

[0029] Meanwhile, as the number of robots increases when a large-scale multi-robot system expands, the state space and joint observation space grow exponentially, and centralized methods require a central unit to collect data from all robots to organize the optimal path for each robot, which can result in a large demand for computing resources. As the system scales, decentralized methods gradually gain attention. In these methods, each robot estimates or communicates the future trajectory of other robots through broadcast or distance-based communication. However, when multiple robots communicate simultaneously, redundant information can be generated, which increases the computational burden of the system and negatively impacts the overall team performance. In addition, due to bandwidth limitations, large amounts of data, and environmental interference, reliable and continuous communication is often affected. Ensuring the convergence of the negotiation process is also a challenge in a fully decentralized framework without prior planning. These limitations can adversely affect the optimality of the solution found and the resilience of the team in dealing with disturbances.

[0030] Therefore, the robot navigation model construction method provided by the embodiments of the present application can be applied to a server to realize the construction of a robot navigation model. The method provided by the embodiments of the present application uses local observation information when constructing the model, reflects the perception limitations in real robot operations, improves the generalization ability of the system in different scale environments, and reduces the dimensionality of the neural network input. The graph neural network model is used to aggregate the nodes corresponding to different neighbor nodes of each robot to better capture the structure and interaction in the multi-agent system. The local observation information of each robot and the neighbor node aggregation information are used as the first state of the corresponding robot for subsequent multi-agent reinforcement learning model training, so that the model can dynamically adapt to changes in the number of entities in the environment, which helps to improve the path planning effect of the model. The robot navigation model constructed by the method can be used to realize the navigation path planning of the robot, has a wide range of applications, and the planned navigation path is more accurate.

[0031] According to an embodiment of the present application, a robot navigation model construction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0032] In this embodiment, a robot navigation model construction method is provided, which can be used in the server Figure 1 The flowchart of the robot navigation model construction method according to an embodiment of the present application is shown in Figure 1 The flowchart includes the following steps:

[0033] In step S101, local observation information of a plurality of robots is obtained, and the local observation information of each robot is used to represent the position information of the corresponding robot, the target position information of the corresponding robot, and the position information of a plurality of other objects in the observation field of view, the other objects including obstacles and other robots in the observation field of view.

[0034] For example, the robot can be a robot that needs to be navigated centrally, and the local observation information o i may be the environmental information within the limited field of view around the robot that the robot can perceive, which can include but is not limited to a 10x10 field of view (FOV). In the present application, in the case of a complete environment map, by setting an appropriate FOV, the robot can have complete observability in the system. In addition, the fixed FOV also allows the strategy to be generalized in different scale environments, reducing the dimensionality of the input to the neural network. In the limited FOV, the observation data is transmitted in the form of a binary matrix, including the positions of obstacles, other agents, their respective target positions, and the optimal guiding path generated by the A-star algorithm. In the limited FOV, the observation data is transmitted in the form of a binary matrix, including the positions of obstacles, other agents, their respective target positions, and the shortest path generated by the A-star algorithm. As shown in Figure 2 C is the current agent, B is the other agent, D is the target, and A is the obstacle. Specifically, the other objects can also include the target of the robot, which can be understood as the final position that the robot needs to reach. "1" indicates that the corresponding cell represents the binary transmission matrix of obstacles, the positions of other agents, their respective target positions, and the optimal guiding path generated by the A-star algorithm.

[0035] In step S102, the corresponding robot in the local observation information of each robot is taken as a proxy node i, each other object is taken as a neighbor node j, and the distance between the proxy node i and each neighbor node j is taken as an edge to construct a graph structure for the corresponding robot.

[0036] Exemplarily, the distance between the agent node i and each neighbor node j can include but is not limited to Manhattan distance. In the embodiments of the present application, in the environment, each object (entity) is defined as an agent, an obstacle or a target, and the agent is obtained according to the abstraction of the robot, and the agent and the robot are one-to-one corresponding. A graph structure of an agent entity graph of each agent i at each time t is defined where each node v∈V is an entity in the environment. If the distance between an agent and other entities is within the perception range, there is an edge e∈E. The edge between the agents is bidirectional, while the edge between the agent and the non-agent is unidirectional, that is, information can only be transmitted from the non-agent entity to the agent. The unidirectional edge is equivalent to the agent perceiving the state of a neighboring entity, while the bidirectional edge is equivalent to the communication channel between the agents. Each edge e ij has an associated edge feature representing the Manhattan distance between the agent i and the entity j.

[0037] In step S103, the node features corresponding to different neighbor nodes j of the corresponding robot are determined based on the graph structure of each robot, and the node features are used to represent the relative position information between the neighbor node j and the agent node i and the relative position information between the target position of the neighbor node j and the agent node i.

[0038] Exemplarily, in the embodiments of the present application, the local observation information o i of each robot is a binary matrix, and each node j on the graph structure g i has a feature wherein, is the relative position information between the neighbor node j and the agent node i, is the relative position information between the target position of the neighbor node j and the agent node i, and entity type(j) is the node type of the neighbor node j, which can include but is not limited to obstacles, robots, etc. If the node j corresponds to an (static / dynamic) obstacle or target, it is set to

[0039] In step S104, the node features corresponding to different neighbor nodes j of each robot are input into a pre-constructed graph neural network for feature aggregation to obtain the neighbor node aggregation information of the corresponding robot.

[0040] Exemplarily, in order to infer the local field information around each robot i, a pre-constructed graph neural network model is used to aggregate the node features of different neighbor nodes corresponding to each robot. In the embodiment of the present application, in order to better realize the aggregation of node features of neighbor nodes, a graph attention mechanism is introduced on the basis of a graph neural network (GNN) combined with a unified message passing model (UniMP) of a message passing framework. Through the attention mechanism, the robot can dynamically adjust the priority of information processing, thereby optimizing the multi-robot cooperation and path planning effect. The following relationship is defined by the pre-constructed graph neural network to aggregate the information of the neighbor nodes of each robot:

[0041] x i ′=W1·x i +∑ j∈N(i) α i,j W2·x j

[0042] wherein x i ′ represents the neighbor node aggregation information of the robot i, N i represents a set of nodes connected to the agent node i corresponding to the robot, W1 and W2 are learnable weight matrices, and α i,j represents an attention coefficient.

[0043] α i,j is calculated by the following formula:

[0044]

[0045] wherein e ij is the feature of the edge connecting nodes i and j, c is the output dimension of the specific layer, and W3, W4 and W5 are weight matrices.

[0046] In step S105, the local observation information and the neighbor node aggregation information of each robot are taken as the first state of the corresponding robot. The preset multi-agent reinforcement learning model is optimized and trained by using the first state corresponding to different robots, the preset action space and the preset reward function, until the preset condition is met, and the robot navigation model of each robot is obtained.

[0047] Exemplarily, the preset multi-agent reinforcement learning model can include but is not limited to a multi-agent reinforcement learning algorithm (MAPPO). The local observation information and the neighbor node aggregation information of each robot can be spliced to obtain a vector As a first state. In the embodiments of the application, in the grid environment, the actions of the robots are discrete, and one unit can be selected to move in four directions or remain stationary. Certain actions can be invalid at each time step, such as trying to move to a wall or the location of another agent. During training, the robot only samples from valid actions, and to avoid the agent converging to an oscillatory policy, it is limited to returning to the last occupied location for a number of time steps, thereby encouraging effective policy learning. The reward function follows the general logic of the grid world: the agent (robot) will be penalized (-0.01) for each time step it does not reach the goal, to encourage it to reach the goal as quickly as possible. An agent that remains stationary will face a higher penalty (-0.05) than an agent that moves, thereby encouraging exploration behavior. Although random sequential action during training can filter out some invalid actions, the agent can still collide and therefore suffer a -0.4 penalty. Successfully completing a set (i.e., all agents reaching the goal at the same time) will give a +2 reward.

[0048] The general flow of the MAPPO algorithm is as follows:

[0049] 1. Initialization: Initialize the policy network and value function network for each agent. The structure of these networks should be flexible enough to adapt to multiple action space types.

[0050] 2. Experience sharing: Agents collect experience data by interacting with the environment and store this data in a shared experience pool for other agents to access and learn from.

[0051] 3. Centralized-distributed training: Perform centralized-distributed training. Agents perform training locally, but can share global information to facilitate better collaborative learning.

[0052] 4. Calculate advantage: Calculate the advantage function for each agent using the value function network, which measures the superiority of the action selected relative to the average level.

[0053] 5. Perform PPO update: Update the policy network for each agent. Use the PPO update rule to maximize the advantage function of the experience and introduce a clipping term to limit the magnitude of a single update.

[0054] 6. Loop iteration: Repeat steps 2 to 5 until a satisfactory performance level or the number of training times is reached.

[0055] In the embodiments of the application, the policy network can be an actor network, and the value function network can be a critic network.

[0056] The robot navigation model construction method provided in the embodiment uses local observation information in model construction, reflects the perception limitations in real robot operation, improves the generalization ability of the system in different scale environments, and reduces the dimension of the neural network input. The node corresponding to each robot and different neighbor nodes is aggregated by using a graph neural network model, so as to better capture the structure and interaction in the multi-agent system. The local observation information of each robot and the aggregated information of the neighbor nodes are used as the first state of the corresponding robot for subsequent multi-agent reinforcement learning model training, so that the model can dynamically adapt to the change of the number of entities in the environment, which helps to improve the path planning effect of the model. The robot navigation model constructed by using the robot navigation model can realize the navigation path planning of the robot, has a wide range of applications, and the planned navigation path is more accurate.

[0057] In the embodiment, a robot navigation model construction method is provided, which can be used in the server described above, Figure 3 is a flowchart of the robot navigation model construction method according to the embodiment of the application, as shown in the figure, the flow includes the following steps: Figure 3

[0058] In step S301, the local observation information of a plurality of robots is obtained, and the local observation information of each robot is used to represent the position information of the corresponding robot, the target position information of the corresponding robot, and the position information of a plurality of other objects in the observation field of view, the other objects including obstacles and other robots in the observation field of view. For details, please refer to step S101 of the embodiment shown in Figure 1 , which will not be repeated here.

[0059] In step S302, the corresponding robot in the local observation information of each robot is taken as an agent node i, each other object is taken as a neighbor node j, and the distance between the agent node i and each neighbor node j is taken as an edge, and the graph structure of the corresponding robot is constructed. For details, please refer to step S102 of the embodiment shown in Figure 1 , which will not be repeated here.

[0060] In step S303, the node features corresponding to different neighbor nodes j of each robot are determined based on the graph structure of each robot, and the node features are used to represent the relative position information between the neighbor node j and the agent node i and the relative position information between the target position of the neighbor node j and the agent node i. For details, please refer to step S103 of the embodiment shown in Figure 1 , which will not be repeated here.

[0061] In step S304, the node features corresponding to different neighbor nodes j of each robot are input into the pre-constructed graph neural network for feature aggregation, and the neighbor node aggregation information of the corresponding robot is obtained. For details, please refer to step S104 of the embodiment shown in Figure 1 , which will not be repeated here.​

[0062] In step S305, the local observation information and the neighbor node aggregation information of each robot are taken as the first state of the corresponding robot, and a preset multi-agent reinforcement learning model is optimized and trained by using the first state corresponding to each robot, a preset action space, and a preset reward function until a preset condition is met, so as to obtain a robot navigation model of each robot.

[0063] Specifically, the above step S305 includes:

[0064] In step S3051, a policy network and a critic network of each robot are obtained.

[0065] For example, in the embodiment of the present application, the policy network can include but is not limited to an actor network.

[0066] In step S3052, t is set to 0, and the policy network and the critic network of each robot are initialized.

[0067] For example, at time step 0, the network parameters of the policy network and the critic network of each robot are initialized.

[0068] In step S3053, t is set to t+1, and the first state of each robot is input into the corresponding policy network, so that the corresponding policy network outputs the first action of the corresponding robot.

[0069] For example, the first state s of the robot is input into the corresponding policy network, and the policy network outputs the first action a of the corresponding robot.

[0070] In step S3054, the second state and the reward value of the corresponding robot after interacting with the environment are determined based on the first action of each robot and a preset reward function.

[0071] For example, in the embodiment of the present application, by executing the first action a, the second state s ′ of each robot and the reward value of the action can be obtained.

[0072] In step S3055, the global state information of the corresponding robot is obtained by aggregating the first state corresponding to each robot.

[0073] In some optional embodiments, the above step S3055 includes:

[0074] In step a1, the local observation information in the first state corresponding to each robot is spliced to obtain global observation information.

[0075] For example, in the embodiment, the splicing manner of the local observation information is not limited, as long as splicing can be achieved.

[0076] Step a2, different robots correspond to the mean pooling processing of the neighbor node aggregation information in the first state, to get the global node feature.

[0077] Exemplarily, in the embodiment of the application, after introducing the graph information aggregation module, the updating process of MAPPO is also adjusted accordingly to adapt to this new information expression method. In the traditional MAPPO, the critic usually relies on the state-action pair of a single agent as input to estimate the value function. However, in the multi-agent environment, especially when using the centralized training, distributed execution (CTDE) framework, the interdependence between agents is very strong. Therefore, the traditional splicing method cannot fully consider the relationship between agents and its dynamic changes. In order to make up for this deficiency, graph neural network (GNN) is used to aggregate the information of neighbor agents to better capture the structure and interaction in the multi-agent system.

[0078] The global state representation is generated by aggregating the information of each agent's neighbor nodes. The feature of each node is updated through the interaction with the neighbor nodes, and this process is realized through the graph neural network. In this way, a more comprehensive global environment representation can be obtained, thereby enhancing the evaluation ability of the critic network on the behavior of each agent. In order to further simplify the processing of information flow, a global mean pooling operator is used to aggregate the neighbor node features of all agents to obtain a fixed-dimensional vector representation, as follows:

[0079]

[0080] where X agg represents the global node feature, N represents the number of robots, represents the neighbor node aggregation information of robot i.

[0081] Step a3, determine the global state information based on the global observation information and the global node feature.

[0082] Exemplarily, in the embodiment of the application, the global observation information and the global node feature are spliced to obtain the global state information, and the splicing method is not limited in the embodiment of the application, and can be determined according to the needs of those skilled in the art.

[0083] Step S3056, input the global state information of each robot into the critic network corresponding to the robot to obtain the first state value of the corresponding robot.

[0084] Exemplarily, the global state information is input into the critic network, and the critic network outputs the value of the first state.

[0085] Step S3057, the first state, the second state, the first action, the first state value and the reward value of each robot are stored into the experience pool as experience data.

[0086] Step S3058, if t is less than the maximum time step T, the second state is taken as the first state, and the step of returning t=t+1 is returned until t=T, and the target experience pool is obtained.

[0087] Exemplarily, the specific content of the maximum time step T is not limited in the embodiments of the present application, and can be determined according to the needs of those skilled in the art.

[0088] Step S3059, the policy network and the critic network of each robot are trained by using the experience data in the target experience pool until the maximum training times are reached, and the robot navigation model corresponding to the robot is obtained. Exemplarily, the policy network and the critic network of each robot are trained, and the network parameters are updated until the maximum training times are reached or the model performance meets the needs, and the robot navigation model of the robot is obtained.

[0089] Exemplarily, in the embodiments of the present application, the updating process of MAPPO continues to use the objective function in PPO for optimization. For each agent, the goal is to maximize the expected return of its policy while maintaining the stability of policy updates. MAPPO uses the “clipped” objective function to limit the magnitude of policy updates, ensuring that the training process will not be unstable due to excessive policy changes. Specifically, the objective function is as follows:

[0090]

[0091] wherein, represents the loss function of the actor network corresponding to the robot i, E t represents the expectation, and θ represents the parameters of the actor network, is the probability ratio of agent i at time step t, is the advantage function; ∈ is a hyperparameter for controlling the change amplitude of the policy at each update.

[0092] In order to further improve the convergence speed of the policy and the stability of the training, the update of the value function is also adopted. The value function of each agent is optimized by minimizing the temporal difference error, and the goal is to minimize the gap between the predicted state value and the actual return as much as possible, and the updating process is as follows:

[0093]

[0094] wherein, represents the loss function of the Critic network, represents the value output by the Critic network of the robot i under the condition that the input is s t t represents the first state of the robot at time t, represents the predicted state value calculated by the time difference (TD) error.

[0095] In some optional embodiments, the step S3059 comprises:

[0096] Step b1, determining the target action sequence and the local observation information sequence of each robot reaching the target position based on a preset path search algorithm.

[0097] Exemplarily, the preset path search algorithm can include but is not limited to A-star algorithm, which is an efficient path search algorithm combining the advantages of best-first search and shortest path search. In the embodiments of the present application, the target action sequence and the local observation information sequence of each robot reaching the target position are dynamically generated by the A-star algorithm.

[0098] Step b2, training the policy network and the critic network of each robot by using the target action sequence and the local observation information sequence corresponding to different robots, the experience data in the target experience pool and the imitation learning method until the maximum training times are reached, to obtain the robot navigation model corresponding to the robot.

[0099] Exemplarily, in the embodiments of the present application, the deep reinforcement learning algorithm can also be combined with the imitation learning method to speed up the training, provide more stable training and higher quality solutions. During the training process, it is randomly selected whether to involve reinforcement learning or imitation learning at the beginning of each training round. These demonstration data are dynamically generated by the A-star algorithm, and the state observation and action sequence are saved for each agent. The schematic diagram of the model training process is shown in Figure 4 The expert policy refers to the target action sequence composed of each expert action planned by the A-star algorithm. In the imitation learning process, the policy is optimized by minimizing the behavior cloning loss, and the loss function is represented by the following formula:

[0100]

[0101] Where L bc represents the cross-entropy loss function, T represents the total time step, a t represents the action of the robot at time t, o t represents the local observation information at time t, and θ represents the model parameter, and π represents the policy.

[0102] ​In this embodiment, the method combining imitation learning and reinforcement learning accelerates the training process and improves the strategy stability. Imitation learning helps the agent quickly enter a high-quality state space, and reinforcement learning further optimizes the strategy, significantly improving the learning efficiency and overall performance of the system.

[0103] In this embodiment, a robot navigation method is also provided, which can be used in the server, Figure 5 is a flowchart of the robot navigation method according to an embodiment of the application, as shown in Figure 5 , the flow includes the following steps:

[0104] Step S501, obtaining the first state of the target robot and the robot navigation model of the target robot, the robot navigation model is constructed by the robot navigation model construction method in the above embodiment, and the first state of the target robot is determined by the local observation information of the target robot.

[0105] For example, in the embodiment of the present application, the first state of the target robot is determined by the local observation information of the target robot, and the specific determination process can refer to the description of related contents in the embodiment shown in Figure 1 , which will not be repeated here.

[0106] Step S502, inputting the first state information of the target robot into the robot navigation model of the target robot, so that the robot navigation model outputs a target action sequence.

[0107] For example, the target action sequence is an action sequence that enables the target robot to complete the task target.

[0108] Step S503, performing navigation operation on the target robot based on the target action sequence.

[0109] For example, by controlling the target robot to execute the target action sequence, the target robot can reach the target position with the optimal path, thereby completing the task target.

[0110] The robot navigation method provided in this embodiment, by obtaining the first state of the target robot, inputting the first state into the pre-constructed robot navigation model, so that the model inputs the target action sequence, the navigation path obtained by the target robot executing the target action sequence is the optimal navigation path, effectively improving the navigation result accuracy information.

[0111] A robot navigation model construction apparatus is also provided in the present embodiment, which is used to implement the above-described embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0112] The present embodiment provides a robot navigation model construction apparatus, as shown in Figure 6 comprises:

[0113] The first acquisition module 601 is configured to acquire local observation information of a plurality of robots, wherein the local observation information of each robot is used to represent position information of the corresponding robot, target position information of the corresponding robot, and position information of a plurality of other objects in an observation field of view, the other objects including obstacles and other robots in the observation field of view;

[0114] The construction module 602 is configured to construct a graph structure of each robot by taking the corresponding robot in the local observation information of each robot as an agent node i, taking each other object as a neighbor node j, and taking a distance between the agent node i and each neighbor node j as an edge;

[0115] The first determination module 603 is configured to determine node features corresponding to different neighbor nodes j of the corresponding robot based on the graph structure of each robot, wherein the node features are used to represent relative position information between the neighbor node j and the agent node i, and relative position information between a target position of the neighbor node j and the agent node i;

[0116] The aggregation module 604 is configured to input the node features corresponding to the different neighbor nodes j of each robot into a pre-constructed graph neural network to perform feature aggregation, to obtain neighbor node aggregation information of the corresponding robot;

[0117] The training module 605 is configured to use the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and use the first state corresponding to different robots, a preset action space, and a preset reward function to perform optimization training on a preset multi-agent reinforcement learning model until a preset condition is met, to obtain a robot navigation model of each robot.

[0118] In some optional embodiments, the training module 605 comprises:

[0119] The acquisition sub-module is configured to acquire a policy network and a critic network of each robot;

[0120] The initialization sub-module is configured to set t = 0, and initialize the policy network and the critic network of each robot;

[0121] a first determining sub-module, configured to input the first state of each robot into a corresponding policy network, so as to make the corresponding policy network output a first action of the corresponding robot, wherein t = t + 1;

[0122] a second determining sub-module, configured to determine a second state and a reward value of the corresponding robot after interaction with the environment based on the first action of each robot and a preset reward function;

[0123] an aggregation sub-module, configured to aggregate the first states of different robots respectively to obtain global state information of the corresponding robot;

[0124] a third determining sub-module, configured to input the global state information of each robot into a critic network of the corresponding robot to obtain a first state value of the corresponding robot;

[0125] a storage sub-module, configured to store the first state, the second state, the first action, the first state value and the reward value of each robot as experience data into an experience pool;

[0126] a fourth determining sub-module, configured to, if t is less than a maximum time step T, take the second state as the first state, return to the step of t = t + 1, until t = T, and obtain a target experience pool;

[0127] a training sub-module, configured to train the policy network and the critic network of each robot by using the experience data in the target experience pool until a maximum training number is reached, and obtain a robot navigation model of the corresponding robot.

[0128] In some optional embodiments, the training sub-module comprises:

[0129] a first determining unit, configured to determine a target action sequence and a local observation information sequence of each robot to reach a target position based on a preset path search algorithm.

[0130] a training unit, configured to train the policy network and the critic network of each robot by using the target action sequence and the local observation information sequence of different robots respectively, the experience data in the target experience pool and a imitation learning method until a maximum training number is reached, and obtain a robot navigation model of the corresponding robot.

[0131] In some optional embodiments, the aggregation sub-module comprises:

[0132] a splicing unit, configured to splice the local observation information in the first state of different robots respectively to obtain global observation information;

[0133] A mean pooling processing unit is configured to perform mean pooling processing on the aggregated information of the neighbor nodes in the first state corresponding to different robots respectively, to obtain global node features.

[0134] A second determining unit is configured to determine global state information based on the global observation information and the global node features.

[0135] The embodiment provides a robot navigation device, as shown in the accompanying drawings, which comprises: Figure 7

[0136] A second obtaining module 701 is configured to obtain a first state of a target robot and a robot navigation model of the target robot, wherein the robot navigation model is constructed by the robot navigation model construction method in the above embodiment, and the first state of the target robot is determined by local observation information of the target robot.

[0137] A second determining module 702 is configured to input the first state information of the target robot into the robot navigation model of the target robot, so that the robot navigation model outputs a target action sequence.

[0138] A navigation module 703 is configured to perform a navigation operation on the target robot based on the target action sequence.

[0139] Further function descriptions of the above modules and units are the same as those of the above corresponding embodiments, and will not be described here.

[0140] The robot navigation model construction device and the robot navigation device in the embodiment are presented in the form of function units, wherein the units refer to ASIC (Application Specific Integrated Circuit, Application Specific Integrated Circuit) circuits, processors and memories for executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] The embodiment of the present application further provides a computer device having the robot navigation model construction device shown in the accompanying drawings or the robot navigation device shown in the accompanying drawings. Figure 6 Figure 7

[0142] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of a computer device provided in an optional embodiment of the present application, as shown in the accompanying drawings. Figure 8 ​​​As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for external devices such as a keyboard and a mouse and peripheral devices such as disk devices or other storage devices. One or more busses 10 can be used to implement the interface between the various internal and external components and can be implemented using any one or more of a variety of bus technologies including a System bus, PCI, SCSI, AGP, Super- I / O bus, etc. Furthermore, various buses can be used in front side buses, back side buses, and other bus configurations based on any bus or messaging technology known to those skilled in the art. Figure 8 The processor 10 is used in the embodiments below as an example.

[0143] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0144] The memory 20 stores instructions that can be executed by the at least one processor 10, so that the at least one processor 10 can perform the method shown in the above embodiments.

[0145] The memory 20 can include a program region and a data region. The program region can store an operating system and application programs required by at least one function. The data region can store data created according to the use of the computer device, etc. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0146] The memory 20 can include a volatile memory such as a random access memory, and can also include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned types of memories.

[0147] The computer device further includes a communication interface 30 for communication with other devices or communication networks.

[0148] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0149] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, the operation of the computer can invoke or provide the method and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file, etc. Correspondingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0150] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method of constructing a robot navigation model, characterized by, The method comprises: obtaining local observation information of a plurality of robots, wherein the local observation information of each robot is used to represent position information of the corresponding robot, target position information of the corresponding robot, and position information of a plurality of other objects in an observation field of view, the other objects including obstacles and other robots in the observation field of view; constructing a graph structure of each robot by taking the corresponding robot in the local observation information of each robot as an agent node i, taking each other object as a neighbor node j, and taking the distance between the agent node i and each neighbor node j as an edge; determining node features corresponding to different neighbor nodes j of each robot based on the graph structure of each robot, wherein the node features are used to represent relative position information between the neighbor node j and the agent node i, and relative position information between the target position of the neighbor node j and the agent node i; inputting the node features corresponding to different neighbor nodes j of each robot into a pre-constructed graph neural network for feature aggregation to obtain neighbor node aggregation information of the corresponding robot; using the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and using the first state corresponding to different robots, a preset action space, and a preset reward function to optimize and train a preset multi-agent reinforcement learning model until a preset condition is met to obtain a robot navigation model of each robot.

2. The method of claim 1, wherein, The step of using the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and using the first state corresponding to different robots, a preset action space, and a preset reward function to optimize and train a preset multi-agent reinforcement learning model until a preset condition is met to obtain a robot navigation model of each robot comprises: obtaining a policy network and a critic network of each robot; initializing the policy network and the critic network of each robot when t = 0; inputting the first state of each robot into the corresponding policy network to make the corresponding policy network output a first action of the corresponding robot when t = t + 1; determining a second state and a reward value of the corresponding robot after interaction with the environment based on the first action of each robot and the preset reward function; obtaining global state information of the corresponding robot by aggregating the first state corresponding to different robots; inputting the global state information of each robot into the critic network of the corresponding robot to obtain a first state value of the corresponding robot; storing the first state, the second state, the first action, the first state value, and the reward value of each robot as experience data into an experience pool; if t is less than a maximum time step T, taking the second state as the first state, returning to the step of setting t = t + 1 until t = T to obtain a target experience pool; training the policy network and the critic network of each robot using the experience data in the target experience pool until a maximum training number is reached to obtain a robot navigation model of the corresponding robot.

3. The method of claim 2, wherein, The steps of training the strategy network and the critic network of each robot by using the experience data in the target experience pool until a maximum training number is reached to obtain the robot navigation model of the corresponding robot, comprising: determining the target action sequence and the local observation information sequence of each robot reaching the target position based on a preset path search algorithm; training the strategy network and the critic network of each robot by using the target action sequence and the local observation information sequence corresponding to different robots, the experience data in the target experience pool, and an imitation learning method until a maximum training number is reached to obtain the robot navigation model of the corresponding robot.

4. The method of claim 3, wherein, The steps of aggregating the first state corresponding to different robots to obtain the global state information of the corresponding robot, comprising: splicing the local observation information in the first state corresponding to different robots to obtain global observation information; performing mean pooling processing on the neighbor node aggregation information in the first state corresponding to different robots to obtain global node features; determining the global state information based on the global observation information and the global node features.

5. A method of robot navigation, characterized by, The method comprises: obtaining the first state of a target robot and the robot navigation model of the target robot, wherein the robot navigation model is constructed by the robot navigation model construction method according to any one of claims 1 to 4, and the first state of the target robot is determined by the local observation information of the target robot; inputting the first state information of the target robot into the robot navigation model of the target robot, so that the robot navigation model outputs a target action sequence; performing a navigation operation on the target robot based on the target action sequence.

6. A robot navigation model construction apparatus characterized by comprising: The device comprises: a first obtaining module configured to obtain local observation information of a plurality of robots, wherein the local observation information of each robot is used to represent position information of the corresponding robot, target position information of the corresponding robot, and position information of a plurality of other objects within an observation field of view, the other objects including obstacles and other robots within the observation field of view; a construction module configured to construct a graph structure of each robot by taking the corresponding robot in the local observation information of each robot as an agent node i, taking each other object as a neighbor node j, and taking the distance between the agent node i and each neighbor node j as an edge; a first determination module configured to determine node features corresponding to different neighbor nodes j of each robot based on the graph structure of each robot, wherein the node features are used to represent relative position information between the neighbor node j and the agent node i, and relative position information between the target position of the neighbor node j and the agent node i; an aggregation module configured to input the node features corresponding to different neighbor nodes j of each robot into a pre-constructed graph neural network for feature aggregation to obtain neighbor node aggregation information of the corresponding robot. The training module is configured to use the local observation information and the neighbor node aggregation information of each robot as a first state of the corresponding robot, and use the first state corresponding to each robot, a preset action space, and a preset reward function to optimize and train a preset multi-agent reinforcement learning model until a preset condition is met, so as to obtain a robot navigation model of each robot.

7. A robot navigation device, characterized in that The device comprises: The second acquisition module is configured to acquire a first state of a target robot and a robot navigation model of the target robot, wherein the robot navigation model is constructed by the robot navigation model construction method according to any one of claims 1 to 4, and the first state of the target robot is determined by local observation information of the target robot. The second determination module is configured to input the first state information of the target robot into the robot navigation model of the target robot, so that the robot navigation model outputs a target action sequence. The navigation module is configured to perform a navigation operation on the target robot based on the target action sequence.

8. A computer device, comprising: The memory and the processor are communicatively connected, and the memory stores computer instructions. The computer readable storage medium stores computer instructions for causing a computer to execute the robot navigation model construction method according to any one of claims 1 to 4 or the robot navigation method according to claim 5.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to execute the robot navigation model construction method according to any one of claims 1 to 4 or the robot navigation method according to claim 5.

10. A computer program product, characterised in that, The computer readable storage medium stores computer instructions for causing a computer to execute the robot navigation model construction method according to any one of claims 1 to 4 or the robot navigation method according to claim 5.

Images