Reachability map calculation method for mobile robotic arm based on graph reinforcement learning

Abstract

The present invention relates to a method for calculating a reachability map of a mobile manipulator based on graph reinforcement learning, comprising: capturing the relative positional relationship between a mobile manipulator and surrounding objects in a target environment based on a graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph; constructing a mobile operation model of the mobile manipulator; obtaining the heterogeneous node graph, the feature vector and the mobile operation model and using graph reinforcement learning to generate a reachability map for the mobile operation of the mobile manipulator; wherein the reachability map comprises: state-action pairs of the mobile manipulator acting on nodes; selecting state-action pairs in the reachability map and inputting them into a critic network to evaluate the state-action pairs of the mobile manipulator acting on nodes. Q value to complete the calculation of the reachability map.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for calculating a reachability map of a mobile robot arm based on graph reinforcement learning. Background Art

[0002] Embodied intelligence is an intelligent system that perceives or acts based on the physical body. It obtains information through monitoring the environment, understands problems, makes decisions and implements actions, thus generating intelligent behavior and adaptability. A frontier area of ​​embodied intelligence research is the interaction and connection between the intelligent agent and the environment. The task requires the intelligent agent to actively interact with the environment to complete the target task. Mobile Manipulation (MM) is a typical example of embodied intelligence application. Due to its mobility and flexibility, it is widely used in home indoor scenes to complete various navigation and interaction tasks. In recent years, related research in this field is growing rapidly, and many challenges such as Habitat Rearrangement, TheThreeDWorld and AI2-THOR have been proposed.

[0003] When a robot performs a mobile operation task, it needs to rely on the robot's chassis and robotic arm at the same time, and whether the robotic arm operation can be successfully performed depends largely on good base positioning. However, how to determine a good chassis position remains challenging. There are many complex objects in indoor environments. For example, obstacles on the ground and obstacles on the desktop are two different types of obstacle information for robots. How to better model the environment so that the robot can understand the relationship between objects in the environment is a problem that needs to be solved urgently. In addition, robots need to switch between different scenes in indoor environments. For example, when performing tasks on a table and a drawer, the robot needs to choose different base positions, so the robot needs to learn to quickly migrate between strategies to quickly adapt to scene changes. Although there has been a lot of research on long-term complex operation tasks, there is little research on how to use reachability maps to solve the handover / switching problem between skills, such as navigation and grasping, or the switching problem between navigation and placement.

[0004] At present, different scholars have proposed many methods for modeling indoor scenes. 3D point cloud modeling is often used in target navigation tasks, but this method consumes a lot of computing resources, while our mobile operating platform has limited computing resources, and the task of generating a reachability map does not require overly detailed modeling, but only needs to help the robot understand the spatial relationship of objects. In addition, a lot of research has been done on how to obtain the reachability data of mobile operating robots in indoor scenes. In the classic method, the inverse reachability map is obtained by performing inverse kinematics solutions multiple times at each base position, but the method is too computationally intensive and requires frequent calculations to update the map and calculate the inverse reachability; the reinforcement learning-based method needs to be trained for each scene, and has poor expansion and migration capabilities for dynamic changes in the actual environment. Summary of the invention

[0005] The purpose of the present invention is to provide a method for calculating a reachability map of a mobile robotic arm based on graph reinforcement learning.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a method for calculating a reachability map of a mobile manipulator based on graph reinforcement learning, comprising:

[0007] S1. Capturing the relative positional relationship between the mobile robot arm and surrounding objects in the target environment based on the graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph;

[0008] S2. Constructing a mobile operation model of the mobile robotic arm;

[0009] S3. Obtain the heterogeneous node graph, the feature vector and the mobile operation model and use graph reinforcement learning to generate a reachability map for the mobile operation of the mobile manipulator; wherein the reachability map includes: the state-action pair of the mobile manipulator acting on the node;

[0010] S4. Select the state-action pair in the reachability map and input it into the critic network to evaluate the state-action pair of the mobile robot acting on the node. Q value to complete the calculation of the reachability map.

[0011] According to one aspect of the present invention, in step S1, the heterogeneous node graph is represented as:

[0012]

[0013] in, represents a node in the heterogeneous node graph, Representing directed edges in the heterogeneous node graph for connecting the nodes;

[0014] The nodes in the heterogeneous node graph are: target object node, container node, ground obstacle node and desktop obstacle node;

[0015] The directed edges of the nodes are: directed edges between the container node, the ground obstacle node, the desktop obstacle node and the target object node respectively.

[0016] According to one aspect of the present invention, in step S1, the step of capturing the relative positional relationship between the mobile robot arm and the surrounding objects in the target environment based on the graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph includes:

[0017] S11. Acquire objects around the mobile robot arm in the target environment based on the robot body vision or known environmental information; wherein the objects are: a target, a container, a ground obstacle and a desktop obstacle;

[0018] S12. Obtain the spatial position relationship between the mobile robot and the surrounding objects, and obtain an isomorphic node graph between the objects; wherein, in the isomorphic node graph, the container node corresponding to the container, the ground obstacle node corresponding to the ground obstacle, and the desktop obstacle node corresponding to the desktop obstacle are respectively around the target node corresponding to the target;

[0019] S13. Based on the spatial distribution of the objects, dimension information is added to each of the nodes of the homogeneous node graph using a one-hot encoding method to generate the heterogeneous node graph and the feature vector.

[0020] According to one aspect of the present invention, the feature vector corresponding to the heterogeneous node graph is expressed as:

[0021]

[0022] in, Representation Node i Features, Representation Node i At the output of the 0th hidden layer, Represents category information, which is the sum of the dimensions of four types of nodes. w i Representation Node i The initial eigenvector of l Used to distinguish different hidden layers, used to distinguish different hidden layers, Representation Node i In the l The feature representation of the hidden layer, Indicates weight A multi-layer perceptron MLP network, Representation Node In the l The output of the hidden layer, Representation Node At the output of the first hidden layer, represents the linear transformation matrix, Representation Node In the hidden layer l The characteristic representation of Representation Node i The set of adjacent nodes of represents the normalized attention weight of the directed edge, Indicates that the directed edge is in the hidden layer l The normalized attention weight of H represents the final representation of the target environment, represents the output of the first layer, Indicates observation, s is an activation function.

[0023] According to one aspect of the present invention, in step S2, in the step of constructing the mobile operation model of the mobile manipulator, the mobile operation model is constructed based on an actor network of a Markov decision process; wherein the mobile operation model is defined by a tuple and is expressed as:

[0024]

[0025] in, S and A denote the state and action space respectively, P represents the state transition function, r represents the reward function, c represents the discount factor, P 0 represents the initial state distribution.

[0026] According to one aspect of the present invention, in step S3, the step of obtaining the heterogeneous node graph, the feature vector and the mobile operation model and using graph reinforcement learning to generate a reachability map for the mobile operation of the mobile manipulator includes:

[0027] S31. Constructing a strategy for the mobile operation of the mobile robot π , where strategy π It is expressed as:

[0028]

[0029] in, s represents the state of the mobile operation of the mobile robot arm, aIndicates in status s the actions taken;

[0030] S32. Constructing a hybrid action space of the mobile operation of the mobile manipulator based on the heterogeneous node graph, the feature vector and the mobile operation model, wherein the hybrid action space is expressed as:

[0031]

[0032]

[0033] in, represents the discrete action space, where the discrete actions represent the operating actions of the mobile robot, represents the continuous action space, where the continuous action represents the base movement action of the mobile robot, r and f denote the radius and angle coordinates in polar coordinates, respectively. i Indicates the current position of the mobile robot. a 1. a 2 means it is defined in discrete action space and represents the start / stop signal of the mobile robot. a 1> a 2, the mobile robot arm performs the grasping action; otherwise, the mobile robot arm remains stationary;

[0034] S33. Performing graph reinforcement learning based on the obtained hybrid action space to generate the reachability map; wherein the state-action pair in the reachability map is represented as: .

[0035] According to one aspect of the present invention, in step S4, a state-action pair in the reachability map is selected and input into a critic network to evaluate the state-action pair of the mobile manipulator acting on the node. Q The steps of calculating the accessibility map include:

[0036] S41. Determine an area in the heterogeneous node graph for evaluating reachability;

[0037] S42. Determine the position coordinates of the mobile mechanical arm in the area, and obtain a transformation matrix about the position coordinates;

[0038] S43. Based on the transformation matrix, the coordinates of the target object node, the container node, the ground obstacle node and the desktop obstacle node in the region are converted from the global coordinate system to the coordinate system of the mobile robot arm;

[0039] S44. Move the mobile robot arm toward the direction of the target object As the center, and determine A 180-degree range with the center as the center, where a point is taken every 10 degrees within the range, a total of 19 points;

[0040] S45. Constructing the action value of the mixed action space of the mobile robot movement operation a i , and determine the state value s i , to obtain the corresponding state-action pair;

[0041] S46. Input the obtained state-action pair into the critic network to complete the state-action pair Q The value is solved to complete the calculation of the reachability map.

[0042] According to a solution of the present invention, the present invention adopts a method of modeling the target scene using a graph attention network, which can effectively model the spatial relationship of the target scene and has good return performance for the transformation of indoor dynamic scenes.

[0043] According to a solution of the present invention, the present invention classifies objects in the target scene into four types, so that the amount of calculation for the modeling process of the present invention is effectively reduced, which can fully improve the generation efficiency of the present invention.

[0044] According to one solution of the present invention, the present invention can further implement graph reinforcement learning based on graph attention network and hybrid action space by simplifying scene modeling by adopting heterogeneous graphs, so as to achieve accurate generation of collision-free and successful grasping reachability maps for mobile robotic arms in complex scenes.

[0045] According to a solution of the present invention, the present invention obtains the accessibility near the desktop by means of graph reinforcement learning, can quickly analyze the spatial relationship in the scene, and obtain accessibility data, so that the present invention can maintain good timeliness in practical applications.

[0046] According to one solution of the present invention, the present invention can evaluate the accessibility near the container and divide the accessibility area into five levels, thereby helping the robot to achieve a higher success rate in actual application scenarios when the base is fixed. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart schematically showing the steps of a method for calculating a reachability map of a mobile manipulator according to an embodiment of the present invention;

[0048] Figure 2is a schematic diagram of a heterogeneous node graph of a target environment in a method for calculating a reachability map of a mobile manipulator according to an embodiment of the present invention;

[0049] Figure 3 is a flow chart schematically showing a method for calculating a reachability map of a mobile manipulator according to an embodiment of the present invention; DETAILED DESCRIPTION

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] When describing the embodiments of the present invention, the orientation or positional relationship expressed by the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or positional relationship shown in the relevant drawings and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0052] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0053] like Figure 1 As shown, according to one embodiment of the present invention, the method for calculating the reachability map of a mobile manipulator based on graph reinforcement learning of the present invention includes:

[0054] S1. Based on the graph attention network, the relative position relationship between the mobile robot arm and the surrounding objects in the target environment is captured to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph;

[0055] S2. Construct a mobile operation model of the mobile robot;

[0056] S3. Obtaining a heterogeneous node graph, feature vectors, and mobile operation models and using graph reinforcement learning to generate a reachability map for mobile robot movement operations; wherein the reachability map includes: a state-action pair of the mobile robot acting on the node;

[0057] S4. Select the state-action pairs in the reachability map and input them into the critic network to evaluate the state-action pairs of the mobile robot acting on the nodes.Q value to complete the calculation of the reachability map.

[0058] Combination Figure 2 and Figure 3 As shown, according to an embodiment of the present invention, in step S1, a graph attention network (GAT) is used to capture and collect objects in the target environment. Taking an indoor scene as an example, the objects contained in the indoor scene can be divided into four types: goal, container, ground obstacles and tabletop obstacles. The goal is the object occupied by the mobile robot; the container is an object used to place the target, such as a table, sofa and cabinet. Since there is only one target, there is only one corresponding container. In addition, the ground obstacle is an obstacle on the bottom surface, and the tabletop obstacle is an obstacle placed on the container. The setting type, quantity, position, etc. are random and will not be repeated here.

[0059] In this embodiment, the indoor scene is recorded as Env, and there are N objects, denoted as In the indoor scene N The target can be further recorded as , .

[0060] Further, in step S1, the step of capturing the relative positional relationship between the mobile robot arm and surrounding objects in the target environment based on the graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph includes:

[0061] S11. Acquire objects around the mobile robot arm in the target environment based on the robot body vision or known environmental information; wherein the objects are: a target, a container, a ground obstacle, and a desktop obstacle;

[0062] S12. Obtain the spatial position relationship between the mobile robot arm and the surrounding objects, and obtain the isomorphic node graph between the objects; wherein, in the isomorphic node graph, the container node corresponding to the container, the ground obstacle node corresponding to the ground obstacle, and the desktop obstacle node corresponding to the desktop obstacle are respectively around the target node corresponding to the target; in this embodiment, based on the advantage that the graph attention network can process inputs of different scales and focus on the most relevant part of the input to make decisions, the features of each node can be linearly transformed, specifically, it can be expressed as:

[0063]

[0064] in, h i Representation Node i Features, W represents the weight matrix, Representation Node i The features of are linearly transformed by the graph attention network.

[0065] Furthermore, for the adjacent nodes i and j , and its corresponding directed edge ( i , j ) Attention coefficient α ij The calculation formula is:

[0066]

[0067] in, a is a shared parameter vector, Representation Node i The set of adjacent nodes of h j Representation Node j characteristics.

[0068] Furthermore, in order to enhance stability and performance, the graph attention network uses multiple attention heads, each of which independently calculates the attention coefficient and aggregates the results:

[0069]

[0070] in, K represents the number of attention heads, s is an activation function, W k It is k The weight matrix unique to each attention head.

[0071] S13. Based on the spatial distribution of the objects, a unique-hot encoding method is used to add dimension information to each node of the homogeneous node graph to generate a heterogeneous node graph and a feature vector. In this embodiment, the heterogeneous node graph is represented as:

[0072]

[0073] in, represents a node in a heterogeneous node graph, Represents a directed edge used to connect nodes in a heterogeneous node graph; wherein the nodes in the heterogeneous node graph are: target object node, container node, ground obstacle node, and desktop obstacle node; specifically, the target object node is represented by an 8-dimensional vector, which is:

[0074]

[0075] in, represents the posture of the mobile robot, represents a quaternion, Represents the Euclidean distance from the target to the robot.

[0076] It should be noted that since each task of the mobile robot arm has only one target object node, especially for the grasping task, the target object node represents the location of the target to be grasped, and for the placement task, the target object node represents the location where the target should be placed.

[0077] In this embodiment, the desktop obstacle node is represented by a 7-dimensional vector, which is specifically:

[0078]

[0079] in, Represents the bounding box coordinates of the desktop obstacle, z j Indicates the height of the desktop obstacle, i j Indicates the orientation angle of the desktop obstacle. distance j represents the Euclidean distance between the desktop obstacle and the robot, m represents the number of obstacles on the desktop and Here, They represent the lower limit coordinates of the bounding box of the desktop obstacle in the robot coordinate system, They respectively represent the upper limit coordinates of the bounding box of the desktop obstacle in the robot coordinate system; among them, the desktop obstacles significantly affect the reachability map. Only by selecting an optimal base position for a mobile robotic arm can the mobile robotic arm avoid collision with obstacles during the grasping operation.

[0080] In this embodiment, the ground obstacle node is represented by a 7-dimensional vector, which is specifically:

[0081]

[0082] in, Represents the bounding box coordinates of ground obstacles, z i Indicates the height of the ground obstacle. i i Indicates the direction angle of the ground obstacle. distance i It represents the Euclidean distance between the ground obstacle and the robot. n represents the number of ground obstacles, and , They represent the lower limit coordinates of the bounding box of the ground obstacle in the robot coordinate system, They respectively represent the upper limit coordinates of the bounding box of the ground obstacle in the robot coordinate system.

[0083] In this embodiment, the container node is represented by a 7-dimensional vector, which is specifically:

[0084]

[0085] in, represents the bounding box coordinates of the container, z p Indicates the height of the container. i p Indicates the orientation angle of the container. distance p represents the Euclidean distance between the container and the robot, p Represents the index of the container, They represent the lower limit coordinates of the container’s bounding box in the robot’s coordinate system, They represent the upper limit coordinates of the container’s bounding box in the robot coordinate system.

[0086] In this embodiment, the pose of each node is first converted to the robot coordinate system before constructing the graph node. The advantage of using the target object as the central node is that the observation is independent of the robot position, which simplifies the calculation and allows the graph to be constructed even without the accurate positioning of the robot body.

[0087] In this embodiment, the directed edges of the nodes are: directed edges between the container node, the ground obstacle node and the desktop obstacle node and the target object node respectively.

[0088] In this embodiment, since the graph attention network is designed for isomorphic graphs, this solution needs to further process the isomorphic graphs to form heterogeneous graphs. Specifically, the feature spaces of the four types of nodes (targets, desktop obstacles, ground obstacles, and containers) are connected in sequence to create a new large feature space. Furthermore, one-hot encoding is used to add the category information of the node at the beginning of each node, and each node is set to 1 only in the relevant dimensions corresponding to its type. After parameterization, the feature vectors of the nodes have the same dimension, which is the sum of the category information and the dimensions of the four types of nodes, and is represented by the symbol w Thus, the feature vector of the heterogeneous node graph can be obtained, which is expressed as:

[0089]

[0090] in, Representation Nodei Features, Representation Node i At the output of the 0th hidden layer, Represents category information, which is the sum of the dimensions of four types of nodes. w i Representation Node i The initial eigenvector of l Used to distinguish different hidden layers, used to distinguish different hidden layers, Representation Node i In the l The feature representation of the hidden layer, Indicates weight A multi-layer perceptron MLP network, Representation Node In the l The output of the hidden layer, Representation Node At the output of the first hidden layer, represents the linear transformation matrix, Representation Node In the hidden layer l The characteristic representation of Representation Node i The set of adjacent nodes of represents the normalized attention weight of the directed edge, Indicates that the directed edge is in the hidden layer l The normalized attention weight of H represents the final representation of the target environment, represents the output of the first layer, Indicates observation.

[0091] According to an embodiment of the present invention, in step S2, in the step of constructing a mobile operation model of the mobile manipulator, the mobile operation model is constructed based on an actor network of a Markov decision process (MDP); wherein the mobile operation model is defined by a tuple and is expressed as:

[0092]

[0093] in, S and A denote the state and action space respectively, P represents the state transition function, r represents the reward function, c represents the discount factor, P 0 represents the initial state distribution.

[0094] According to an embodiment of the present invention, in step S3, the step of obtaining a heterogeneous node graph, a feature vector and a mobile operation model and using graph reinforcement learning to generate a reachability map for mobile operation of a mobile manipulator includes:

[0095] S31. Build a strategy for mobile manipulator movement π , where strategy π It is expressed as:

[0096]

[0097] in, s Indicates the status of the mobile robot's moving operation. a Indicates in status s the actions taken;

[0098] S32. Construct a hybrid action space of the mobile robot movement operation based on the heterogeneous node graph, the feature vector and the mobile operation model, wherein the hybrid action space is expressed as:

[0099]

[0100]

[0101] in, represents the discrete action space, where the discrete actions represent the operating actions of the mobile robot, represents the continuous action space, where the continuous action represents the base movement action of the mobile robot, r and f denote the radius and angle coordinates in polar coordinates, respectively. i Indicates the current position of the mobile robot. a 1. a 2 means it is defined in discrete action space and represents the start / stop signal of the mobile robot. a 1> a 2, the mobile robot arm performs the grasping action; otherwise, the mobile robot arm remains stationary;

[0102] S33. Performing graph reinforcement learning based on the obtained hybrid action space to generate a reachability map; wherein the state-action pair in the reachability map is represented as: .

[0103] According to one embodiment of the present invention, in step S4, a state-action pair in the reachability map is selected and input into the critic network to evaluate the state-action pair of the mobile robot acting on the node. Q The steps to complete the calculation of the accessibility map include:

[0104] S41. Determine the region in the heterogeneous node graph for evaluating reachability; in this embodiment, the region determined for evaluating reachability is P Specifically, assuming that the generated accessibility map is generated with a container (such as a table) as the center, the area P Set to a rectangular area whose length is a , the width is b , where the area P The center of is the center of the container, and is set to ( x c , y c ). It is worth noting that since the base position of the mobile robot cannot overlap with the container, the area Ω occupied by the container itself needs to be excluded, which is expressed as:

[0105]

[0106] in, x represents the horizontal coordinate of the area to be evaluated, y represents the ordinate of the area to be evaluated, r Indicates the length of the evaluation area.

[0107] S42. Determine the position coordinates of the mobile robot in the area and obtain the transformation matrix about the position coordinates; in this embodiment, the base of the mobile robot is located at point When the corresponding transformation matrix is ​​determined, the transformation matrix is ​​expressed as:

[0108]

[0109] S43. Based on the transformation matrix, the coordinates of the target object node, container node, ground obstacle node and desktop obstacle node in the region are converted from the global coordinate system to the coordinate system of the mobile robot; in this embodiment, each object is represented in the global coordinate system as ( x obj , y obj ), the coordinates after conversion to the robot coordinate system can be expressed as :

[0110]

[0111] S44. Move the robot arm toward the direction of the target object As the center, and determine A 180-degree range is taken as the center, wherein a point is taken every 10 degrees within the range, and a total of 19 points are taken; in this embodiment, the minimum value is taken in the counterclockwise direction. , the maximum value is obtained in the clockwise direction In this 180-degree range, a point is taken every 10 degrees. Therefore, the set of angle samples is , that is, an angle sequence with a step size of 10 degrees. Specifically, the total angle range is [ i tgt -90°, i tgt +90°], with a step length of 10 degrees, that is, [ i tgt -90°, i tgt -80°, i tgt -70°, ..., i tgt , …, i tgt +80°, i tgt +90°], a total of 19 points were sampled. Therefore, compared with taking only a single point, taking multiple points at the same time can improve the robustness of the accessibility score, smooth possible environmental noise, and avoid the algorithm from falling into a local optimal value. At the same time, the total consideration can provide more comprehensive environmental information.

[0112] S45. Constructing the action value of the hybrid action space of the mobile robot movement operation a i , and determine the state value s i , to obtain the corresponding state-action pair; in this embodiment, since the robot has reached a position near the target, it does not need to move further in the direction. Therefore, the discrete action Set to 1, Set to 0 to indicate that the robot should be enabled at the current position. Finally, get each point The corresponding status , thereby obtaining the corresponding state-action pair.

[0113] S46. Input the obtained state-action pair into the critic network to complete the state-action pair Q The value is solved to complete the calculation of the reachability map; in this embodiment, the action of the mobile robot arm near the constructed container is and current status Then, the state-action pair Input to the pre-trained critic network and calculate the corresponding Q value Q ( s i , a i ). Then we can QThe value score evaluates the robot's accessibility to the location point. High Q The value indicates that the expected cumulative reward of the state-action pair is high, and the reachability of the location point is good, thus completing the calculation of the reachability map.

[0114] The above contents are merely examples of specific solutions of the present invention. For devices and structures not described in detail therein, it should be understood that they can be implemented by adopting general devices and general methods available in the art.

[0115] The above is only one solution of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for calculating the reachability map of a mobile robot based on graph reinforcement learning, characterized in that: include: S1. Capturing the relative positional relationship between the mobile robot and surrounding objects in the target environment based on the graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph; wherein the heterogeneous node graph is represented as: in, represents a node in the heterogeneous node graph, Representing directed edges in the heterogeneous node graph for connecting the nodes; The nodes in the heterogeneous node graph are: target object node, container node, ground obstacle node and desktop obstacle node; The directed edges of the nodes are respectively: directed edges between the container node, the ground obstacle node and the desktop obstacle node and the target object node; S2. Constructing a mobile operation model of the mobile robotic arm; S3. Obtain the heterogeneous node graph, the feature vector and the mobile operation model and use graph reinforcement learning to generate a reachability map for the mobile operation of the mobile manipulator; wherein the reachability map includes: the state-action pair of the mobile manipulator acting on the node; which includes: S31. Constructing a strategy for the mobile operation of the mobile robot π , where strategy π It is expressed as: in, s represents the state of the mobile operation of the mobile robot arm, a Indicates in status s the actions taken; S32. Constructing a hybrid action space of the mobile operation of the mobile manipulator based on the heterogeneous node graph, the feature vector and the mobile operation model, wherein the hybrid action space is expressed as: in, represents the discrete action space, where the discrete actions represent the operating actions of the mobile robot, represents the continuous action space, where the continuous action represents the base movement action of the mobile robot, r and φ denote the radius and angle coordinates in polar coordinates, respectively. θ Indicates the current position of the mobile robot. a 1. a 2 is defined in discrete action space, a 1. a 2 indicates the start / stop signal of the mobile robot. a 1> a 2, the mobile robot arm performs the grasping action; otherwise, the mobile robot arm remains stationary; S33. Performing graph reinforcement learning based on the obtained hybrid action space to generate the reachability map; wherein the state-action pair in the reachability map is represented as: ; S4. Select the state-action pair in the reachability map and input it into the critic network to evaluate the state-action pair of the mobile robot acting on the node. Q value to complete the calculation of the reachability map.

2. The method for calculating the reachability map of a mobile manipulator according to claim 1, characterized in that: In step S1, the step of capturing the relative positional relationship between the mobile robot arm and surrounding objects in the target environment based on the graph attention network to model the target environment and generate a heterogeneous node graph about the target environment and a feature vector corresponding to the heterogeneous node graph includes: S11. Acquire objects around the mobile robot arm in the target environment based on the robot body vision or known environmental information; wherein the objects are: a target, a container, a ground obstacle and a desktop obstacle; S12. Obtain the spatial position relationship between the mobile robot and the surrounding objects, and obtain an isomorphic node graph between the objects; wherein, in the isomorphic node graph, the container node corresponding to the container, the ground obstacle node corresponding to the ground obstacle, and the desktop obstacle node corresponding to the desktop obstacle are respectively around the target node corresponding to the target; S13. Based on the spatial distribution of the objects, dimension information is added to each of the nodes of the homogeneous node graph using a one-hot encoding method to generate the heterogeneous node graph and the feature vector.

3. The method for calculating the reachability map of a mobile manipulator according to claim 2, characterized in that: The feature vector corresponding to the heterogeneous node graph is expressed as: in, Representation Node i Features, Representation Node i At the output of the 0th hidden layer, Represents category information, which is the sum of the dimensions of four types of nodes. w i Representation Node i The initial eigenvector of l Used to distinguish different hidden layers, used to distinguish different hidden layers, Representation Node i In the l The feature representation of the hidden layer, Indicates weight A multi-layer perceptron MLP network, Representation Node In the l The output of the hidden layer, Representation Node At the output of the first hidden layer, represents the linear transformation matrix, Representation Node In the hidden layer l The characteristic representation of Representation Node i The set of adjacent nodes of represents the normalized attention weight of the directed edge, Indicates that the directed edge is in the hidden layer l The normalized attention weights of H represents the final representation of the target environment, represents the output of the first layer, Indicates observation, σ is an activation function.

4. The method for calculating the reachability map of a mobile manipulator according to claim 3, characterized in that: In step S2, in the step of constructing a mobile operation model of the mobile manipulator, the mobile operation model is constructed based on an actor network of a Markov decision process; wherein the mobile operation model is defined by a tuple and is expressed as: in, S and A denote the state and action space respectively, P represents the state transition function, r represents the reward function, γ represents the discount factor, P 0 represents the initial state distribution.

5. The method for calculating the reachability map of a mobile manipulator according to claim 4, characterized in that: In step S4, a state-action pair in the reachability map is selected and input into a critic network to evaluate the state-action pair of the mobile robot acting on the node. Q The steps to complete the calculation of the accessibility map include: S41. Determine an area in the heterogeneous node graph for evaluating reachability; S42. Determine the position coordinates of the mobile mechanical arm in the area, and obtain a transformation matrix about the position coordinates; S43. Based on the transformation matrix, the coordinates of the target object node, the container node, the ground obstacle node and the desktop obstacle node in the region are converted from the global coordinate system to the coordinate system of the mobile robot arm; S44. Move the mobile robot arm toward the direction of the target object As the center, and determine A 180-degree range with the center as the center, where a point is taken every 10 degrees within the range, a total of 19 points; S45. Constructing the action value of the mixed action space of the mobile robot movement operation a i , and determine the state value s i , to obtain the corresponding state-action pair; S46. Input the obtained state-action pair into the critic network to complete the state-action pair Q The value is solved to complete the calculation of the reachability map.

Images