Multi-robot autonomous navigation mapping method and device

Abstract

The invention provides a multi-robot autonomous navigation mapping method and equipment, and the method comprises the steps: employing a global planner to learn a neural distance between a robot and a leading edge point through a multi-graph neural network, and carrying out the building of an affinity matrix for graph matching; based on an affinity matrix of graph matching, through a differentiable linear distribution layer, a bipartite graph matching problem is solved, and a unique leading edge point is distributed to each robot as a target point pose. And adopting a local planner and a motion controller, planning a barrier-free moving track according to the current pose of the robot, the estimated target point pose and the constructed 2D grid map, and executing specific actions. And repeating the process until the mapping is completed. According to the invention, the limitation of the prior art can be effectively overcome, and the complete construction of the scene map can be completed in the shortest time.

Description

Technical Field

[0001] The present invention relates to a multi-robot autonomous navigation and mapping method and equipment. Background Art

[0002] With the advancement of technology, multi-robot collaborative mapping plays an increasingly important role in building maps in unknown environments. Existing multi-robot mapping methods primarily rely on frontier selection or long-term value maximization methods based on reinforcement learning. However, these methods have significant limitations: frontier selection methods are often short-sighted, resulting in low time efficiency; while reinforcement learning-based methods can maximize long-term value, they cannot guarantee map integrity. Furthermore, existing methods complicate the problem of multi-robot collaboration because the action space increases linearly with the number of robots, making it difficult to find a theoretically optimal solution. Summary of the Invention

[0003] The purpose of the present invention is to provide a multi-robot autonomous navigation and mapping method and equipment.

[0004] To solve the above problems, the present invention provides a multi-robot autonomous navigation and mapping method, comprising:

[0005] Step S1, constructing a 2D grid map of the environment where the robot node is located based on the current posture and sensor depth map of each robot node in the explored environment;

[0006] Step S2, when the 2D grid map contains a frontier, updating the feature vector of each robot node r_i of the multi-graph neural network based on the 2D grid map and the current pose of each robot node, wherein the frontier is the boundary between the unknown area and the unoccupied area in the 2D grid map;

[0007] Step S3, based on the 2D grid map, the current pose of each robot node and the feature vector of each robot node of the updated multi-graph neural network, a unique frontier is assigned to each robot as the target point pose at the next time point;

[0008] Step S4, based on the 2D grid map, the current position of each robot and the position of the target point, calculate the optimal movement route of each robot node on the 2D grid map;

[0009] Step S5, based on the optimal moving path of the robot node, control the robot to move to the next target trajectory point in the optimal path, and then go to step S1.

[0010] Furthermore, in the above method, based on the current position and sensor depth map of each robot node in the explored environment, a 2D grid map of the environment in which the robot node is located is constructed, including:

[0011] The sensor depth map and pose information of all robots are used as input to construct a 2D grid map. Each pixel in the 2D grid map is divided into three types of areas: unoccupied, occupied, and unknown;

[0012] Based on the sensor depth map and pose information of each robot node at each time step t, the areas in the 2D grid map that receive sensor feedback are marked as occupied, the areas that do not receive sensor feedback are marked as unknown, and the areas that the sensor penetrates are marked as unoccupied. A frontier point map is formed while generating the 2D grid map.

[0013] Furthermore, in the above method, when the 2D grid map contains a frontier, the feature vector of each robot node r_i of the multi-graph neural network is updated based on the 2D grid map and the current pose of each robot node, including:

[0014] Based on the current posture of the robot, the feature vector of the robot node is initialized to obtain the initial feature vector of each robot node in the 0th layer of the multi-graph neural network;

[0015] The initial feature vector of the robot node features in the 0th layer of the multi-graph neural network. In each layer of the multi-graph neural network, the corresponding query vector, key vector and value vector are defined;

[0016] Calculate the edge weight between two robot nodes in each layer of the multi-graph neural network through the softmax function;

[0017] Based on the calculated edge weight between the two robot nodes, the feature vector of the robot node r_i is updated;

[0018] Through the softmax function, cross-graph calculation is performed on the current 2D grid map and the frontier point map f;

[0019] Based on the results of cross-graph calculation and the feature vectors of the robot nodes, the feature vectors of each robot node are updated.

[0020] Furthermore, in the above method, based on the current posture of the robot, the feature vector of the robot node is initialized to obtain the initial feature vector of each robot node in the 0th layer of the multi-graph neural network, including:

[0021]

[0022] in, is the initial feature vector of each robot node r_i in the 0th layer of the multi-graph neural network. For the robot node r_i,s r_iThe original information of the robot, including the current posture and semantic label of the robot, is obtained through the multi-layer perceptron f init Map the low-dimensional features in the original information to the high-dimensional space to generate the initial feature vector of each robot node r_i 0 represents the 0th layer of the multigraph neural network.

[0023] Furthermore, in the above method, the initial feature vector of the robot node features at layer 0 of the multi-graph neural network defines a corresponding query vector, key vector, and value vector at each layer of the multi-graph neural network, including:

[0024]

[0025] in, By function f query Output query vector of robot node i at layer l;

[0026] By function f key Output key vector of robot node i in layer l;

[0027] By function f value The output value vector.

[0028] Furthermore, in the above method, the edge weight between two robot nodes in each layer of the multi-graph neural network is calculated through the softmax function, including:

[0029] In each layer l of the multi-graph neural network, the edge weight between two robot nodes r_i and r_j

[0030] Calculation is done using the following equation:

[0031]

[0032] In the numerator, the input is the query vector of robot node r_i and the bond vector of robot node r_j. The exponential function exp is used to calculate the similarity score of the dot product of the two.

[0033] In the denominator, the unnormalized attention scores of all neighboring robot nodes r_h of robot node r_i are normalized by performing the same dot product and exponential operation on node r_i and all neighboring nodes r_h, and summing them up.

[0034] All neighbor robot nodes r_h include robot node r_j;

[0035] εr is the set of edges between robot nodes in the multi-graph neural network; (i,h)∈ε r Indicates that there is an edge from robot node r_i to robot node r_h; It means to sum all neighbor nodes h of node i.

[0036] Furthermore, in the above method, based on the calculated edge weight between the two robot nodes, the feature vector of the robot node r_i is updated, including:

[0037] Based on the edge weight between two robot nodes r_i and r_j The feature vector of the robot node r_i at layer l+1 Update as follows:

[0038]

[0039] Among them, f node The function takes the current feature vector of the robot node r_i and the information of the aggregated neighboring robot node r_h as input, performs nonlinear feature transformation, and generates a new feature vector of the i node in the l+1 layer;

[0040] Represents the feature vector of the robot node r_i at the lth layer;

[0041] is the updated feature vector of the robot node r_i in the l+1th layer;

[0042] is the weight of the edge from robot node r_i to neighboring robot node r_h in layer l;

[0043] is the value vector of the neighbor robot node r_h in the lth layer;

[0044] f node Is a node update function.

[0045] Furthermore, in the above method, a cross-graph calculation is performed on the current 2D grid map and the frontier point map f through the softmax function, including:

[0046] Through the softmax function, in the l+1 layer of the multi-graph neural network, cross-graph calculations are performed on the current 2D grid map and the frontier point graph f. The edge feature calculation method from the robot node i to the frontier node f_j in the frontier point graph f is as follows:

[0047]

[0048] in, It is the edge feature from the robot node r_i to the frontier node f_j at the (l+1) layer, which measures the degree of association between the two;

[0049] is the query vector of the (l+1)th layer robot node r_i;

[0050] is the key vector of the (l+1)th frontier node f_j;

[0051] d ij is the distance between the robot node r_i and the frontier node f_j;

[0052] f edge It is a function that combines the query vector, key vector and distance information to calculate the edge relevance score;

[0053] is the sum of the robot node r_i and all frontier nodes f_j, where the edge (i,h) is in the edge set ε rf In, ε rf represents the set of edges from robot node r_i to frontier node f_j.

[0054] Furthermore, in the above method, based on the result of the cross-graph calculation and the feature vector of the robot node, the feature vector of each robot node is updated, including:

[0055] Based on the results of cross-graph calculation and the feature vector of robot node r_i Update the feature vector of each robot node r_i as follows:

[0056]

[0057] in, is the feature vector of the robot node r_i in the l+2 layer, representing the state representation of the robot node in the new layer;

[0058] is the feature vector of the robot node r_i in the l+1th layer, that is, the node feature of the previous layer, which serves as the basis for updating;

[0059] f node It is a node update function, usually composed of neural network modules, used to perform nonlinear transformation and feature extraction on input information; is the edge feature from the robot node r_i to the frontier node f_j at layer l+1, obtained by the previous cross-graph calculation;

[0060] is the value vector of the frontier node f_h in the l+1th layer;

[0061] For all (i,h) in the edge set ε rf The frontier nodes f_h in are summed up.

[0062] Furthermore, in the above method, based on the 2D grid map, the current pose of each robot node and the feature vector of each robot node of the updated multi-graph neural network, a unique frontier is assigned to each robot as the target point pose at the next time point, including:

[0063] When there are still frontier points in the 2D grid map constructed by the mapping module, the 2D grid map, the robot node's pose, and the updated feature vector of the robot node are passed to the global planner;

[0064] Based on the current 2D grid map, the pose of each robot node and the updated feature vector, the neural distance between the robot node and the frontier point is learned through a multi-graph neural network;

[0065] Based on the neural distances between robot nodes and frontier points, an affinity matrix for graph matching is constructed;

[0066] Based on the affinity matrix, a differentiable linear assignment layer is used to solve the bipartite graph matching problem and assign a unique frontier to each robot as the target point pose at the next time point.

[0067] Furthermore, in the above method, each robot is assigned a unique frontier as the target point pose at the next time point, including:

[0068] The following formula is used to assign a unique frontier to each robot as the target point pose at the next time point:

[0069]

[0070] Among them, P ij Matrix, representing the allocation scheme for assigning robot node r_i to target point j, P ij Indicates the degree of assigning robot node r_i to target point j;

[0071] C ij represents the cost of assigning robot node r_i to target point j;

[0072] 1 is a vector whose elements are all 1;

[0073] a and b are parameters in the constraints, usually related to the number of robot nodes ri and target points j;

[0074] Objective function It means to find an allocation plan P that minimizes the total allocation cost.

[0075] P1=a is a constraint on resource allocation, ensuring that each resource is allocated in a certain amount or proportion, that is, the total allocation of robot nodes r_i is equal to the corresponding element in a;

[0076] It is a constraint on task allocation, ensuring that each task is allocated resources in a certain amount or proportion, that is, the sum of resources obtained by task j is equal to the corresponding element in b.

[0077] P≥0, which means that the elements in the allocation scheme cannot be negative.

[0078] Furthermore, in the above method, based on the 2D grid map, the current position of each robot and the position of the target point, the optimal movement route of each robot node is calculated on the 2D grid map, including:

[0079] Based on the 2D grid map, the current pose of each robot node, and the target pose of each robot node calculated in step S3, the optimal movement path of each robot node is calculated on the 2D grid map by the fast marching algorithm, as shown in the following formula, given the grid map and robot pose at the current time t by the fast marching algorithm (FMM), Represents the path of the robot node r_i moving to the target point at time t to obtain the optimal path of the robot node:

[0080]

[0081] in, It represents the path distance value of the robot node r_i moving to the target point at time t. This value is the object to be updated in the formula;

[0082] ∈ is a very small positive constant, which is used to avoid the denominator being zero and ensure the stability of the formula calculation;

[0083] v is a set threshold value used to Compare sizes;

[0084] Represents the obstacle distance value or other constraint distance value related to the robot node r_i at time t;

[0085] λ o It is a weight coefficient used to adjust the influence of the obstacle-related part in the denominator.

[0086] Furthermore, in the above method, based on the optimal moving path of the robot node, controlling the robot to move to the next target trajectory point in the optimal path includes:

[0087] Based on the optimal movement path of the robot node, the robot is controlled to move to the next target trajectory point in the optimal movement path. The controller determines the three movement modes: forward, left, and right based on the next target trajectory point. Finally, the robot node takes the action given by the action controller and waits for the sensor depth map to be collected at the next time step:

[0088]

[0089] in, represents the action taken by robot node r_i at time t;

[0090] move_forward represents the action of the robot moving forward;

[0091] turn_left represents the action of the robot turning left;

[0092] turn_right represents the action of the robot turning right;

[0093] θ a is the angle between the robot's current orientation and the direction of the next target trajectory point;

[0094] λ a It is a set angle threshold used to determine the critical angle value of the robot node action.

[0095] According to another aspect of the present invention, a computer-readable storage medium is further provided, on which computer-executable instructions are stored, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt any one of the methods described above.

[0096] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0097] processor; and

[0098] A memory arranged to store computer executable instructions which, when executed, cause the processor to: employ any of the methods described above.

[0099] Compared with the existing technology, the present invention aggregates the information of all robot nodes into a 2D grid map, where the 2D grid map contains information about unexplored areas and whether there are frontier points. If there are frontier points, the local planner arranges each robot to explore the target area, that is, the target point position at the next time point. The local planner plans the optimal movement path for each robot to the target point position, and then the motion controller controls the robot to move along the optimal movement path.

[0100] In addition, the present invention uses a global planner to learn the neural distance between the robot and the frontier point through a multi-graph neural network, which is used to construct the affinity matrix for graph matching; based on the affinity matrix of graph matching, a differentiable linear assignment layer is then used to solve the bipartite graph matching problem, and a unique frontier point is assigned to each robot as the target point pose. A local planner and a motion controller are used to plan an obstacle-free movement trajectory and perform specific actions based on the robot's current pose, the estimated target point pose, and the constructed 2D grid map. This process is repeated until the map is built. The present invention can effectively overcome the limitations of the existing technology and achieve the complete construction of the scene map in the shortest time.

[0101] This paper combines the advantages of frontier selection and long-term value maximization methods based on reinforcement learning. By converting the problem into a bipartite graph matching problem, the paper establishes the node correspondence between the robot and the frontier. The paper introduces a multi-graph neural network (mGNN) to learn neural distances to fill the affinity matrix and optimizes the mGNN through a differentiable linear assignment layer to maximize the long-term value, thereby improving time efficiency and map completeness. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 4 is a flowchart of a multi-robot autonomous navigation and mapping method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0103] The present invention is further described in detail below with reference to the accompanying drawings.

[0104] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party all include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0105] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0106] Computer-readable media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0107] like Figure 1 As shown, the present invention provides a multi-robot autonomous navigation and mapping method, the method comprising:

[0108] Step S1: construct a 2D grid map of the environment where the robot node is located based on the current position and sensor depth map of each robot node in the explored environment;

[0109] Here, step S1 may include:

[0110] Step S11: In the mapping module, the sensor depth maps and pose information of all robots are used as input to construct a 2D grid map. Each pixel in the 2D grid map is divided into three types of areas: unoccupied, occupied, and unknown.

[0111] Here, for each robot node r_i, (x, y, theta) is used to represent the current posture of the robot, that is, the current coordinate position and angle. The sensor can be, for example, a radar sensor.

[0112] Step S12, based on the sensor depth map and posture information of each robot node at each time step t, mark the area where sensor feedback is obtained in the 2D grid map as occupied, mark the area where no sensor feedback is obtained as unknown, and mark the area where the sensor penetrates as unoccupied; and form a frontier point map while generating the 2D grid map.

[0113] Here, the 3D point cloud information obtained by the robot node's sensor will be mapped on the 2D grid map. During the 2D grid map construction process, the map units that obtain sensor feedback will be marked as occupied, indicating areas with obstacles; areas without sensor feedback are unknown; areas penetrated by the sensor are marked as unoccupied, indicating areas without obstacles and areas where the robot node can walk.

[0114] Step S2: when the 2D grid map contains a frontier, updating the feature vector of each robot node r_i of the multi-graph neural network based on the 2D grid map and the current pose of each robot node, wherein the frontier is the boundary between the unknown area and the unoccupied area in the 2D grid map;

[0115] Here, the 2D grid map M at time t can be expressed as:

[0116] M (t) ∈[0,1] X×Y×2

[0117] Where X and Y represent the width and length of the 2D grid map.

[0118] The map is complete when, at time t, all robots' exploration areas no longer have a frontier, where the frontier is the boundary between unknown and unoccupied areas. If a frontier still exists, the information acquired by each robot is fed into the global planner, which updates the feature vector of each robot node r_i based on the 2D grid map and the current pose of each robot node.

[0119] The final completed 2D grid map is divided into two types: explored unoccupied and occupied. There is no unknown area, that is, no frontier.

[0120] Preferably, step S2 includes:

[0121] Step S21: Initialize the feature vector of the robot node based on the current posture of the robot to obtain the initial feature vector of each robot node r_i in the 0th layer of the multi-graph neural network.

[0122]

[0123] Among them, for the robot node r_i,s r_i The original information of the robot, including the current posture and semantic label of the robot, is obtained through the multi-layer perceptron f init Map the low-dimensional features in the original information to the high-dimensional space to generate the initial feature vector of each robot node r_i 0 represents the 0th layer of the multigraph neural network;

[0124] Step S22, based on the initial feature vector of the robot node feature r_i of the 0th layer of the multi-graph neural network obtained in step S21 In each layer l of the multi-graph neural network, the corresponding query vector, key vector, and value vector are defined as the basis for information propagation in the multi-graph neural network, as shown in the following formula:

[0125]

[0126]

[0127] in, By function f query Output query vector of robot node i at layer l; By function f key Output key vector of robot node i in layer l; By function f value Output value vector. Here, as shown in the subsequent step 5, Can be based on get;

[0128] Step S23, calculate the edge weight between two robot nodes r_i and r_j in each layer l of the multi-graph neural network through the softmax function It is used to express the contribution of neighbor robot node r_j to robot node r_i and is calculated by the following equation:

[0129]

[0130] In the numerator, the input is the query vector of robot node r_i and the bond vector of robot node r_j. The exponential function exp is used to calculate the similarity score of the dot product of the two.

[0131] In the denominator, the unnormalized attention scores of all neighboring robot nodes r_h of robot node r_i are normalized by performing the same dot product and exponential operation on node r_i and all neighboring nodes r_h, and summing them up.

[0132] All neighbor robot nodes r_h include robot node r_j;

[0133] ε r is the set of edges between robot nodes in the multi-graph neural network; (i,h)∈ε r Indicates that there is an edge from robot node r_i to robot node r_h; It means to sum all neighbor nodes h of node i.

[0134] In this way, the information interaction of robot nodes within the same multi-graph neural network graph can be obtained, so that each robot node can update its own features by exchanging information with its neighbor robot node r_h, so that the robot node r_i can focus more on important neighbors when aggregating the information of the neighbor robot node r_h.

[0135] Step S24, the edge weight between the two robot nodes r_i and r_j calculated based on the softmax function in step S23 The feature vector of the robot node r_i at layer l+1 Update as follows:

[0136]

[0137] Among them, f node The function takes the current feature vector of the robot node r_i and the information of the aggregated neighboring robot node r_h as input, performs nonlinear feature transformation, and generates a new feature vector of the i node in the l+1 layer;

[0138] Represents the feature vector of the robot node r_i at the lth layer;

[0139] is the updated feature vector of the robot node r_i in the l+1th layer;

[0140] is the weight of the edge from robot node r_i to neighboring robot node r_h in layer l;

[0141] is the value vector of the neighbor robot node r_h in the lth layer;

[0142] f node It is a node update function, usually composed of neural network layers, such as a multi-layer perceptron (MLP). It accepts two inputs and is used to perform a nonlinear transformation on the node features.

[0143] Here, the result of step S24 can be used in the subsequent step S26.

[0144] Step S25, cross-graph calculation: Similar to step S23, cross-graph calculation is performed on the current 2D grid map and the frontier point graph f through the softmax function in the l+1 layer of the multi-graph neural network. In step 1, the frontier point graph is formed at the same time as the 2D grid map is generated. The edge feature calculation method from the robot node r_i to the frontier node f_j in the frontier point graph f is as follows:

[0145]

[0146] in, It is the edge feature from the robot node r_i to the frontier node f_j at the (l+1) layer, which measures the degree of association between the two;

[0147] is the query vector of the (l+1)th layer robot node r_i;

[0148] is the key vector of the (l+1)th frontier node f_j;

[0149] d ij is the distance between the robot node r_i and the frontier node f_j;

[0150] f edge It is a function that combines the query vector, key vector and distance information to calculate the edge relevance score;

[0151] is the sum of the robot node r_i and all frontier nodes f_j, where the edge (i,h) is in the edge set ε rf In, ε rf represents the set of edges from robot node r_i to frontier node f_j.

[0152] Step S26, based on the result of the cross-graph calculation in step S25 and the feature vector of the robot node r_i in step S24 Update the feature vector of each robot node i as follows:

[0153]

[0154] in, is the feature vector of the robot node r_i in the l+2 layer, representing the state representation of the robot node in the new layer;

[0155] is the feature vector of the robot node r_i in the l+1th layer, that is, the node feature of the previous layer, which serves as the basis for updating;

[0156] f node It is a node update function, usually composed of neural network modules, used to perform nonlinear transformation and feature extraction on input information; is the edge feature from the robot node r_i to the frontier node f_j at the l+1th layer, reflecting the degree of association between the two, obtained by the previous cross-graph calculation;

[0157] is the value vector of the frontier node f_h in the l+1th layer;

[0158] For all (i,h) in the edge set ε rf The frontier nodes f_h in are summed up.

[0159] Step S3, based on the 2D grid map, the current pose of each robot node and the feature vector of each robot node r_i of the updated multi-graph neural network, a unique frontier is assigned to each robot as the target point pose at the next time point;

[0160] Here, the mapping module can collect the current depth map and robot pose from all robot sensors, build a continuously updated 2D grid map, and pass the 2D grid map to the global planner. The global planner can output the target point pose of each robot node at the next time point.

[0161] Preferably, step S3 includes:

[0162] Step S31: When there are still frontier points in the 2D grid map constructed by the mapping module, the 2D grid map, the position and pose of the robot node, and the updated feature vector of the robot node are passed to the global planner;

[0163] Step S32: The global planner learns the neural distance between the robot node and the frontier point through a multi-graph neural network based on the 2D grid map at the current time, the pose of each robot node, and the updated feature vector;

[0164] Step S33, the global planner constructs an affinity matrix for graph matching based on the neural distances between the robot nodes and the frontier points;

[0165] In step S34, the global planner solves the bipartite graph matching problem based on the affinity matrix through a differentiable linear assignment layer, and assigns a unique frontier to each robot as the target point pose at the next time point.

[0166] Here, the 2D grid map provides information about the overall structure of the environment, including obstacle distribution, explored areas, and unexplored frontier areas. The robot's pose specifies the robot's specific position and orientation within the current environment. The feature vector of the robot node r_i obtained in step S26 contains the comprehensive features of the robot after interaction and calculation with other nodes (such as the frontier node). This information serves as the basis for subsequent processing.

[0167] During neural distance learning, the global planner uses multiple graph neural networks to learn the neural distance between the robot and the frontier based on the 2D grid map and robot pose, which is then used to construct the graph matching affinity matrix. To construct the affinity matrix, the global planner uses multiple graph neural networks, combined with the 2D grid map, robot pose, and the updated eigenvectors of the robot node r_i obtained in step S26, to learn the neural distance between the robot and the frontier, and then construct the graph matching affinity matrix. In this process, the 2D grid map and robot pose provide information related to physical space and position, while the updated eigenvectors of the robot node r_i participate in the calculation of neural distance from the perspective of node features and associations, jointly influencing the values ​​of the elements in the affinity matrix, that is, the degree of affinity between the robot and each frontier.

[0168] To assign target poses, a differentiable linear assignment layer solves the bipartite graph matching problem based on the constructed affinity matrix. This layer uses the information in the affinity matrix to find the most suitable single frontier for each robot as the target pose, taking into account all robots and frontiers. This ensures that the overall assignment scheme is optimal in some way, such as minimizing the total distance or total cost from the robot to its assigned frontier.

[0169] By iterating through steps S21 to S26, a graph neural network is constructed between each robot node and the frontier point, generating an affinity matrix. Using a linear assignment layer, the robot nodes are assigned to the target point with the lowest global cost. Based on the Sinkhorn algorithm, the linear assignment layer adjusts the affinity matrix to a doubly random matrix, where the sum of each row and column is 1, solving the problem of assigning robot nodes between target points. The global planner then outputs the target point poses that each robot needs to reach.

[0170]

[0171] Among them, P ij Matrix, representing the allocation scheme for assigning robot node r_i to target point j, P ij Indicates the degree of assigning robot node r_i to target point j;

[0172] C ij represents the cost of assigning robot node r_i to target point j;

[0173] 1 is a vector whose elements are all 1;

[0174] a and b are parameters in the constraints, usually related to the number of robot nodes ri and target points j;

[0175] Objective function It means to find an allocation plan P that minimizes the total allocation cost.

[0176] Constraints:

[0177] P1=a is a constraint on resource allocation, ensuring that each resource is allocated in a certain amount or proportion, that is, the total allocation of robot nodes r_i is equal to the corresponding element in a;

[0178] It is a constraint on task allocation, ensuring that each task is allocated resources in a certain amount or proportion, that is, the sum of resources obtained by task j is equal to the corresponding element in b.

[0179] P≥0, which means that the elements in the allocation scheme cannot be negative.

[0180] Step S4, based on the 2D grid map, the current position of each robot and the position of the target point, calculate the optimal movement route of each robot node on the 2D grid map;

[0181] Here, the 2D grid map is passed into the local planner, along with the target point poses of each robot, which are obtained by the global planner through global planning at regular intervals.

[0182] Preferably, step S4 includes:

[0183] The local planner calculates the optimal movement path of each robot node on the 2D grid map based on the 2D grid map, the current pose of each robot node, and the target pose of each robot node calculated in step S3, using the fast marching algorithm. The formula is as follows: Represents the path of the robot node r_i moving to the target point at time t to obtain the optimal path of the robot node:

[0184]

[0185] in, It represents the path distance value of the robot node r_i moving to the target point at time t. This value is the object to be updated in the formula;

[0186] ∈ is a very small positive constant, which is used to avoid the denominator being zero and ensure the stability of the formula calculation;

[0187] τ is a threshold value set for Compare sizes;

[0188] Represents the obstacle distance value or other constraint distance value related to the robot node r_i at time t;

[0189] λ o It is a weight coefficient used to adjust the influence of the obstacle-related part in the denominator.

[0190] Step S5, based on the optimal moving path of the robot node, control the robot to move to the next target trajectory point in the optimal path, and then go to step S1.

[0191] Preferably, step S5 includes:

[0192] Based on the optimal movement path of the robot node obtained in step S4, the robot is controlled to move to the next target trajectory point in the optimal movement path. The controller determines the three movement modes: forward, left, and right based on the next target trajectory point. Finally, the robot node takes the action given by the action controller and waits for the sensor depth map to be collected at the next time step:

[0193]

[0194] in, represents the action taken by robot node r_i at time t;

[0195] move_forward represents the action of the robot moving forward;

[0196] turn_left represents the action of the robot turning left;

[0197] turn_right represents the action of the robot turning right;

[0198] θ a is the angle between the robot's current orientation and the direction of the next target trajectory point;

[0199] λ a It is a set angle threshold used to determine the critical angle value of the robot node action.

[0200] Here, an area such as a room or a work area can correspond to a robot node for exploration respectively, and the areas explored by each robot node can be spliced ​​into a complete 2D grid map.

[0201] The present invention aggregates the information of all robot nodes into a 2D grid map, wherein the 2D grid map contains information about unexplored areas and whether there are frontier points. If there are frontier points, the local planner arranges each robot to explore the target area, that is, the target point posture at the next time point. The local planner plans the optimal movement path for each robot to the target point posture, and then the motion controller controls the robot to move along the optimal movement path.

[0202] In addition, the present invention uses a global planner to learn the neural distance between the robot and the frontier point through multiple graph neural networks, which is used to construct the affinity matrix for graph matching. Based on the graph matching affinity matrix, a differentiable linear assignment layer is then used to solve the bipartite graph matching problem, assigning each robot a unique frontier point as the target point pose. Using a local planner and a motion controller, the robot's current pose, the estimated target point pose, and the constructed 2D grid map are used to plan an obstacle-free movement trajectory and execute specific actions. This process is repeated until the map is completed.

[0203] This paper combines the advantages of frontier selection and long-term value maximization methods based on reinforcement learning. By transforming the problem into a bipartite graph matching problem, it establishes node correspondences between robots and frontiers. By introducing a multi-graph neural network (mGNN) to learn neural distances to populate the affinity matrix, and optimizing the mGNN via a differentiable linear assignment layer to maximize long-term value, the paper improves time efficiency and map completeness. This paper effectively overcomes the limitations of existing technologies and completes the construction of scene maps in the shortest possible time.

[0204] According to another aspect of the present invention, a computer-readable storage medium is further provided, on which computer-executable instructions are stored, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt any one of the methods described above.

[0205] According to another aspect of the present invention, there is also provided a computer device, comprising:

[0206] processor; and

[0207] A memory arranged to store computer executable instructions which, when executed, cause the processor to: employ any of the methods described above.

[0208] The detailed contents of the various device embodiments of the present invention can be found in the corresponding parts of the various method embodiments, which will not be repeated here.

[0209] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

[0210] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0211] In addition, a portion of the present invention may be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instructions for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-carrying medium, and / or stored in a working memory of a computer device that operates according to the program instructions. Here, according to one embodiment of the present invention, a device is included, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the device is triggered to operate based on the aforementioned methods and / or technical solutions according to multiple embodiments of the present invention.

[0212] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalents of the claims be encompassed within the present invention. Any figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A multi-robot autonomous navigation and mapping method, characterized in that: include: Step S1, constructing a 2D grid map of the environment where the robot node is located based on the current posture and sensor depth map of each robot node in the explored environment; Step S2, when the 2D grid map contains a frontier, updating the feature vector of each robot node of the multi-graph neural network based on the 2D grid map and the current pose of each robot node, wherein the frontier is the boundary between the unknown area and the unoccupied area in the 2D grid map; Step S3, based on the 2D grid map, the current pose of each robot node and the feature vector of each robot node of the updated multi-graph neural network, a unique frontier is assigned to each robot as the target point pose at the next time point; Step S4, based on the 2D grid map, the current position of each robot and the position of the target point, calculate the optimal movement route of each robot node on the 2D grid map; Step S5, based on the optimal moving path of the robot node, control the robot to move to the next target trajectory point in the optimal path, and then go to step S1.

2. The multi-robot autonomous navigation and mapping method according to claim 1, characterized in that: Based on the current pose and sensor depth map of each robot node in the explored environment, a 2D grid map of the robot node's environment is constructed, including: The sensor depth map and pose information of all robots are used as input to construct a 2D grid map. Each pixel in the 2D grid map is divided into three types of areas: unoccupied, occupied, and unknown; Based on the sensor depth map and pose information of each robot node at each time step t, the areas in the 2D grid map that receive sensor feedback are marked as occupied, the areas that do not receive sensor feedback are marked as unknown, and the areas that the sensor penetrates are marked as unoccupied. A frontier point map is formed while generating the 2D grid map.

3. The multi-robot autonomous navigation and mapping method according to claim 1, wherein: When the 2D grid map contains a frontier, the feature vectors of each robot node of the multi-graph neural network are updated based on the 2D grid map and the current pose of each robot node, including: Based on the current posture of the robot, the feature vector of the robot node is initialized to obtain the initial feature vector of each robot node in the 0th layer of the multi-graph neural network; The initial feature vector of the robot node features in the 0th layer of the multi-graph neural network. In each layer of the multi-graph neural network, the corresponding query vector, key vector and value vector are defined; Calculate the edge weight between two robot nodes in each layer of the multi-graph neural network through the softmax function; Based on the calculated edge weight between the two robot nodes, the feature vector of the robot node r_i is updated; Through the softmax function, cross-graph calculation is performed on the current 2D grid map and the frontier point map f; Based on the results of cross-graph calculation and the feature vectors of the robot nodes, the feature vectors of each robot node are updated.

4. The multi-robot autonomous navigation and mapping method according to claim 3, wherein: Based on the current posture of the robot, the feature vector of the robot node is initialized to obtain the initial feature vector of each robot node in the 0th layer of the multi-graph neural network, including: in, is the initial feature vector of each robot node r_i in the 0th layer of the multi-graph neural network. For the robot node r_i,s r-i The original information of the robot, including the current posture and semantic label of the robot, is obtained through the multi-layer perceptron f init Map the low-dimensional features in the original information to the high-dimensional space to generate the initial feature vector of each robot node r_i 0 represents the 0th layer of the multigraph neural network.

5. The multi-robot autonomous navigation and mapping method according to claim 4, characterized in that: The initial feature vector of the robot node features at layer 0 of the multi-graph neural network. At each layer of the multi-graph neural network, the corresponding query vector, key vector, and value vector are defined, including: in, By function f query Output query vector of robot node i at layer l; By function f key Output key vector of robot node i in layer l; By function f value The output value vector.

6. The multi-robot autonomous navigation and mapping method according to claim 5, characterized in that: The softmax function is used to calculate the edge weights between two robot nodes in each layer of the multi-graph neural network, including: In each layer l of the multi-graph neural network, the edge weight between two robot nodes r_i and r_j Calculation is done using the following equation: In the numerator, the input is the query vector of robot node r_i and the bond vector of robot node r_j. The exponential function exp is used to calculate the similarity score of the dot product of the two. In the denominator, the unnormalized attention scores of all neighboring robot nodes r_h of robot node r_i are normalized by performing the same dot product and exponential operation on node r_i and all neighboring nodes r_h, and summing them up. All neighbor robot nodes r_h include robot node r_j; ε r is the set of edges between robot nodes in the multi-graph neural network; (i,h)∈ε r Indicates that there is an edge from robot node r_i to robot node r_h; It means to sum all neighbor nodes h of node i.

7. The multi-robot autonomous navigation and mapping method according to claim 6, characterized in that: Based on the calculated edge weight between the two robot nodes, the feature vector of the robot node r_i is updated, including: Based on the edge weight between two robot nodes r_i and r_j The feature vector of the robot node r_i at layer l+1 Update as follows: Among them, f node The function takes the current feature vector of the robot node r_i and the information of the aggregated neighboring robot node r_h as input, performs nonlinear feature transformation, and generates a new feature vector of the i node in the l+1 layer; Represents the feature vector of the robot node r_i at the lth layer; is the updated feature vector of the robot node r_i in the l+1th layer; is the weight of the edge from robot node r_i to neighboring robot node r_h in layer l; is the value vector of the neighbor robot node r_h in the lth layer; f node Is a node update function.

8. The multi-robot autonomous navigation and mapping method according to claim 7, wherein: Through the softmax function, cross-graph calculations are performed on the current 2D grid map and the frontier point map f, including: Through the softmax function, in the l+1 layer of the multi-graph neural network, cross-graph calculations are performed on the current 2D grid map and the frontier point graph f. The edge feature calculation method from the robot node i to the frontier node f_j in the frontier point graph f is as follows: in, It is the edge feature from the robot node r_i to the frontier node f_j at the (l+1) layer, which measures the degree of association between the two; is the query vector of the (l+1)th layer robot node r_i; is the key vector of the (l+1)th frontier node f_j; d ij is the distance between the robot node r_i and the frontier node f_j; f edge It is a function that combines the query vector, key vector and distance information to calculate the edge relevance score; is the sum of the robot node r_i and all frontier nodes f_j, where the edge (i,h) is in the edge set ε rf In, ε rf represents the set of edges from robot node r_i to frontier node f_j.

9. The multi-robot autonomous navigation and mapping method according to claim 8, characterized in that: Based on the results of cross-graph computation and the feature vectors of the robot nodes, the feature vectors of each robot node are updated, including: Based on the results of cross-graph calculation and the feature vector of robot node r_i Update the feature vector of each robot node r_i as follows: in, is the feature vector of the robot node r_i in the l+2 layer, representing the state representation of the robot node in the new layer; is the feature vector of the robot node r_i in the l+1th layer, that is, the node feature of the previous layer, which serves as the basis for updating; f node It is a node update function, usually composed of neural network modules, used to perform nonlinear transformation and feature extraction on input information; is the edge feature from the robot node ri to the frontier node fj at layer l+1, obtained by the previous cross-graph calculation; is the value vector of the frontier node f_h in the l+1th layer; For all (i,h) in the edge set ε rf The frontier nodes f_h in are summed up.

10. The multi-robot autonomous navigation and mapping method according to claim 1, wherein: Based on the 2D grid map, the current pose of each robot node and the feature vector of each robot node after the update of the multi-graph neural network, a unique frontier is assigned to each robot as the target point pose at the next time point, including: When there are still frontier points in the 2D grid map constructed by the mapping module, the 2D grid map, the robot node's pose, and the updated feature vector of the robot node are passed to the global planner; Based on the current 2D grid map, the pose of each robot node and the updated feature vector, the neural distance between the robot node and the frontier point is learned through a multi-graph neural network; Based on the neural distances between robot nodes and frontier points, an affinity matrix for graph matching is constructed; Based on the affinity matrix, a differentiable linear assignment layer is used to solve the bipartite graph matching problem and assign a unique frontier to each robot as the target point pose at the next time point.

11. The multi-robot autonomous navigation and mapping method according to claim 10, wherein: Assign each robot a unique frontier as the target point pose at the next time point, including: The following formula is used to assign a unique frontier to each robot as the target point pose at the next time point: Among them, P ij Matrix, representing the allocation scheme for assigning robot node r_i to target point j, P ij Indicates the degree of assigning robot node r_i to target point j; C ij represents the cost of assigning robot node r_i to target point j; 1 is a vector whose elements are all 1; a and b are parameters in the constraints, usually related to the number of robot nodes ri and target points j; Objective function It means to find an allocation plan P that minimizes the total allocation cost. P1=a is a constraint on resource allocation, ensuring that each resource is allocated in a certain amount or proportion, that is, the total allocation of robot nodes r_i is equal to the corresponding element in a; P T 1=b is a constraint on task allocation, ensuring that each task is allocated resources in a certain amount or proportion, that is, the sum of resources obtained by task j is equal to the corresponding element in b. P≥0, which means that the elements in the allocation scheme cannot be negative.

12. The multi-robot autonomous navigation and mapping method according to claim 1, wherein: Based on the 2D grid map, the current position of each robot and the target point position, the optimal movement route of each robot node is calculated on the 2D grid map, including: Based on the 2D grid map, the current pose of each robot node, and the target pose of each robot node calculated in step S3, the optimal movement path of each robot node is calculated on the 2D grid map by the fast marching algorithm, as shown in the following formula, given the grid map and robot pose at the current time t by the fast marching algorithm (FMM), Represents the path of the robot node r_i moving to the target point at time t to obtain the optimal path of the robot node: in, It represents the path distance value of the robot node r_i moving to the target point at time t. This value is the object to be updated in the formula; ∈ is a very small positive constant, which is used to avoid the denominator being zero and ensure the stability of the formula calculation; τ is a threshold value set for Compare sizes; Represents the obstacle distance value or other constraint distance value related to the robot node r_i at time t; λ o It is a weight coefficient used to adjust the influence of the obstacle-related part in the denominator.

13. The multi-robot autonomous navigation and mapping method according to claim 1, wherein: Based on the optimal movement path of the robot node, control the robot to move to the next target trajectory point in the optimal path, including: Based on the optimal movement path of the robot node, the robot is controlled to move to the next target trajectory point in the optimal movement path. The controller determines the three movement modes: forward, left, and right based on the next target trajectory point. Finally, the robot node takes the action given by the action controller and waits for the sensor depth map to be collected at the next time step: in, represents the action taken by robot node r_i at time t; move_forward represents the action of the robot moving forward; turn_left represents the action of the robot turning left; turn_right t represents the action of the robot turning right; θ a is the angle between the robot's current orientation and the direction of the next target trajectory point; λ a It is a set angle threshold used to determine the critical angle value of the robot node action.

14. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: adopt a method as described in any one of claims 1 to 13.

15. Calculator equipment, wherein include: processor; as well as A memory arranged to store computer executable instructions which, when executed, cause the processor to: perform a method as claimed in any one of claims 1 to 13.

Images