A multi-machine cooperation SLAM method based on active deep reinforcement learning
By employing a multi-machine collaborative SLAM method, deep reinforcement learning and active perception strategies are used to optimize pose estimation, thus solving the problem of cumulative SLAM error in long-term loop-free scenarios and improving localization accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-05-29
- Publication Date
- 2026-06-05
AI Technical Summary
In complex environments with no loopback over long periods, single-machine SLAM systems struggle to effectively eliminate accumulated errors, leading to pose drift and insufficient positioning accuracy.
A multi-machine cooperative SLAM method based on active deep reinforcement learning is adopted. Through deep reinforcement learning and active perception strategies among multiple machines, the pose estimation of each machine is optimized, the TD3 algorithm is used for trajectory optimization, and the cumulative error is eliminated through information interaction between the robots.
It improves the positioning accuracy and robustness of SLAM systems in complex environments, narrows the gap between the model and the actual problem, and enhances the overall accuracy and stability of multi-machine collaborative SLAM.
Smart Images
Figure CN116721154B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multi-robot SLAM, and is used for eliminating cumulative SLAM errors in complex scenarios without loops. Specifically, it relates to a multi-robot collaborative SLAM method based on active deep reinforcement learning. Background Technology
[0002] Simultaneous localization and mapping (SLAM) refers to a system equipped with specific sensors that performs self-localization and builds a map of the environment while moving in an unknown environment. SLAM effectively solves the localization problem in the absence of GPS signals and has been widely used in unmanned vehicles, robots, and other fields. Currently, single-machine visual SLAM methods have relatively mature solutions, but with the expansion of the operating environment, single-machine SLAM is inefficient and has a long estimation time, often failing to achieve good results. To address this issue, multi-machine cooperative SLAM schemes have attracted widespread attention from academia and industry. Multi-machine cooperative SLAM generally refers to a group of mobile robots that, in an unknown environment, are equipped with cameras to acquire environmental information, exchange data, estimate and optimize their own localization information, and build a map of the environment. However, the cumulative error of the SLAM system is difficult to eliminate in the long-term absence of loop closure.
[0003] Currently, mainstream SLAM algorithms use loop closure detection algorithms to eliminate accumulated errors in the system, thereby improving the accuracy of localization and mapping. However, in scenarios without loop closures for a long time, large pose drift is likely to occur. For such problems in SLAM, the ability of a single robot system to solve them is limited (A Distributed Multi-Machine Collaborative Visual SLAM Method and System_Jiang Chaoyang).
[0004] Therefore, given the above background, it is necessary to develop a multi-machine active learning collaborative SLAM algorithm to address and handle the SLAM pose accuracy problem in complex dynamic environments. Summary of the Invention
[0005] The purpose of this invention is to provide a multi-machine collaborative visual SLAM method based on active deep reinforcement learning, which can eliminate the cumulative error of the SLAM system in long-term loop-free scenarios.
[0006] To achieve the objectives of this invention, a multi-machine collaborative SLAM method based on active deep reinforcement learning is provided. Its main feature is that it can eliminate SLAM errors in complex scenes, especially in long-term loop-free scenes. Under the influence of complex and variable lighting and dynamic objects, it can achieve a certain accuracy in SLAM pose estimation and map construction, improve the robustness of the SLAM system, and thus improve the performance of the SLAM algorithm in complex environments.
[0007] This invention considers a multi-machine cooperative SLAM algorithm for deep reinforcement learning among multiple robots, improving and optimizing their respective pose estimations through mutual learning. Furthermore, it introduces an active perception strategy based on the reinforcement learning algorithm of a single robot, realizing a collaborative learning strategy among multiple robots. By actively perceiving the multi-machine SLAM poses and their accuracy probabilities P, the learning and learning objects are selected. Through multi-machine cooperation, the accumulated error of the SLAM system is achieved, thereby improving the Kyodo and robustness of the SLAM algorithm.
[0008] The objective of this invention is achieved by at least one of the following technical solutions.
[0009] A multi-machine collaborative SLAM method based on active deep reinforcement learning includes the following steps:
[0010] S1. Run the ORB-SLAM2 program on the robot, acquire images through the camera to perform pose estimation, and obtain the initial motion trajectory pose map of the multi-robot.
[0011] S2. Based on the obtained robot motion trajectory pose map, the trajectory is optimized using the deep reinforcement learning TD3 algorithm to obtain a more accurate pose.
[0012] S3. Based on the reinforcement learning algorithm, an active perception strategy is introduced to optimize the pose of multiple robots simultaneously. According to the real-time SLAM estimated probability value P, the corresponding robot is selected to optimize the pose information using the TD3 algorithm.
[0013] S4. Robots transmit their pose information and actual distance information to each other, and use the TD3 algorithm to optimize the back-end of the SLAM trajectory to eliminate accumulated errors.
[0014] Furthermore, step S1 is detailed as follows:
[0015] The system uses its own camera to perceive the surrounding environment, perform pose estimation and mapping. First, it extracts ORB feature points from the real images acquired by the camera, including two steps: extracting FAST corner points and calculating BRIEF descriptors.
[0016] After extracting ORB feature points, initial pose estimation is performed by performing feature matching between adjacent frames on the ORB feature points. Feature matching solves the data association problem in SLAM, that is, determining the correspondence between the currently seen landmark and the previously seen landmark. The initial pose estimation can be obtained by accurately matching the BRIEF descriptors to determine the feature matching between adjacent frames. This invention uses the Fast Approximate Nearest Neighbor (FLANN) algorithm for feature matching.
[0017] Based on the data association established by feature matching, camera pose and spatial point position estimation are performed to obtain the initial motion trajectory pose map of multiple cameras.
[0018] Furthermore, the process of estimating camera pose and spatial point location through data association established by feature matching includes:
[0019] The solution process is essentially solving a Bundle Adjustment problem, which is a problem of minimizing reprojection error. Considering n 3D spatial points P and their projections p, we aim to calculate the camera pose R,t, represented by ε in Lie algebra. Let i represent the i-th feature point, and assume the coordinates of the i-th spatial point are P. i =[X i ,Y i Z i ] T Its projected pixel coordinates are U i =[u i ,v i ] T The relationship between pixel position and spatial point is as follows:
[0020]
[0021] Written in matrix form:
[0022] s i U i =K exp(ε^)P i
[0023] Where i is the i-th feature point, ε is the Lie algebra of the camera pose, and s i Let K be the depth parameter corresponding to the i-th feature point, and K be the camera parameter.
[0024] Due to the unknown camera pose and noise at the observation points, this equation contains an error. Therefore, by summing the errors and constructing a least-squares problem, we can find the best camera pose, minimize the matching error of all observation points, and finally obtain the initial motion trajectory pose diagram of the multi-camera system.
[0025]
[0026] The minimum matching error ε* for all observation points is obtained by using the matrix least squares method.
[0027] Furthermore, in step S2, based on the initial pose of SLAM (with errors), and on the multi-machine initial pose trajectory map obtained from SLAM, the deep reinforcement learning TD3 algorithm is used for Markov decision process optimization of the SLAM trajectory, as follows:
[0028] The direction of decision control is determined using a reinforcement learning module, and the mathematical expression of the Markov decision process is shown below:
[0029]
[0030]
[0031] In this context, the agent's initial state at time 0 is S0. The agent freely selects action a0 from an action set A to execute. After action a0 at time 0 is executed, the agent receives the immediate reward r0 for action a0 at time 0. Simultaneously, the agent... The probability of randomly transitioning to the next state, i.e., state S1 at time 1, is... This represents the probability of action a0 at time 0 corresponding to the initial state S0 at time 0; in state S1 at time 1, the next action, action a1 at time 1, is then executed. After execution, the agent receives the immediate reward r1 for action a1 at time 1. The agent then... The probability is randomly transferred to the next state, i.e., state S2 at time 2. This process is repeated to complete the entire transfer. It is the probability that action a1 corresponds to the initial state S1. Let A be a joint probability, representing the probability that, given the choice of action a, the state transitions from s to S'. t Let S be the action set at time t. t+1 =S' is the state set, Let R be the reward value in state s of action a, E be the expected value of the state in the next time step, and R be the reward value in state s. t+1 Let t+1 be the reward function.
[0032] Furthermore, the offline training process of the deep reinforcement learning algorithm includes:
[0033] At each time step, the samples obtained by the agent from the environment, including the current action a, state s, and reward r, are stored in the experience replay pool;
[0034] During each training session, samples are randomly drawn from the experience replay pool, and the Q-value is updated accordingly.
[0035] Every preset number of training iterations, the parameters of the current Q network are copied to the target Q network, where θ'←θ, θ' is the parameter of the target Q network, and θ is the parameter of the current Q network.
[0036] The loss during training changes as follows:
[0037]
[0038] Where L' is the loss function, Q(S) t ,a t ,θ) represents the original Q-network, For the target Q-network, θ and Let S represent the weights of the original network and the target network, respectively, where a' is the action corresponding to the next state. t a t and r t These represent the state, action, and reward at time t, respectively, with γ being the depreciation coefficient.
[0039] The adaptive rule scheduler selector chooses different scheduling rules for training and feeds the selected state values back to the current Q network to complete the learning process again.
[0040] Furthermore, a learning function `learn` is used to extract samples from the experience replay pool to complete the data interaction between the agent and the environment. The Q-value is updated as follows:
[0041]
[0042] Among them, Q π*(s,a) This indicates that for any Markov model, there always exists an optimal policy π* in state s, where taking action a and following an optimal policy will yield the optimal value function; P(s'|s,a) indicates that at each decision point, the agent observes the current state and chooses action a, then transitions from state s to a new state s'; r(s,a,s') represents the reward obtained after transitioning from the current state s to the new state s'. To maximize the expected total of long-term rewards.
[0043] Furthermore, in step S3, an active perception strategy is introduced to transfer reinforcement learning from single-machine learning to multi-machine learning, as detailed below:
[0044] The classic SLAM model consists of a motion equation and an observation equation, as shown in the following equation:
[0045]
[0046] Where k represents the robot's pose at time k, j represents the j-th observation, and the state equation X and observation equation Z,X are expressed as functions.K-1 U represents the pose at time k-1. k f(x) represents the velocity input from time k-1 to time k. k-1 ,u k Construct the pose estimation equation from time k-1 to time k, w k The noise at time k; y in the observation equation j Let z represent the j-th observation point. k,j This indicates that the j-th observation point y is observed at time k. j ,h(y j ,x k Construct the pose x at time k k The observation equation, v k,j This represents the noise corresponding to time k and the j-th observation point;
[0047] In the equations of motion and observation, two noise terms w are typically assumed. k v k,j A Gaussian distribution with mean 0:
[0048] w k ~N(0,R k ),u k ~N(0,Q) k,j )
[0049] Among them, R k and Q k,j These represent the two corresponding noise terms w. k v k,j Given the variance of the data, under the influence of these noises, we hope to infer the pose x and map y, as well as the probability distribution of pose x and map y, from the noisy data z and u. This constitutes a state estimation problem; that is, estimating the robot's state is to calculate the conditional probability distribution P(x|z,u) of state x given the input data u and the observation data z.
[0050] Furthermore, to estimate the conditional distribution of the state variables, using Bayes' theorem, we have:
[0051]
[0052] Where P(x|z) represents the probability of pose X given the observation data Z, P(z) represents the observation probability, and P(x) represents the pose estimate;
[0053] Then we can solve for the maximum likelihood estimate (MLE) of x. * MLE :
[0054] x * MLE =arg max P(z|x).
[0055] Furthermore, considering pose estimation for multiple robots, the pose of each robot is selected based on the maximum P-value, and the robot with the smallest P-value is selected for active learning:
[0056] x'=argminP(z n |x n )
[0057] Where n represents the nth robot, meaning that the robot with the smallest P value is selected to enter the learning state, P(z n |x n X' represents the pose estimation accuracy of the nth robot in the current frame, and X' represents the pose of the robot with the lowest accuracy.
[0058] Furthermore, in step S4, the pose transformation of SLAM has the following model, with corresponding rotation and transformation matrices for the robot's motion between frames:
[0059]
[0060]
[0061] R represents the rotation matrix, and t represents the translation vector;
[0062] Among them, the three-dimensional rotation matrix forms a special orthogonal group SO(3), and the transformation matrix forms a special Euclidean group SE(3), which represents the robot's motion in three-dimensional space. The robot's trajectory is optimized using the ideas of Lie groups and Lie algebras. The method of multi-robot collaborative SLAM is adopted to eliminate the accumulated error in the system. A distributed multi-machine system is constructed to perform independent SLAM. The relative pose information between the machines (SO(3), SE(3)) is used as prior knowledge. The reinforcement learning algorithm is used to interact with the environment, so that the map information learns from the real environment and achieves the effect of eliminating accumulated error.
[0063] Compared with the prior art, the beneficial effects that the present invention can achieve are at least as follows:
[0064] (1) The deep reinforcement learning optimization SLAM trajectory algorithm proposed in this invention transforms the SLAM pose estimation problem into a trajectory optimization policy problem (i.e., the dynamic transfer process of Markov decision) by defining environment, state, action, reward and policy. It constructs a Markov decision model as the SLAM trajectory optimization model, narrows the gap between the model expression and the actual problem, and improves the localization accuracy and robustness of SLAM.
[0065] (2) An active perception strategy was introduced for multi-machine collaborative learning on the basis of single-machine SLAM, employing an active deep reinforcement learning multi-machine collaborative strategy. Based on the probabilistic model of SLAM, the robot actively perceives the probability of pose estimation accuracy of each robot in the current frame. The robot with high probability and accurate pose is selected as the learning target, while the robot with pose deviation uses a deep reinforcement learning action strategy to learn and optimize its pose and trajectory based on the reward and penalty in the next frame. The multi-machine collaborative trajectory optimization strategy improves the overall accuracy and robustness of multi-machine SLAM. Attached Figure Description
[0066] Figure 1 This is a schematic diagram of SLAM trajectory optimization in an embodiment of the present invention.
[0067] Figure 2 This is a flowchart illustrating the deep reinforcement learning algorithm solution in an embodiment of the present invention.
[0068] Figure 3 This is a flowchart of a multi-machine collaborative visual SLAM system in an embodiment of the present invention. Detailed Implementation
[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0070] Please see Figures 1-3 This invention provides a multi-robot collaborative SLAM method based on active deep reinforcement learning, comprising the following steps:
[0071] S1. Run the visual SLAM program on the robot, acquire images through the camera to estimate the pose, and obtain the initial motion trajectory pose map of the multi-robot system, as follows:
[0072] The system uses its own camera to perceive the surrounding environment, perform pose estimation and mapping. First, it extracts ORB feature points from the real images acquired by the camera, including two steps: extracting FAST corner points and calculating BRIEF descriptors.
[0073] After extracting ORB feature points, initial pose estimation is performed by performing feature matching between adjacent frames on the ORB feature points. Feature matching solves the data association problem in SLAM, that is, determining the correspondence between the currently seen landmark and the previously seen landmark. The initial pose estimation can be obtained by accurately matching the BRIEF descriptors to determine the feature matching between adjacent frames. In one embodiment, the Fast Approximate Nearest Neighbor (FLANN) algorithm is used for feature matching.
[0074] Based on the data association established by feature matching, camera pose and spatial point position estimation are performed to obtain the initial motion trajectory pose map of multiple cameras.
[0075] The process of estimating camera pose and spatial point location through data association established by feature matching includes:
[0076] The solution process is essentially solving a Bundle Adjustment problem, which is a problem of minimizing reprojection error. Considering n 3D spatial points P and their projections p, we aim to calculate the camera pose R,t, represented by ε in Lie algebra. Let i represent the i-th feature point, and assume the coordinates of the i-th spatial point are P. i =[X i ,Y i Z i ] T Its projected pixel coordinates are U i =[u i ,v i ] T The relationship between pixel position and spatial point is as follows:
[0077]
[0078] Written in matrix form:
[0079] s i U i =K exp(ε^)P i
[0080] Where i is the i-th feature point, ε is the Lie algebra of the camera pose, and s i Let K be the depth parameter corresponding to the i-th feature point, and K be the camera parameter.
[0081] Due to the unknown camera pose and noise at the observation points, this equation contains an error. Therefore, by summing the errors and constructing a least-squares problem, we can find the best camera pose, minimize the matching error of all observation points, and finally obtain the initial motion trajectory pose diagram of the multi-camera system.
[0082]
[0083] The minimum matching error ε* for all observation points is obtained by using the matrix least squares method.
[0084] In one embodiment, such as Figure 1 As shown, a visual SLAM program is run on the robot to acquire images via the camera and perform pose estimation to obtain the initial motion trajectory pose of multiple robots. First, ORB feature points are extracted from the real images acquired by the camera, specifically in two steps: extracting FAST corner points and calculating the BRIEF descriptor. After extracting ORB feature points, initial pose estimation is performed by feature matching between adjacent frames. Based on the data association established by feature matching, camera pose and spatial point position estimation are performed. The solution process is essentially solving a Bundle Adjustment problem, which is a problem of minimizing reprojection error. Due to the unknown camera pose and noise at the observation points, this equation contains an error. Therefore, we sum the errors, construct a least-squares problem, and then find the best camera pose to minimize it, thus obtaining the initial pose estimation shown in the figure. There is a certain deviation in the pose estimation between the robots. Using the distance between the robots as a constraint, deep reinforcement learning algorithms can be used to make their poses more accurate. The specific reinforcement learning process is described in step two.
[0085] S2. Based on the obtained robot motion trajectory pose map, the trajectory is optimized using the deep reinforcement learning TD3 algorithm to obtain a more accurate pose.
[0086] Based on the initial pose (with errors) of SLAM, and using the multi-machine initial pose trajectory map obtained from SLAM, the deep reinforcement learning TD3 algorithm is applied to the Markov decision process to optimize the SLAM trajectory, as follows:
[0087] The direction of decision control is determined using a reinforcement learning module, and the mathematical expression of the Markov decision process is shown below:
[0088]
[0089]
[0090] In this context, the agent's initial state at time 0 is S0. The agent freely selects action a0 from an action set A to execute. After action a0 at time 0 is executed, the agent receives the immediate reward r0 for action a0 at time 0. Simultaneously, the agent... The probability of randomly transitioning to the next state, i.e., state S1 at time 1, is... This represents the probability of action a0 at time 0 corresponding to the initial state S0 at time 0; in state S1 at time 1, the next action, action a1 at time 1, is then executed. After execution, the agent receives the immediate reward r1 for action a1 at time 1. The agent then... The probability is randomly transferred to the next state, i.e., state S2 at time 2. This process is repeated to complete the entire transfer. It is the probability that action a1 corresponds to the initial state S1. Let A be a joint probability, representing the probability that, given the choice of action a, the state transitions from s to S'. t Let S be the action set at time t. t+1 =S' is the state set, Let R be the reward value in state s of action a, E be the expected value of the state in the next time step, and R be the reward value in state s. t+1 Let t+1 be the reward function.
[0091] Furthermore, the offline training process of the deep reinforcement learning algorithm includes:
[0092] At each time step, the samples obtained by the agent from the environment, including the current action a, state s, and reward r, are stored in the experience replay pool;
[0093] During each training session, samples are randomly drawn from the experience replay pool, and the Q-value is updated accordingly.
[0094] Every preset number of training iterations, the parameters of the current Q network are copied to the target Q network, where θ'←θ, θ' is the parameter of the target Q network, and θ is the parameter of the current Q network.
[0095] The loss during training changes as follows:
[0096]
[0097] Where L' is the loss function, Q(S) t ,a t ,θ) represents the original Q-network, For the target Q-network, θ and Let S represent the weights of the original network and the target network, respectively, where a' is the action corresponding to the next state. t a t and r t These represent the state, action, and reward at time t, respectively, with γ being the depreciation coefficient.
[0098] The adaptive rule scheduler selector chooses different scheduling rules for training and feeds the selected state values back to the current Q network to complete the learning process again.
[0099] Furthermore, a learning function `learn` is used to extract samples from the experience replay pool to complete the data interaction between the agent and the environment. The Q-value is updated as follows:
[0100]
[0101] Among them, Q π*(s,a) This indicates that for any Markov model, there always exists an optimal policy π* in state s, where taking action a and following an optimal policy will yield the optimal value function; P(s'|s,a) indicates that at each decision point, the agent observes the current state and chooses action a, then transitions from state s to a new state s'; r(s,a,s') represents the reward obtained after transitioning from the current state s to the new state s'. To maximize the expected total of long-term rewards.
[0102] In one embodiment, the reinforcement learning environment is as follows: Figure 1 As shown, robot visual SLAM pose estimation and mapping are interactively learned through robot motion trajectories. Figure 1 As shown, Robot 2 selects reinforcement learning action policies based on the difference between its actual distance to Robots 1 and 3 and the mapping distance estimated by the current SLAM pose. Rewards and penalties are applied based on the distance difference after trajectory changes, thereby continuously optimizing the trajectory. The specific deep reinforcement learning algorithm is as follows: Figure 2 As shown. However, for single-machine robot 2SLAM, if robot 1 and robot 3 show a larger deviation, then deep reinforcement learning will be in the wrong direction. Therefore, this invention introduces an active perception strategy to optimize single-machine learning into multi-machine collaborative learning, as described in step three.
[0103] S3. Based on the reinforcement learning algorithm, an active perception strategy is introduced to optimize the pose of multiple robots simultaneously. According to the real-time SLAM estimated probability value P, the corresponding robot is selected to optimize the pose information using the TD3 algorithm.
[0104] An active perception strategy is introduced to transfer reinforcement learning from single-machine to multi-machine learning, as detailed below:
[0105] The classic SLAM model consists of a motion equation and an observation equation, as shown in the following equation:
[0106]
[0107] Where k represents the robot's pose at time k, j represents the j-th observation, and the state equation X and observation equation Z,X are expressed as functions. K-1 U represents the pose at time k-1. k f(x) represents the velocity input from time k-1 to time k.k-1 ,u k Construct the pose estimation equation from time k-1 to time k, w k The noise at time k; y in the observation equation j Let z represent the j-th observation point. k,j This indicates that the j-th observation point y is observed at time k. j ,h(y j ,x k Construct the pose x at time k k The observation equation, v k,j This represents the noise corresponding to time k and the j-th observation point;
[0108] In the equations of motion and observation, two noise terms w are typically assumed. k v k,j A Gaussian distribution with mean 0:
[0109] w k ~N(0,R k ),u k ~N(0,Q) k,j )
[0110] Among them, R k and Q k,j These represent the two corresponding noise terms w. k v k,j Given the variance of the data, under the influence of these noises, we hope to infer the pose x and map y, as well as the probability distribution of pose x and map y, from the noisy data z and u. This constitutes a state estimation problem; that is, estimating the robot's state is to calculate the conditional probability distribution P(x|z,u) of state x given the input data u and the observation data z.
[0111] Furthermore, to estimate the conditional distribution of the state variables, using Bayes' theorem, we have:
[0112]
[0113] Where P(x|z) represents the probability of pose X given the observation data Z, P(z) represents the observation probability, and P(x) represents the pose estimate;
[0114] Then we can solve for the maximum likelihood estimate (MLE) of x. * MLE :
[0115] x * MLE =argmaxP(z|x).
[0116] Furthermore, considering pose estimation for multiple robots, the pose of each robot is selected based on the maximum P-value, and the robot with the smallest P-value is selected for active learning:
[0117] x'=argminP(z n |x n )
[0118] Where n represents the nth robot, meaning that the robot with the smallest P value is selected to enter the learning state, P(z n |x n X' represents the pose estimation accuracy of the nth robot in the current frame, and X' represents the pose of the robot with the lowest accuracy.
[0119] In one embodiment, by means of Figure 3 The overall multi-machine collaborative learning strategy shown can better leverage the collaborative capabilities of multiple machines and improve the overall accuracy and robustness of SLAM.
[0120] S4. Robots transmit their pose information and actual distance information to each other, and use the TD3 algorithm to optimize the back-end of the SLAM trajectory to eliminate accumulated errors.
[0121] The pose transformation of SLAM has the following model, and for the robot's motion between frames, there are corresponding rotation and transformation matrices:
[0122]
[0123]
[0124] R represents the rotation matrix, and t represents the translation vector;
[0125] Among them, the three-dimensional rotation matrix forms a special orthogonal group SO(3), and the transformation matrix forms a special Euclidean group SE(3), which represents the robot's motion in three-dimensional space. The robot's trajectory is optimized using the ideas of Lie groups and Lie algebras. The method of multi-robot collaborative SLAM is adopted to eliminate the accumulated error in the system. A distributed multi-machine system is constructed to perform independent SLAM. The relative pose information between the machines (SO(3), SE(3)) is used as prior knowledge. The reinforcement learning algorithm is used to interact with the environment, so that the map information learns from the real environment and achieves the effect of eliminating accumulated error.
[0126] This invention provides an active deep reinforcement learning multi-machine collaborative SLAM method. By using a simulation model, the gap between the model representation and the actual problem is narrowed. The combination of simulation and deep reinforcement learning integrates SLAM with reinforcement learning and reinforcement learning with active perception, thereby improving the overall accuracy of multi-machine collaborative SLAM and eliminating the cumulative error present in SLAM systems.
[0127] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A multi-machine collaborative SLAM method based on active deep reinforcement learning, characterized in that, Includes the following steps: S1. Run the ORB-SLAM2 program on the robot, acquire images through the camera to perform pose estimation, and obtain the initial motion trajectory pose map of the multi-robot. S2. Based on the obtained robot motion trajectory pose map, the trajectory is optimized using the deep reinforcement learning TD3 algorithm to obtain a more accurate pose. S3. Based on the reinforcement learning algorithm, an active perception strategy is introduced to simultaneously optimize the pose of multiple machines. According to the real-time SLAM estimated probability value P, the appropriate robot is selected for TD3 algorithm optimization of pose information. The active perception strategy is introduced to transfer reinforcement learning from single-machine to multi-machine learning, as detailed below: The classic SLAM model consists of a motion equation and an observation equation, as shown in the following equation: Where k represents time k, j represents the j-th observation, and the state equation X and observation equation Z are expressed in functional form. This represents the pose at time k-1. This represents the velocity input from time k-1 to time k. Construct the pose estimation equation from time k-1 to time k. The noise at time k; in the observation equation This represents the j-th observation point. This indicates that the j-th observation point was observed at time k. , Construct the pose at time k The observation equation, This represents the noise corresponding to time k and the j-th observation point; In the equations of motion and observation, two noise terms are typically assumed. , A Gaussian distribution with mean 0: in, and These represent the two corresponding noise terms. , Given the variance of the data, and considering the influence of noise, we aim to infer the pose x and map y, as well as the probability distributions of pose x and map y, from noisy data z and u. This constitutes a state estimation problem; that is, estimating the robot's state is to calculate the conditional probability distribution of state x given the input data u and the observation data z. ; S4. Robots transmit their pose and actual distance information to each other, and use the TD3 algorithm for backend optimization of SLAM trajectories to eliminate accumulated errors. The pose transformation of SLAM has the following model, and for the robot's motion between frames, there are corresponding rotation and transformation matrices: R represents the rotation matrix, and t represents the translation vector; Among them, the three-dimensional rotation matrix forms a special orthogonal group SO(3), and the transformation matrix forms a special Euclidean group SE(3), which represents the robot's motion in three-dimensional space. The robot's trajectory is optimized using the ideas of Lie groups and Lie algebras. The method of multi-robot collaborative SLAM is adopted to eliminate the accumulated error in the system. A distributed multi-machine system is constructed to perform independent SLAM. The relative pose information between the machines (SO(3), SE(3)) is used as prior knowledge. The reinforcement learning algorithm is used to interact with the environment, so that the map information learns from the real environment and achieves the effect of eliminating accumulated error.
2. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 1, characterized in that, Step S1 is as follows: The system uses its own camera to perceive the surrounding environment, perform pose estimation and mapping. First, it extracts ORB feature points from the real images acquired by the camera, including two steps: extracting FAST corner points and calculating BRIEF descriptors. After extracting ORB feature points, initial pose estimation is performed by feature matching between adjacent frames of ORB feature points. Feature matching solves the data association problem in SLAM, that is, determining the correspondence between the currently seen landmark and the previously seen landmark. The initial pose estimation can be obtained by accurately matching the BRIEF descriptors to determine the feature matching between adjacent frames. The Fast Approximate Nearest Neighbor (FLANN) algorithm is used for feature matching. Based on the data association established by feature matching, camera pose and spatial point position estimation are performed to obtain the initial motion trajectory pose map of multiple cameras.
3. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 2, characterized in that, The process of estimating camera pose and spatial point location through data association established by feature matching includes: The solution process is essentially solving a Bundle Adjustment problem, which is a problem of minimizing reprojection error. Considering n 3D spatial points P and their projections p, we aim to calculate the camera pose R,t, expressed in Lie algebra as follows: ; i represents the i-th feature point, assuming the coordinates of the i-th spatial point are Its projected pixel coordinates are The relationship between pixel position and spatial point is as follows: Written in matrix form: Where i is the i-th feature point, For the Lie algebra of the camera pose, Let K be the depth parameter corresponding to the i-th feature point, and K be the camera parameter. Due to the unknown camera pose and noise at the observation points, this equation contains an error. Therefore, by summing the errors and constructing a least-squares problem, we can find the best camera pose, minimize the matching error of all observation points, and finally obtain the initial motion trajectory pose diagram of the multi-camera system. The method of matrix least squares is used to minimize the matching error of all observation points. .
4. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 1, characterized in that, In step S2, based on the initial pose of SLAM and the multi-machine initial pose trajectory map obtained from SLAM, the deep reinforcement learning TD3 algorithm is used for Markov decision process optimization of SLAM trajectory, as follows: The direction of decision control is determined using a reinforcement learning module, and the mathematical expression of the Markov decision process is shown below: Wherein, the initial state of agent at time 0 is An agent freely chooses actions from a set of actions A. To execute, the action at time 0. After execution, the action at time 0 is obtained. Instant rewards At the same time, the intelligent agent uses The probability of randomly transitioning to the next state, i.e., the state at time 1. , , The action at time 0 The initial state corresponding to time 0 The probability; the state at time 1 Next, it immediately begins executing the next action, namely the action at time 1. After execution, the action at time 1 is obtained. Instant rewards Intelligent agents are also The probability is randomly transferred to the next state, i.e., the state at time 2. , This process is repeated to complete the entire transfer. It is an action Corresponding to the initial state The probability, Let be a joint probability, representing the transition from state s to given the choice of action 'a'. The probability, Let be the set of actions at time t. For a set of states, Let E be the reward value of action a in state s, and E be the expected value of the state in the next moment. Let t+1 be the reward function.
5. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 3, characterized in that, The offline training process of the deep reinforcement learning algorithm includes: At each time step, the samples obtained by the agent from the environment, including the current action a, state s, and reward r, are stored in the experience replay pool; During each training session, samples are randomly drawn from the experience replay pool, and the Q-value is updated accordingly. Re-copy the current training data every preset number of training iterations. Network parameters to target network, , For the goal Network parameters, θ For the present Network parameters; The loss during training changes as follows: in, For loss function, Represents the original Q-network. For the target Q-network, θ and These represent the weights of the original network and the target network, respectively. The action corresponding to the next state. , and These represent the state, action, and reward at time t. This is the loss factor; The adaptive rule-based scheduler selector chooses different scheduling rules for training and feeds back the selected state value to the current system. I completed my studies again online.
6. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 5, characterized in that, A learning function `learn` is used to draw samples from the experience replay pool to complete the data interaction between the agent and the environment. The Q-value is updated as follows: in, This means that for any Markov model, there always exists an action a in state s that follows an optimal policy π*, and the optimal value function can be obtained by using this policy. This means that at each decision point, the agent observes the current state and selects action a, and then enters a new state s' from state s; This represents the reward obtained after transitioning from the current state s to the new state s'; To maximize the expected total of long-term rewards.
7. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 1, characterized in that, To estimate the conditional distribution of the state variables, using Bayes' theorem, we have: in, Let P(z) represent the probability of pose X given the observed data Z, and let P(x) represent the pose estimate. Then we can solve for the maximum likelihood estimate (MLE) of x. : 。 8. The multi-machine collaborative SLAM method based on active deep reinforcement learning according to claim 7, characterized in that, Considering pose estimation for multiple robots, the pose of each robot is selected based on the maximum P-value, and the robot with the smallest P-value is selected for active learning: Where n represents the nth robot, meaning the robot with the smallest P-value is selected to enter the learning state. This represents the pose estimation accuracy of the nth robot in the current frame. This represents the pose of the robot with the lowest accuracy.