Robot navigation method and system based on external memory

Abstract

The application provides a robot navigation method and system based on external memory, which comprises: a mileage pose estimation module inputs two adjacent frames of RGB images, processes to obtain multi-size feature maps, performs mutual convolution operation, and processes through a deeper network layer to obtain robot pose estimation; the pose is taken as input and written into an external memory pool, and a feature vector containing robot motion trajectory information is read from the memory pool; a motion strategy is output by using the feature vector to control the robot motion; the mileage pose estimation network is pre-trained, an integrated network optimization strategy is constructed, and joint optimization of supervised learning and reinforcement learning is performed on each module. The application couples external memory resources with a neural network, and improves the memory of traditional environmental features to the memory of historical pose sequences, thereby avoiding the calculation and storage resource overhead caused by redundant environmental features, increasing the memory capacity of the network, and expanding the function of the neural network.

Description

Technical Field

[0001] The present invention relates to the technical field of robot autonomous navigation algorithms, and in particular to a robot navigation method and system based on external memory. Background Art

[0002] Effective navigation in an unknown environment is fundamental to many robotic tasks. As robotic tasks become increasingly complex, the demand for accurate and efficient navigation becomes even stronger. Navigation algorithms must be robust enough to adapt to complex and changing environments while also making optimal use of computing resources. This requires policy networks to utilize memory at varying timescales, using external memory mechanisms to provide them with long- and short-term memory to adapt to both static and dynamic elements in the real environment.

[0003] Some traditional methods (such as SLAM) divide positioning and mapping tasks into several modules, and then integrate multiple modules during use. This can easily lead to problems such as poor module compatibility and low operating efficiency. In addition, some traditional internal memory neural networks (such as RNN and LSTM) can only provide short-term memory, which cannot provide sufficient information for navigation and exploration tasks.

[0004] The end-to-end method directly outputs the final result based on the input information, which has higher computational efficiency and robustness than the traditional non-end-to-end method; secondly, adding auxiliary tasks (such as depth map prediction, optical flow map prediction, detection of loop closures, etc.) to the learning task can further improve the learning effect.

[0005] The Neural Turing Machine (NTM) transforms the traditional Turing machine using a neural network structure, making it differentiable, which means it can learn from training data. Compared to RNNs and LSTMs, NTMs introduce an external memory pool to increase memory capacity. The number of network parameters is independent of the memory pool size, allowing NTMs to provide long-term memory using limited computing resources.

[0006] Patent document CN112975969A (application number: CN202110219918.5) discloses a robot control and visual perception integrated controller system and method, wherein the system includes: a robot control module, a visual perception module and an industrial bus communication module, wherein the robot control module, the visual perception module and the industrial bus communication module are integrated into an IPC, the robot control module and the visual perception module are communicatively connected, and the robot control module interacts with the robot body through the industrial bus communication module, wherein the robot control module includes: a first real-time Linux operating system unit, a PC-based bottom-level motion control unit, and an upper-level motion control unit based on ROS2; the visual perception module includes: a second real-time Linux operating system unit, a Detectron2-based image recognition unit and an image processing unit. However, this invention does not couple external memory resources with neural networks through the neural Turing machine structure. Summary of the Invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a robot navigation method and system based on external memory.

[0008] A robot navigation method based on external memory provided by the present invention includes:

[0009] Step S1: Input two adjacent RGB image frames, process them to obtain multi-scale feature maps, perform inter-convolution operations, and obtain robot pose estimation through deeper network layer processing;

[0010] Step S2: Take the pose as input and write it into the external memory pool, and read the feature vector containing the robot motion trajectory information from the memory pool;

[0011] Step S3: Use the feature vector to output the motion strategy and control the robot motion;

[0012] Step S4: Pre-train the odometry pose estimation network, build an integrated network optimization strategy, and jointly optimize each module.

[0013] Preferably, in step S1:

[0014] Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate. Specifically, it includes the following steps:

[0015] Step S1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction part and an expansion part. The contraction part consists of a convolutional layer for feature extraction, while the expansion part consists of a deconvolution layer for restoring the depth map.

[0016] Step S1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate.

[0017] Step S1.3: After the current RGB image is convolved, it is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution, the feature maps of the corresponding size of the shrunk part, and the depth map obtained by upsampling are spliced ​​together as the input of the deeper deconvolution layer to obtain depth map estimates of different resolutions.

[0018] Preferably, in step S2:

[0019] According to the Neural Turing Machine structure, a memory module based on an external memory pool is built. Specifically, the following steps are included:

[0020] Step S2.1: Read:

[0021] That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in;

[0022] in, is the memory matrix at time t, with a size of N×N, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element;

[0023] Therefore, the vector r read from the memory at time t t for

[0024]

[0025] Step S2.2: Write:

[0026] Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t :

[0027] Erase:

[0028]

[0029] New:

[0030]

[0031] That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method;

[0032] Step S2.3: Update the position weight vector ω t ;

[0033] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ;

[0034] The specific steps to update the position weight vector are as follows:

[0035] Step S2.3.1: Content-based addressing, i.e. generating a content-based position weight vector

[0036]

[0037] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element;

[0038] The formula for calculating the cosine similarity K of vectors u and v is as follows:

[0039]

[0040] Step S2.3.2: Location-based addressing:

[0041] Interpolation operation:

[0042]

[0043] The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0044] Offset operation:

[0045]

[0046] right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel;

[0047] Sharpening operation:

[0048]

[0049] Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0050] Preferably, in step S3:

[0051] Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control robot movement. Specifically, it includes the following steps:

[0052] Step S3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the sensor information at the current moment;

[0053] Step S3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as input to the policy network, which then outputs a motion policy to control the robot's motion.

[0054] Step S3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0055] Preferably, in step S4:

[0056] Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network. Specifically, the following steps are included:

[0057] Step D1.1: Two adjacent RGB images are input into the odometry pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0058] Step D1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0059] Step D1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0060] Step D1.4: Perform attention feature fusion on feature vectors a1 and a2 to obtain fused feature vector a;

[0061] Step D1.5: Use the fused feature vector a as the input to the policy output network, which outputs the policy to control the robot's motion. Use the fused feature vector a as the input to the value output network, which outputs the value for reinforcement learning network training.

[0062] Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This includes the following steps:

[0063] Step D2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0064] Step D2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

[0065] Build an integrated network optimization strategy and jointly optimize each module; the specific steps include:

[0066] Step D3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment rewards or penalizes each action of the robot. The specific steps are:

[0067] Step D3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round; c The absolute value of the collision penalty value set;

[0068] Step D3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0069] Step D3.1.3: Repeated exploration penalty is r re-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0070] Step D3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term is in d obs Close when it is greater than a certain threshold;

[0071] Step D3.1.5: The straight reward is r straight =P st , where P st is the value of the straight reward;

[0072] Step D3.1.6: The rotation penalty is r rotate =-P ro , the robot stays near the same position for too long, and will receive this penalty to avoid the robot rotating near a certain position for a long time, where P ro is the absolute size of the rotation penalty value;

[0073] Step D3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0074] Step D3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in steps D3.1.1 through D3.1.6 are summed up as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0075] Step D3.2.2: After each round of data collection, calculate the network loss using the reinforcement learning algorithm and the collected data. Backpropagate the loss through the network to iterate the integrated network parameters constructed in step D1.

[0076] Step D3.2.3: During the reinforcement learning network training process, the robot continuously repeats step D3.2.1 data collection and step D3.2.2 network training until the collection and training reaches the preset number of rounds, forming an optimized robot navigation method based on external memory.

[0077] According to the present invention, a robot navigation system based on external memory is provided, which executes the robot navigation method based on external memory, including:

[0078] Odometry pose estimation module: This module estimates the robot's pose changes based on sensor information and inputs the pose into a memory module based on an external memory pool.

[0079] Memory module based on external memory pool: obtain long-term memory through external memory pool;

[0080] Strategy output module: outputs appropriate motion strategies based on current sensor information and memory information to control robot movement;

[0081] End-to-end reinforcement learning network: Integrates the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module to build an integrated network optimization strategy and perform distributed optimization on it.

[0082] Preferably, in the odometry pose estimation module:

[0083] Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate:

[0084] Module M1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction part and an expansion part. The contraction part consists of a convolutional layer for feature extraction, while the expansion part consists of a deconvolution layer for restoring the depth map.

[0085] Module M1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate.

[0086] Module M1.3: After convolution, the current RGB image is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution are concatenated with the feature maps of the corresponding size of the shrunk part and the depth map obtained by upsampling. These are used as input to deeper deconvolution layers to obtain depth map estimates of different resolutions.

[0087] Preferably, in the memory module based on the external memory pool:

[0088] According to the neural Turing machine structure, build a memory module based on the external memory pool:

[0089] Module M2.1: Read:

[0090] That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in;

[0091] in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element;

[0092] Therefore, the vector r read from the memory at time t t for

[0093]

[0094] Module M2.2: Write:

[0095] Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t :

[0096] Erase:

[0097]

[0098] New:

[0099]

[0100] That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method;

[0101] Module M2.3: Update position weight vector ω t ;

[0102] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ;

[0103] The specific steps to update the position weight vector are as follows:

[0104] Module M2.3.1: Content-based addressing, i.e. generating content-based position weight vectors

[0105]

[0106] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element;

[0107] The formula for calculating the cosine similarity K of vectors u and v is as follows:

[0108]

[0109] Module M2.3.2: Location-Based Addressing:

[0110] Interpolation operation:

[0111]

[0112] The controller generates a threshold gt Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0113] Offset operation:

[0114]

[0115] right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel;

[0116] Sharpening operation:

[0117]

[0118] Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0119] Preferably, in the policy output module:

[0120] Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control the robot's movement:

[0121] Module M3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the current sensor information;

[0122] Module M3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as the input to the policy network, which then outputs a motion policy to control the robot's motion.

[0123] Module M3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0124] Preferably, in the end-to-end reinforcement learning network:

[0125] Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network:

[0126] Module E1.1: Two adjacent RGB images are input into the odometry pose estimation network, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0127] Module E1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0128] Module E1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0129] Module E1.4: Feature vectors a1 and a2 are fused with attention features to obtain the fused feature vector a;

[0130] Module E1.5: Use the fused feature vector a as the input of the policy output network, which outputs the policy to control the robot's motion; use the fused feature vector a as the input of the value output network, which outputs the value for reinforcement learning network training;

[0131] Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This includes the following steps:

[0132] Module E2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0133] Module E2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, use the depth map corresponding to the RGB image and the robot pose as supervision signals, calculate the loss of multi-scale depth maps and pose estimation, and train the network using backpropagation gradient descent.

[0134] Build an integrated network optimization strategy and jointly optimize each module; the specific steps include:

[0135] Module E3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment rewards or penalizes each action of the robot. The specific steps are:

[0136] Module E3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round; c The absolute value of the collision penalty value set;

[0137] Module E3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0138] Module E3.1.3: Repeated exploration penalty is r re-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0139] Module E3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term is in d obs Close when it is greater than a certain threshold;

[0140] Module E3.1.5: The straight reward is r straight =P st , where P st is the value of the straight reward;

[0141] Module E3.1.6: The penalty for rotation in place is r rotate =-p ro , the robot stays near the same position for too long, and will receive this penalty to avoid the robot rotating near a certain position for a long time, where P ro is the absolute size of the rotation penalty value;

[0142] Module E3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0143] Module E3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in modules E3.1.1 through E3.1.6 are summed as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0144] Module E3.2.2: After each round of data collection, the network loss is calculated using the reinforcement learning algorithm and the collected data. The loss is back-propagated through the network to iterate the integrated network parameters constructed in Module E1.

[0145] Module E3.2.3: During the reinforcement learning network training process, the robot continuously repeats Module E3.2.1 data acquisition and Module E3.2.2 network training until the acquisition and training reaches the preset number of rounds, forming an optimized robot navigation method based on external memory.

[0146] Compared with the prior art, the present invention has the following beneficial effects:

[0147] 1. The present invention couples external memory resources with the neural network through the neural Turing machine structure, thereby increasing the memory capacity of the network and expanding the function of the neural network;

[0148] 2. This invention introduces an external memory pool into the policy network and uses reinforcement learning for optimization. By designing an end-to-end integrated network, it solves the problems of general neural networks lacking memory and recurrent neural networks having only short-term memory capabilities. It also improves the ability of reinforcement learning networks to perform tasks that rely on long-term memory, such as autonomous environmental exploration by robots.

[0149] 3. Unlike previous methods of directly memorizing environmental features, the present invention avoids the ineffective processing of a large number of redundant features by memorizing historical trajectory sequences, which can improve the efficiency and capacity of the memory mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0150] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0151] Figure 1 A system framework diagram of a robot navigation method based on external memory designed for the present invention;

[0152] Figure 2 End-to-end reinforcement learning network structure diagram designed for this invention;

[0153] Figure 3 Detailed structural diagram of the odometer pose estimation module designed for the present invention;

[0154] Figure 4 Detailed structural diagram of the neural Turing machine designed for this invention;

[0155] Figure 5 Flowchart of the reinforcement learning network training optimization designed for this invention. DETAILED DESCRIPTION

[0156] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0157] Example 1:

[0158] The present invention proposes a robot navigation method and system based on external memory. The system includes three modules: an odometer pose estimation module, a memory module based on an external memory pool, and a strategy output module. The input of the odometer pose estimation module is two adjacent frames of RGB images acquired by the sensor, which are subjected to multiple convolution processes to obtain multi-scale feature maps; the feature maps output by the deepest convolution layer are subjected to mutual convolution operations and processed by deeper network layers to obtain the robot pose estimation as the output of the odometer module. The memory module based on the external memory pool takes the pose as input and writes it into the external memory pool, while reading the feature vector containing the robot motion trajectory information from the memory pool as the output of the memory module. The final strategy output module uses the feature vector read from the memory pool to output the motion strategy to control the robot motion.

[0159] Reinforcement learning, also known as reinforcement learning, evaluative learning, or enhanced learning, is a paradigm and methodology in machine learning. It describes and solves the problem of how an intelligent agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment. Loss function: In mathematical optimization and decision theory, a loss function maps one or more events related to one or more variables to true values, representing the loss or risk of an event. In machine learning model training, optimization and decision-making are achieved by reducing loss.

[0160] Distributed optimization is used for network optimization. First, the odometry pose estimation network is pre-trained until it can accurately estimate the robot's pose changes from two adjacent RGB image frames. This reduces the difficulty of training the reinforcement learning network and increases training speed. An integrated network optimization strategy is then constructed to jointly optimize each module, ultimately resulting in an optimized external memory-based robot navigation method and system.

[0161] According to a robot navigation method based on external memory provided by the present invention, Figure 1-Figure 5 Shown, including:

[0162] Step S1: Input two adjacent RGB image frames, process them to obtain multi-scale feature maps, perform inter-convolution operations, and obtain robot pose estimation through deeper network layer processing;

[0163] Specifically, in step S1:

[0164] Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate. Specifically, it includes the following steps:

[0165] Step S1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction part and an expansion part. The contraction part consists of a convolutional layer for feature extraction, while the expansion part consists of a deconvolution layer for restoring the depth map.

[0166] Step S1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate.

[0167] Step S1.3: After the current RGB image is convolved, it is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution, the feature maps of the corresponding size of the shrunk part, and the depth map obtained by upsampling are spliced ​​together as the input of the deeper deconvolution layer to obtain depth map estimates of different resolutions.

[0168] Step S2: Take the pose as input and write it into the external memory pool, and read the feature vector containing the robot motion trajectory information from the memory pool;

[0169] Specifically, in step S2:

[0170] According to the Neural Turing Machine structure, a memory module based on an external memory pool is built. Specifically, the following steps are included:

[0171] Step S2.1: Read:

[0172] That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in;

[0173] in, is the memory matrix at time t, with a size of N×N, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t(i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element;

[0174] Therefore, the vector r read from the memory at time t t for

[0175]

[0176] Step S2.2: Write:

[0177] Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t :

[0178] Erase:

[0179]

[0180] New:

[0181]

[0182] That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method;

[0183] Step S2.3: Update the position weight vector ω t ;

[0184] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ;

[0185] The specific steps to update the position weight vector are as follows:

[0186] Step S2.3.1: Content-based addressing, i.e. generating a content-based position weight vector

[0187]

[0188] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element;

[0189] The formula for calculating the cosine similarity K of vectors u and v is as follows:

[0190]

[0191] Step S2.3.2: Location-based addressing:

[0192] Interpolation operation:

[0193]

[0194] The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0195] Offset operation:

[0196]

[0197] right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel;

[0198] Sharpening operation:

[0199]

[0200] Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0201] Step S3: Use the feature vector to output the motion strategy and control the robot motion;

[0202] Specifically, in step S3:

[0203] Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control robot movement. Specifically, it includes the following steps:

[0204] Step S3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the sensor information at the current moment;

[0205] Step S3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as input to the policy network, which then outputs a motion policy to control the robot's motion.

[0206] Step S3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0207] Step S4: Pre-train the odometry pose estimation network, build an integrated network optimization strategy, and jointly optimize each module.

[0208] Specifically, in step S4:

[0209] Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network. Specifically, the following steps are included:

[0210] Step D1.1: Two adjacent RGB images are input into the odometry pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0211] Step D1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i)As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0212] Step D1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0213] Step D1.4: Perform attention feature fusion on feature vectors a1 and a2 to obtain fused feature vector a;

[0214] Step D1.5: Use the fused feature vector a as the input to the policy output network, which outputs the policy to control the robot's motion. Use the fused feature vector a as the input to the value output network, which outputs the value for reinforcement learning network training.

[0215] Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This includes the following steps:

[0216] Step D2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0217] Step D2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

[0218] Build an integrated network optimization strategy and jointly optimize each module; the specific steps include:

[0219] Step D3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment rewards or penalizes each action of the robot. The specific steps are:

[0220] Step D3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round; c The absolute value of the collision penalty value set;

[0221] Step D3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0222] Step D3.1.3: Repeated exploration penalty is r re-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0223] Step D3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term is in d obs Close when it is greater than a certain threshold;

[0224] Step D3.1.5: The straight reward is r straight =P st , where P st is the value of the straight reward;

[0225] Step D3.1.6: The rotation penalty is r rotate =-P ro , the robot stays near the same position for too long, and will receive this penalty to avoid the robot rotating near a certain position for a long time, where P ro is the absolute size of the rotation penalty value;

[0226] Step D3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0227] Step D3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in steps D3.1.1 through D3.1.6 are summed up as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0228] Step D3.2.2: After each round of data collection, calculate the network loss using the reinforcement learning algorithm and the collected data. Backpropagate the loss through the network to iterate the integrated network parameters constructed in step D1.

[0229] Step D3.2.3: During the reinforcement learning network training process, the robot continuously repeats step D3.2.1 data collection and step D3.2.2 network training until the collection and training reaches the preset number of rounds, forming an optimized robot navigation method based on external memory.

[0230] According to the present invention, a robot navigation system based on external memory is provided, which executes the robot navigation method based on external memory, including:

[0231] Odometry pose estimation module: This module estimates the robot's pose changes based on sensor information and inputs the pose into a memory module based on an external memory pool.

[0232] Specifically, in the odometry pose estimation module:

[0233] Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate:

[0234] Module M1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction part and an expansion part. The contraction part consists of a convolutional layer for feature extraction, while the expansion part consists of a deconvolution layer for restoring the depth map.

[0235] Module M1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate.

[0236] Module M1.3: After convolution, the current RGB image is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution are concatenated with the feature maps of the corresponding size of the shrunk part and the depth map obtained by upsampling. These are used as input to deeper deconvolution layers to obtain depth map estimates of different resolutions.

[0237] Memory module based on external memory pool: obtain long-term memory through external memory pool;

[0238] Specifically, in the memory module based on the external memory pool:

[0239] According to the neural Turing machine structure, build a memory module based on the external memory pool:

[0240] Module M2.1: Read:

[0241] That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in;

[0242] in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ωt The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element;

[0243] Therefore, the vector r read from the memory at time t t for

[0244]

[0245] Module M2.2: Write:

[0246] Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t :

[0247] Erase:

[0248]

[0249] New:

[0250]

[0251] That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method;

[0252] Module M2.3: Update position weight vector ω t ;

[0253] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ;

[0254] The specific steps to update the position weight vector are as follows:

[0255] Module M2.3.1: Content-based addressing, i.e. generating content-based position weight vectors

[0256]

[0257] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element;

[0258] The formula for calculating the cosine similarity K of vectors u and v is as follows:

[0259]

[0260] Module M2.3.2: Location-Based Addressing:

[0261] Interpolation operation:

[0262]

[0263] The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0264] Offset operation:

[0265]

[0266] right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel;

[0267] Sharpening operation:

[0268]

[0269] Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0270] Strategy output module: outputs appropriate motion strategies based on current sensor information and memory information to control robot movement;

[0271] Specifically, in the strategy output module:

[0272] Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control the robot's movement:

[0273] Module M3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the current sensor information;

[0274] Module M3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as the input to the policy network, which then outputs a motion policy to control the robot's motion.

[0275] Module M3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0276] End-to-end reinforcement learning network: Integrates the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module to build an integrated network optimization strategy and perform distributed optimization on it.

[0277] Specifically, in the end-to-end reinforcement learning network:

[0278] Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network. Specifically, the following steps are included:

[0279] Module E1.1: Two adjacent RGB images are input into the odometry pose estimation network, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0280] Module E1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0281] Module E1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0282] Module E1.4: Feature vectors a1 and a2 are fused with attention features to obtain the fused feature vector a;

[0283] Module E1.5: Use the fused feature vector a as the input of the policy output network, which outputs the policy to control the robot's motion; use the fused feature vector a as the input of the value output network, which outputs the value for reinforcement learning network training;

[0284] Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB images:

[0285] Module E2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0286] Module E2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, use the depth map corresponding to the RGB image and the robot pose as supervision signals, calculate the loss of multi-scale depth maps and pose estimation, and train the network using backpropagation gradient descent.

[0287] Build an integrated network optimization strategy and jointly optimize each module; the specific steps include:

[0288] Module E3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment rewards or penalizes each action of the robot. The specific steps are:

[0289] Module E3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round; c The absolute value of the collision penalty value set;

[0290] Module E3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0291] Module E3.1.3: Repeated exploration penalty is r re-search =-P r, when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0292] Module E3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term is in d obs Close when it is greater than a certain threshold;

[0293] Module E3.1.5: The straight reward is r straight =P st , where P st is the value of the straight reward;

[0294] Module E3.1.6: The penalty for rotation in place is r rotate =-P ro , the robot stays near the same position for too long, and will receive this penalty to avoid the robot rotating near a certain position for a long time, where P ro is the absolute size of the rotation penalty value;

[0295] Module E3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0296] Module E3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in modules E3.1.1 through E3.1.6 are summed as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0297] Module E3.2.2: After each round of data collection, the network loss is calculated using the reinforcement learning algorithm and the collected data. The loss is back-propagated through the network to iterate the integrated network parameters constructed in Module E1.

[0298] Module E3.2.3: During the reinforcement learning network training process, the robot continuously repeats Module E3.2.1 data acquisition and Module E3.2.2 network training until the acquisition and training reaches the preset number of rounds, forming an optimized robot navigation method based on external memory.

[0299] Example 2:

[0300] Example 2 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0301] The technical solution proposed in this invention is as follows: a robot navigation method and system based on external memory. Specifically, it includes the following modules:

[0302] Module A: Build an odometer pose estimation module to estimate the robot's pose changes based on sensor information;

[0303] Module B: Build a memory module based on an external memory pool to obtain long-term memory through the external memory pool;

[0304] Module C: Build a strategy output module to output the appropriate motion strategy based on the current sensor information and memory information to control the robot's movement;

[0305] Module D: Build an end-to-end reinforcement learning network, construct an integrated network optimization strategy, and perform distributed optimization on it.

[0306] 1. Preferably, the module A specifically includes the following modules:

[0307] Module A1: Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate. Specifically, it includes the following steps:

[0308] Step A1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction (encoding) part and an expansion (decoding) part. The contraction part is mainly composed of convolutional layers for feature extraction, while the expansion part is mainly composed of deconvolutional layers for restoring the depth map.

[0309] Step A1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate.

[0310] Step A1.3: After convolution, the current RGB image is gradually restored to its original size through deconvolution. The deconvolution-derived feature maps of each size, the feature maps of the corresponding sizes of the shrunk parts, and the upsampled depth map are concatenated as input to the deeper deconvolution layer, ultimately obtaining depth map estimates of different resolutions.

[0311] 2. Preferably, the module B specifically includes the following modules:

[0312] Module B1: Building a memory module based on an external memory pool based on the Neural Turing Machine (NTM) architecture. Specifically, it includes the following steps:

[0313] Step B1.1: Read;

[0314] That is, through the position weight vector ω t For the memory matrix M t The N memory units in are weighted summed.

[0315] in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N;

[0316] Therefore, the vector read from the memory at time t is

[0317]

[0318] Step B1.2: Write;

[0319] Each write to the memory matrix can be divided into two steps: erasing and adding, that is, first starting from the memory matrix M at time t-1 t-1 Erase some information and then add new information to generate the memory matrix M at time t t

[0320] Erase:

[0321]

[0322] New:

[0323]

[0324] That is, a write to the memory matrix is ​​completed; wherein, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method.

[0325] Step B1.3: Update the position weight vector ω t ;

[0326] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、gt 、s t , γ t The specific steps to update the position weight vector are as follows:

[0327] Step B1.3.1: Content-based addressing, i.e. generating a content-based position weight vector

[0328]

[0329] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; the similarity measure K calculates the cosine similarity:

[0330]

[0331] Step B1.3.2: Location-Based Addressing

[0332] First is the interpolation operation:

[0333]

[0334] The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0335] Next is the offset operation:

[0336]

[0337] right Each element in It is considered that some elements adjacent to it are related, so the new elements generated by circular convolution It includes and surrounding elements. t is the convolution kernel.

[0338] Finally, the sharpening operation:

[0339]

[0340] When the weights in the offset operation are relatively even, the above convolution operation will lead to data dispersion and leakage, so the difference in weight size needs to be strengthened. That is, the parameter γ generated by the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0341] 3. Preferably, the module C specifically includes the following modules:

[0342] Module C1: Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control robot movement. Specifically, it includes the following steps:

[0343] Step C1.1: Build the observation network; reuse the encoding network of the odometry pose estimation network to obtain the feature map from the current RGB image as the input of the observation network; after processing by the observation network, output a feature vector containing the sensor information at the current moment;

[0344] Step C1.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as input to the policy network, which then outputs a motion policy to control the robot's motion.

[0345] Step C1.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0346] 4. Preferably, the module D specifically includes the following modules:

[0347] Module D1: Integrates the odometry pose estimation module, the external memory pool-based memory module, and the policy output module, coupling each functional module into an end-to-end reinforcement learning network. Specifically, it includes the following steps:

[0348] Step D1.1: Two adjacent RGB images are input into the odometry pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0349] Step D1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) , as the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0350] Step D1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0351] Step D1.4: Perform attention feature fusion on feature vectors a1 and a2 to obtain fused feature vector a;

[0352] Step D1.5: Use the fused feature vector a as the input to the policy output network, which outputs the policy to control the robot's motion. Use the fused feature vector a as the input to the value output network, which outputs the value used for reinforcement learning network training.

[0353] Module D2: Network Distribution Optimization: Pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This includes the following steps:

[0354] Step D2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0355] Step D2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

[0356] Module D3: Build an integrated network optimization strategy and jointly optimize each module; specifically, it includes the following steps:

[0357] Step D3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment will reward or punish the robot for each action. The specific steps are:

[0358] Step D3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round. c The absolute value of the collision penalty value set;

[0359] Step D3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0360] Step D3.1.3: Repeated exploration penalty is rre-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0361] Step D3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term can be obs Close when it is greater than a certain threshold;

[0362] Step D3.1.5: The straight reward is r straight =P st , the robot's straight action is conducive to expanding the exploration rate and avoiding repeated exploration, where P st is the value of the straight reward;

[0363] Step D3.1.6: The rotation penalty is r rotate =-P ro , the robot will receive this penalty if it stays near the same position for too long, to avoid the robot rotating near a certain position for a long time. ro is the absolute size of the rotation penalty value;

[0364] Step D3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0365] Step D3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in steps D3.1.1 through D3.1.6 are summed up as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0366] Step D3.2.2: After each round of data collection, calculate the network loss using the reinforcement learning algorithm and the collected data. Backpropagate the loss through the network to iterate the integrated network parameters constructed in step D1.

[0367] Step D3.2.3: During the reinforcement learning network training process, the robot continuously repeats step D3.2.1 data collection and step D3.2.2 network training until the collection-training reaches a preset number of rounds, thereby forming an optimized external memory-based robot navigation method and system.

[0368] Example 3:

[0369] Example 3 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0370] Autonomous systems, capable of autonomous navigation in complex environments, are fundamental to many robotic tasks. As robotic tasks become increasingly complex, the demand for accurate and efficient navigation becomes even stronger. Robotic navigation requires multiple complex tasks, including localization and mapping, perception and planning, autonomous decision-making, and motion control. It also requires robustness to adapt to complex and changing environments. This requires integrated deep coupling and an efficient and robust end-to-end learning framework. Furthermore, navigation requires robots to possess long- and short-term memory to adapt to both static and dynamic elements in their environments.

[0371] Therefore, the core challenges are: first, how to design the overall network framework to ensure the integrated and deep integration of the reinforcement learning network and improve the network's efficiency and robustness; second, how to increase the network's memory capacity and improve its ability to utilize long-term memory.

[0372] The present invention provides a robot navigation method based on external memory, which targets the research deficiencies of intelligent robots and the above-mentioned research focuses, and mainly includes four steps:

[0373] Step A: Build an odometer pose estimation module to estimate the robot's pose changes based on sensor information; the odometer pose estimation module structure is as follows: Figure 3 shown.

[0374] The input is two adjacent RGB images, which are processed by multiple convolutional layers to obtain multi-scale feature maps. The feature maps output by the deepest convolutional layer undergo inter-convolution operations and are processed by deeper network layers to obtain pose estimates, which serve as the network output and are stored in an external memory pool in the end-to-end reinforcement learning network. The feature map obtained by convolution of the current RGB image is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution, the feature maps of the corresponding sizes of the convolution part, and the depth map obtained by upsampling are spliced ​​together as the input of the deeper deconvolution layer, ultimately obtaining depth map estimates of different resolutions. When training the odometry network on the dataset, depth map restoration is used as an auxiliary task. The depth map corresponding to the RGB image and the robot pose are used as supervision signals. The loss of the multi-scale depth map and the pose estimation loss are calculated, and the network can be trained using the backpropagation gradient descent method.

[0375] Skip connections concatenate the deconvolution feature maps of various sizes with the convolution feature maps of corresponding sizes and the upsampled depth map. These serve as input to deeper deconvolution layers. This helps preserve sensor information lost during convolution, improves network performance, and addresses the vanishing and exploding gradient issues in deep networks. Using depth estimation as an auxiliary task allows the network to better restore the depth map, improving the accuracy of pose estimation.

[0376] Step B: Build a memory module based on an external memory pool to obtain long-term memory through the external memory pool; the detailed structure of the neural Turing machine used is as follows Figure 4 shown.

[0377] An external memory pool is introduced through a Neural Turing Machine. The pose output by the odometry pose estimation module is written to the external memory pool to store the robot's historical motion trajectory information, improving the network's memory capacity. Simultaneously, feature vectors containing the robot's historical motion trajectory information are read from the memory pool and input into the policy output module, improving the network's ability to utilize long-term memory information.

[0378] Step C: Build a strategy output module to output the appropriate motion strategy based on the current sensor information and memory information to control the robot's movement;

[0379] First, the encoding network of the odometry pose estimation network is reused. A feature map is obtained from the current RGB image and input into the observation network. After processing, a feature vector containing the current sensor information is output. The feature vector containing the robot's motion trajectory information read from the external memory pool is then fused with the feature vector output by the observation network. This is used as the input to the policy network, which outputs a motion policy to control the robot's motion. Reusing the encoding network—using the pre-trained odometry pose estimation network to process sensor information—helps reduce the size of the reinforcement learning network and speed up network training.

[0380] Step D: Build an end-to-end reinforcement learning network, construct an integrated network optimization strategy, and perform distributed optimization on it.

[0381] In the end-to-end reinforcement learning network, two adjacent frames of RGB images are input into the odometer pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation Select the feature map m generated by the convolutional network of the current RGB image (t,i) , as the input of the observation network, the network outputs a feature vector a1 containing the sensor information at the current moment; the robot pose estimation Write to the external memory pool, and read the feature vector a2 containing the robot trajectory information from the external memory pool according to the feature vector a1; perform attention feature fusion on the feature vectors a1 and a2 to obtain the fused feature vector a; the fused feature vector a is used as the input of the policy output network, which outputs the policy to control the robot movement; the fused feature vector a is used as the input of the value output network, which outputs the value for reinforcement learning network training.

[0382] During network optimization, the odometry pose estimation network is first pre-trained until it can accurately estimate the robot's pose changes from two adjacent RGB image frames. This reduces the difficulty of training the reinforcement learning network and increases training speed. An integrated network optimization strategy is then constructed to jointly optimize each module.

[0383] Wherein, the step A specifically includes the following steps:

[0384] Step A1: Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate. Specifically, it includes the following steps:

[0385] Step A1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction (encoding) part and an expansion (decoding) part. The contraction part is mainly composed of convolutional layers for feature extraction, while the expansion part is mainly composed of deconvolutional layers for restoring the depth map.

[0386] Step A1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then convolved with each other and processed through a deeper network layer to obtain the robot pose estimate.

[0387] Step A1.3: After convolution, the current RGB image is gradually restored to its original size through deconvolution. The deconvolution-derived feature maps of each size, the feature maps of the corresponding sizes of the shrunk parts, and the upsampled depth map are concatenated as input to the deeper deconvolution layer, ultimately obtaining depth map estimates of different resolutions.

[0388] The step B specifically includes the following steps:

[0389] Step B1: Building a memory module based on an external memory pool according to the Neural Turing Machine (NTM) architecture. Specifically, the following steps are included:

[0390] Step B1.1: Read;

[0391] By position weight vector ω t For the memory matrix M t The N memory units in are weighted summed.

[0392] in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N;

[0393] Therefore, the vector read from the memory at time t is

[0394]

[0395] Step B1.2: Write;

[0396] Each write to the memory matrix can be divided into two steps: erasing and adding, that is, first starting from the memory matrix M at time t-1 t-1 Erase some information and then add new information to generate the memory matrix M at time t t

[0397] Erase:

[0398]

[0399] New:

[0400]

[0401] That is, a write to the memory matrix is ​​completed; wherein, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method.

[0402] Step B1.3: Update the position weight vector ω t ;

[0403] That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: kt , β t 、g t 、s t , γ t The specific steps to update the position weight vector are as follows:

[0404] Step B1.3.1: Content-based addressing, i.e. generating a content-based position weight vector

[0405]

[0406] Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; the similarity measure K calculates the cosine similarity:

[0407]

[0408] Step B1.3.2: Location-Based Addressing

[0409] First is the interpolation operation:

[0410]

[0411] The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation

[0412] Next is the offset operation:

[0413]

[0414] right Each element in It is considered that some elements adjacent to it are related, so the new elements generated by circular convolution It includes and surrounding elements. t is the convolution kernel.

[0415] Finally, the sharpening operation:

[0416]

[0417] When the weights in the offset operation are relatively even, the above convolution operation will lead to data dispersion and leakage, so the difference in weight size needs to be strengthened. That is, the parameter γ generated by the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

[0418] The step C specifically includes the following steps:

[0419] Step C1: Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control robot movement. Specifically, it includes the following steps:

[0420] Step C1.1: Build the observation network; reuse the encoding network of the odometry pose estimation network to obtain the feature map from the current RGB image as the input of the observation network; after processing by the observation network, output a feature vector containing the sensor information at the current moment;

[0421] Step C1.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as input to the policy network, which then outputs a motion policy to control the robot's motion.

[0422] Step C1.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

[0423] The step D specifically includes the following steps:

[0424] Step D1: Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module to couple the functional modules into an end-to-end reinforcement learning network. Specifically, it includes the following steps:

[0425] Step D1.1: Two adjacent RGB images are input into the odometry pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation

[0426] Step D1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) , as the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment;

[0427] Step D1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information;

[0428] Step D1.4: Perform attention feature fusion on feature vectors a1 and a2 to obtain fused feature vector a;

[0429] Step D1.5: Use the fused feature vector a as the input to the policy output network, which outputs the policy to control the robot's motion. Use the fused feature vector a as the input to the value output network, which outputs the value used for reinforcement learning network training.

[0430] Step D2: Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This specifically includes the following steps:

[0431] Step D2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes;

[0432] Step D2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

[0433] Step D3: Build an integrated network optimization strategy and jointly optimize each module;

[0434] Step D3.1: Construct a reinforcement learning reward signal to iterate the robot's end-to-end environment exploration capabilities. The environment will reward or punish the robot for each action. The specific steps are:

[0435] Step D3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the exploration task of this round. c The absolute value of the collision penalty value set;

[0436] Step D3.1.2: Exploration rate increase reward is r search =Ps , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate;

[0437] Step D3.1.3: Repeated exploration penalty is r re-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set;

[0438] Step D3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term can be obs Close when it is greater than a certain threshold;

[0439] Step D3.1.5: The straight reward is r straight =P st , the robot's straight action is conducive to expanding the exploration rate and avoiding repeated exploration, where P st is the value of the straight reward;

[0440] Step D3.1.6: The rotation penalty is r rotate =-P ro , the robot will receive this penalty if it stays near the same position for too long, to avoid the robot rotating near a certain position for a long time. ro is the absolute size of the rotation penalty value;

[0441] Step D3.2: Train the reinforcement learning network in a simulation environment. Network training includes two parts: data collection and training. The specific steps are as follows:

[0442] Step D3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in steps D3.1.1 through D3.1.6 are summed up as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment.

[0443] Step D3.2.2: After each round of data collection, calculate the network loss using the reinforcement learning algorithm and the collected data. Backpropagate the loss through the network to iterate the integrated network parameters constructed in step D1.

[0444] Step D3.2.3: During the reinforcement learning network training process, the robot continuously repeats step D3.2.1 data collection and step D3.2.2 network training until the collection-training reaches the preset number of rounds, thereby forming an optimized robot navigation system based on external memory.

[0445] This invention addresses the long-term memory dependency of current reinforcement learning networks when performing navigation tasks such as environmental exploration. It proposes a robot navigation method and system based on external memory. This system introduces an external memory pool through a neural Turing machine, increasing memory capacity and expanding the functionality of the neural network. This solves the long-term memory dependency issue of existing navigation strategy networks and enhances the robot's perception and decision-making capabilities in unknown environments. Furthermore, the end-to-end integrated network reduces inter-module redundancy and coupling debugging costs, facilitating practical applications.

[0446] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0447] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A robot navigation method based on external memory, characterized in that: include: Step S1: Input two adjacent RGB image frames, process them to obtain multi-scale feature maps, perform inter-convolution operations, and obtain robot pose estimation through deeper network layer processing; Step S2: taking the pose as input and writing it into the external memory pool, and simultaneously reading the feature vector containing the robot motion trajectory information from the memory pool; the external memory pool includes a memory module; the memory module based on the external memory pool: obtaining long-term memory through the external memory pool; Step S3: The feature vector containing the robot motion trajectory information read from the external memory pool is fused with the feature vector output by the observation network as the input of the policy network, and the policy network outputs the motion strategy to control the robot motion; Step S4: Pre-train the odometry pose estimation network, build an integrated network optimization strategy, and jointly optimize each module; The integrated network optimization strategy is constructed, and each module is optimized jointly: Module E3.1: Constructing reinforcement learning reward signals to iterate the robot's end-to-end environment exploration capabilities. The environment rewards or punishes the robot for each action: Module E3.1.1: Collision penalty is r collision =-P c , when the robot touches an object in the environment, it will immediately receive this penalty and end the round of exploration task; c The absolute value of the collision penalty value set; Module E3.1.2: Exploration rate increase reward is r search =P s , when the robot's exploration rate of the environment increases, it will receive this reward, where P s Increase the reward value for the set exploration rate; Module E3.1.3: Repeated exploration penalty is r re-search =-P r , when the robot's repeated exploration rate of the environment increases, it will receive this penalty, where P r The size of the penalty value for repeated exploration is set; Module E3.1.4: The penalty for approaching an obstacle is r danger =-η*d obs , where η is the penalty signal strength parameter, d obs is the straight-line distance to the nearest obstacle from the robot. This penalty term is in d obs Close when it is greater than a certain threshold; Module E3.1.5: The straight reward is r straight =P st , where P st is the value of the straight reward; Module E3.1.6: The penalty for rotation in place is r rotate =-P ro , the robot stays near the same position for too long, and will receive this penalty to avoid the robot rotating near a certain position for a long time, where P ro is the absolute size of the rotation penalty value; Module E3.2: Train reinforcement learning networks in a simulation environment; network training includes data collection and training: Module E3.2.1: The robot freely explores the environment according to the strategy output by the reinforcement learning network, recording the sensor information, actions, strategy, NTM memory matrix, and internal state at each step. The reward and penalty values ​​obtained in modules E3.1.1 through E3.1.6 are summed as the reward for each step and recorded. This data is collected until the robot reaches the preset number of steps or collides with the environment. Module E3.2.2: After each round of data collection, the network loss is calculated using the reinforcement learning algorithm and the collected data. The loss is back-propagated through the network to iterate the integrated network parameters constructed in Module E1. Module E3.2.3: During the reinforcement learning network training process, the robot continuously repeats Module E3.2.1 data acquisition and Module E3.2.2 network training until the acquisition and training reaches the preset number of rounds, forming an optimized robot navigation method based on external memory; In step S1: Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate. Specifically, it includes the following steps: Step S1.1: Two adjacent RGB image frames are fed into the odometry pose estimation network. The network consists of a contraction part and an expansion part. The contraction part consists of a convolutional layer for feature extraction. The expansion part consists of deconvolution layers to restore the depth map; Step S1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate. Step S1.3: After the current RGB image is convolved, it is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution, the feature maps of the corresponding size of the shrunk part, and the depth map obtained by upsampling are concatenated as input to the deeper deconvolution layer to obtain depth map estimates of different resolutions. In step S2: According to the Neural Turing Machine structure, a memory module based on an external memory pool is built. Specifically, the following steps are included: Step S2.1: Read: That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in; in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element; Therefore, the vector r read from the memory at time t t for Step S2.2: Write: Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t : Erase: New: That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method; Step S2.3: Update the position weight vector ω t ; That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ; The specific steps to update the position weight vector are as follows: Step S2.3.1: Content-based addressing, i.e. generating a content-based position weight vector Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element; The formula for calculating the cosine similarity K of vectors u and v is as follows: Step S2.3.2: Location-based addressing: Interpolation operation: The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation Offset operation: right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel; Sharpening operation: Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

2. The robot navigation method based on external memory according to claim 1, characterized in that: In step S3: Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control robot movement. Specifically, it includes the following steps: Step S3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the sensor information at the current moment; Step S3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as input to the policy network, which then outputs a motion policy to control the robot's motion. Step S3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

3. The robot navigation method based on external memory according to claim 1, characterized in that: In step S4: Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network. Specifically, the following steps are included: Step D1.1: Two adjacent RGB images are input into the odometry pose estimation network respectively, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation Step D1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment; Step D1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information; Step D1.4: Perform attention feature fusion on feature vectors a1 and a2 to obtain fused feature vector a; Step D1.5: Use the fused feature vector a as the input of the policy output network, and the network outputs the policy to control the robot motion; The fused feature vector a is used as the input of the value output network, and the network output value is used for reinforcement learning network training; Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB image frames. This includes the following steps: Step D2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes; Step D2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

4. A robot navigation system based on external memory, characterized in that: The method for robot navigation based on external memory according to claim 1 comprises: Odometry pose estimation module: This module estimates the robot's pose changes based on sensor information and inputs the pose into a memory module based on an external memory pool. Strategy output module: outputs appropriate motion strategies based on current sensor information and memory information to control robot movement; End-to-end reinforcement learning network: Integrates the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module to build an integrated network optimization strategy and perform distributed optimization on it.

5. The robot navigation system based on external memory according to claim 4, characterized in that: In the odometry pose estimation module: Build an odometry pose estimation network. The network input is two adjacent RGB images, and the output is the robot pose estimate: Module M1.1: Two adjacent RGB image frames are input into the odometry pose estimation network, which includes a contraction part and an expansion part; The contraction part consists of convolutional layers for feature extraction; The expansion part consists of deconvolution layers to restore the depth map; Module M1.2: Two adjacent RGB images are convolved through multiple convolutional layers to obtain two feature maps. The feature maps are then inter-convolved and processed through a deeper network layer to obtain the robot pose estimate. Module M1.3: After convolution, the current RGB image is gradually restored to its original size through deconvolution. The feature maps of each size obtained by deconvolution are concatenated with the feature maps of the corresponding size of the shrunk part and the depth map obtained by upsampling. These are used as input to deeper deconvolution layers to obtain depth map estimates of different resolutions.

6. The robot navigation system based on external memory according to claim 4, characterized in that: In the external memory pool based memory module: According to the neural Turing machine structure, build a memory module based on the external memory pool: Module M2.1: Read: That is, through the position weight vector ω t For the memory matrix M t Perform weighted summation on the N memory units in; in, is the memory matrix at time t, of size N×M, where N is the number of memory cells and M is the number of memory cells M t (i) length; position weight vector ω t Each element ω t (i) is each memory unit M at time t t (i) The corresponding weight size, and ω t The length of is N; i=1,2,...,N-1,N, is the index of the memory matrix and position weight vector element; Therefore, the vector r read from the memory at time t t for Module M2.2: Write: Each write to the memory matrix can be divided into two steps: erasing and adding. First, the memory matrix M at time t-1 is t-1 Erase the information and then add the information to generate the memory matrix M at time t t : Erase: New: That is, a write to the memory matrix is ​​completed; among them, The memory matrix element obtained by the erase operation, the erase vector e t , add vector a t The length is M, the position weight vector ω t , erase vector e t , add vector a t All are given by the controller, and the parameters can be trained by gradient descent method; Module M2.3: Update position weight vector ω t ; That is, the position weight vector ω at time t-1 t-1 , memory matrix M at time t t And the trainable controller parameters generate the position weight vector ω at time t t ; Among them, the controller parameters include: k t , β t 、g t 、s t , γ t ; Update the position weight vector as follows: Module M2.3.1: Content-based addressing, i.e. generating content-based position weight vectors Among them, k t is a key vector of size 1×M, calculated by the similarity measure K with the memory unit M t (i) similarity; β t Used to amplify or attenuate accuracy; j = 1, 2, ..., N-1, N, is the index of the memory matrix element; The formula for calculating the cosine similarity K of vectors u and v is as follows: Module M2.3.2: Location-Based Addressing: Interpolation operation: The controller generates a threshold g t Weight vector for the current content position and the position weight vector ω at time t-1 t-1 Perform interpolation operation, and the calculation result is the output value of the interpolation operation Offset operation: right Each element in It is considered that some of the elements adjacent to it are related, and the new elements generated by circular convolution are Contains and surrounding elements, where s t is the convolution kernel; Sharpening operation: Strengthen the difference in weight size and generate the parameter γ through the controller t , for each weight Perform exponential operation and then normalize to complete the position weight vector ω t Updates.

7. The robot navigation system based on external memory according to claim 4, characterized in that: In the policy output module: Build a strategy output module to output appropriate motion strategies based on current sensor information and memory information to control the robot's movement: Module M3.1: Build an observation network; reuse the encoding network of the odometry pose estimation network to obtain a feature map from the current RGB image as input to the observation network; after processing by the observation network, output a feature vector containing the current sensor information; Module M3.2: Build a policy output network. This network combines the feature vector containing the robot's motion trajectory information read from the external memory pool with the feature vector output by the observation network as the input to the policy network, which then outputs a motion policy to control the robot's motion. Module M3.3: Build a value output network; the network input is the fused feature vector, and the output is the value used for reinforcement learning network training.

8. The robot navigation system based on external memory according to claim 4, characterized in that: In the end-to-end reinforcement learning network: Integrate the odometry pose estimation module, the memory module based on the external memory pool, and the policy output module, and couple each functional module into an end-to-end reinforcement learning network: Module E1.1: Two adjacent RGB images are input into the odometry pose estimation network, and feature maps of different sizes are obtained through multi-layer convolution. (t,1) ,m (t,2) ,...,m (t,k) , m (t-1,1) ,m (t-1,2) ,...,m (t-1,k) , k is the number of convolution layers; feature map m (t,k) ,m (t-1,k) Mutual convolution and deeper network to obtain robot pose estimation Module E1.2: Reuse the encoding network of the odometry pose estimation module and select the feature map m generated by the encoding network of the current RGB image (t,i) As the input of the observation network; the network outputs a feature vector a1, which contains the sensor information at the current moment; Module E1.3: Estimating the robot pose Write to the external memory pool, and read the feature vector a2 from the external memory pool according to the feature vector a1, which contains the robot's motion trajectory history information; Module E1.4: Feature vectors a1 and a2 are fused with attention features to obtain the fused feature vector a; Module E1.5: The fused feature vector a is used as the input of the policy output network, and the network outputs the policy to control the robot motion; The fused feature vector a is used as the input of the value output network, and the network output value is used for reinforcement learning network training; Network distribution optimization: First, pre-train the odometry pose estimation network until it can accurately estimate the robot pose change from two adjacent RGB images: Module E2.1: Build a simulation environment and collect a dataset containing RGB images, depth maps, and corresponding robot pose changes; Module E2.2: Train the odometry pose estimation network on the dataset; use depth map restoration as an auxiliary task, and use the depth map corresponding to the RGB image and the robot pose as supervision signals. Calculate the loss of multi-scale depth maps and the pose estimation loss, and train the network using backpropagation gradient descent.

Images