A mobile robot mapless autonomous navigation method and system based on a PID structure memory module
By introducing a PID structural memory module and a Twin-Critic network, the problem of DRL navigation methods getting trapped in local optima in complex environments is solved, enabling robots to achieve efficient escape and stable navigation in map-less conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YANSHAN UNIV
- Filing Date
- 2026-01-23
- Publication Date
- 2026-06-02
Smart Images
Figure CN122130070A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mapless navigation technology, specifically to a mapless autonomous navigation method and system for mobile robots based on a PID structure memory module. Background Technology
[0002] With the rapid development of Industrial Internet of Things (IIoT) technology, mobile edge agents are being deployed more and more widely in unstructured environments. Traditional SLAM navigation methods rely on global map construction and cloud interaction, but their adaptability is significantly limited in edge node scenarios with limited communication or insufficient computing power. To address this issue, edge-based mapless navigation methods based on deep reinforcement learning (DRL) have gradually become a research hotspot. These methods learn navigation strategies through real-time interaction between the agent and the environment, eliminating the need for pre-constructing a global map and thus reducing dependence on external communication and computing resources. However, existing DRL navigation methods still face many challenges in complex topological environments.
[0003] In existing technologies, reward function design based on Euclidean distance is a common approach for DRL navigation. However, it is prone to "geometric deception" due to environmental geometric features, causing the agent to become trapped in local optima such as U-shaped traps. Furthermore, while general recurrent networks (such as LSTM) can handle temporal information, they lack specific inductive biases for navigation stalling problems, making it difficult to effectively utilize historical accumulated errors to generate escape strategies. This deficiency makes the agent highly susceptible to deadlock in complex environments, severely limiting the robustness and practicality of the navigation system. Summary of the Invention
[0004] Based on the aforementioned technical problem, existing mobile edge intelligence agents, in map-free navigation tasks, are prone to overestimating value due to the lack of a global map and limitations in onboard computing resources when facing environments with deceptive geometric features. This leads to a local optimum from which they cannot escape. Therefore, this invention provides a map-free autonomous navigation method and system for mobile robots based on a PID structural memory module. This invention mainly utilizes a navigation system architecture based on PIDM-PPO, a multi-task assisted supervised training method, and a Twin-Critic-based anti-interference decision-making approach to achieve the following technical effects: breaking through local optimum deadlock and achieving elastic escape; suppressing value overestimation and improving decision stability; enhancing topology understanding and improving data efficiency.
[0005] The technical means employed in this invention are as follows: A map-free autonomous navigation method for mobile robots based on a PID structure memory module includes the following steps: Acquire the robot's local perception stream data, which includes lidar ranging data and the polar coordinate vector of the target point; A dataset is constructed based on the local perceptual stream data, and the dataset is divided into a training set, a validation set, and a test set; An autonomous navigation model is constructed, which includes an encoder module, a supervision module, a sharing module, a dual-value module, and a strategy module. The encoder module includes a PIDM encoder module, which includes a parallel proportional submodule, an integral submodule, and a micro-molecule module. The autonomous navigation model is trained using the training and validation sets. During training, the temporal reachability loss and future expected reward loss output by the supervision module are used to optimize the encoder module. The dual-value module outputs two value estimates based on two Critic networks, and the minimum value of the two value estimates is found. The minimum value is combined with a value pruning mechanism to calculate the dual-value loss. The policy gradient update of the autonomous navigation model is guided based on the dual-value loss, temporal reachability loss, and future expected reward loss. The test set is input into the trained autonomous navigation model to generate motion commands for the robot.
[0006] Furthermore, the network architecture of the autonomous navigation model includes an encoder module, a sharing module, and a policy module connected in sequence. The data output of the encoder module is connected to the data input of the supervision module, and the data output of the sharing module is connected to the data input of the dual-value module. The workflow of the autonomous navigation model includes: The local perceptual flow model is input into the encoder module to obtain the historical encoded vector; The historical encoded vectors are input into the supervision module and the sharing module respectively. The sharing module outputs the processed historical encoded vectors, and the supervision module outputs the temporal reachability loss and the future expected reward loss. The processed historical encoded vectors are input into the strategy module and the dual-value module, respectively. The strategy module outputs the robot's motion commands, and the dual-value module outputs the dual-value loss.
[0007] Furthermore, the workflow of the encoder module includes: The lidar ranging data and the polar coordinate vector of the target point are passed through the first linear layer to obtain the ranging data and the polar coordinate vector passing through the linear layer. The ranging data passing through the linear layer and the polar coordinate vector passing through the linear layer are fused together to obtain the fused data; The fused data is passed through a second linear layer to obtain encoded input data; The coded input data is passed through the PIDM encoder module to obtain historical encoded vectors.
[0008] Furthermore, the workflow of the PIDM encoder module includes: The scaling submodule processes the encoded input data through a multilayer perceptron to obtain the scaling features of the current time step; The micro-molecule module concatenates the proportional features of the current time step and the proportional features of the previous time step, and then maps the concatenated features to the differential features of the current time step through the network. The integral submodule, on the one hand, passes the proportional features of the current time step and the memory of the previous time step sequentially through the third linear layer and the Sigmoid function to obtain the update gate; on the other hand, it passes the proportional features of the current time step and the memory of the previous time step sequentially through the fourth linear layer and the Tanh function to obtain the candidate memory. At the same time, based on the update gate, the memory of the previous time step and the candidate memory, it calculates the memory of the current time step, and passes the memory of the current time step sequentially through the fifth linear layer and the Tanh function to obtain the integral features of the current time step. The proportional features, differential features, and integral features of the current time step are sequentially passed through the sixth linear layer and the Softmax function to obtain the weights corresponding to each feature. The proportional features, differential features, and integral features of the current time step are used to generate fused features based on the weights. The fused features, the memory of the current time step, and the differential features of the current time step are then passed through the seventh linear layer and the Tanh function to obtain the historical encoding vector.
[0009] Furthermore, the workflow of the supervision module includes: The historical encoding vector is used to assist the prediction network to obtain the time-series reachability prediction value and the future expected return prediction value. The auxiliary prediction network includes a fourteenth linear layer, a ReLU activation function, and a fifteenth linear layer connected in sequence. A timing truncation mask is established, and the formula for calculating the timing truncation mask is as follows:
[0010] in, M t For timing truncation mask, t For a moment, T end For moments of failure, K To truncate the window size; Based on the temporal truncation mask and the predicted temporal reachability value, the temporal reachability loss is calculated using the following formula:
[0011] in, For time-series reachability loss, For historical encoding vectors, For expectation operator, It is the Sigmoid activation function. The predicted time-series reachability value. The normalized remaining time label; Based on historical encoding vectors and generalized advantage estimation of returns, the expected future return loss is calculated. The formula for calculating the expected future return loss is as follows:
[0012] in, Loss of expected future returns This is a forecast of future expected returns. Estimate returns for generalized advantage.
[0013] Furthermore, the operation of the shared module includes: The historical encoding vector and the encoded input data are sequentially passed through the eighth linear layer, the ReLU function, the ninth linear layer, and the Tanh function to obtain the processed historical encoding vector.
[0014] Furthermore, the workflow of the dual-value module includes: The processed historical encoded vector is passed through the tenth linear layer, and the first value estimate and the first linearly processed historical encoded vector are output. The historical encoded vector after the first linear processing is passed through the eleventh linear layer to output the second value estimate; The minimum value is selected from the first value estimate and the second value estimate. The minimum value is then combined with a value trimming mechanism to calculate the double value loss. The formula for calculating the double value loss is as follows:
[0015] in, This results in a double loss of value. For expectation operator, A conservative value estimate for the current state. For the valuation of the cropped product, Estimate returns for generalized advantage.
[0016] Furthermore, the workflow of the strategy module includes: The processed historical encoding vector is passed through the twelfth linear layer to obtain the mean of the action probability distribution and the second linearly processed historical encoding vector; The historical encoded vector after the second linear processing is passed through the thirteenth linear layer to obtain the logarithm of the standard deviation of the action probability distribution; The standard deviation is obtained by exponentially calculating the logarithm of the standard deviation of the action probability distribution:
[0017] in, Standard deviation This is the logarithm of the standard deviation of the probability distribution of the actions; Construct a Gaussian distribution using the mean and standard deviation of the action probability distribution; The original action vector is obtained by sampling from a Gaussian distribution, and the formula for calculating the original action vector is as follows:
[0018] in, For the original action vector, Let be the mean of the probability distribution of the actions. It is noise.
[0019] Furthermore, the policy gradient update of the autonomous navigation model guided by the dual value loss, temporal reachability loss, and future expected return loss includes: A total loss function is constructed based on the dual value loss, temporal reachability loss, and future expected return loss. This total loss function guides the policy gradient update of the autonomous navigation model. The formula for calculating the total loss function is as follows:
[0020] in, For the total loss function, The objective function for policy optimization is... This is the weighting coefficient for value loss. This results in a double loss of value. For entropy loss, It is the sum of the expected future return loss and the time-series accessibility loss.
[0021] This invention also includes a map-free autonomous navigation system for mobile robots based on a PID structure memory module, used to implement a map-free autonomous navigation method for mobile robots based on a PID structure memory module, comprising: An environmental perception module is used to acquire the robot's local perception stream data, which includes lidar ranging data and the polar coordinate vector of the target point. A dataset construction module, which constructs a dataset based on the local perceptual stream data and divides the dataset into a training set, a validation set, and a test set; An autonomous navigation model, comprising an encoder module, a supervision module, a sharing module, a dual-value module, and a policy module, wherein: The encoder module includes parallel proportional submodules, integral submodules, and micro-molecule modules, used to construct historical encoding vectors to simulate the physical characteristics of a PID controller; The dual-value module outputs value estimates based on two Critic networks and employs a minimum value strategy to estimate the value and output the value loss. The supervision module is used to calculate the reachability difference and the expected future return difference between the current state and the target state, and output the loss value. The strategy module generates motion commands for the robot; The training and optimization module uses the training set and validation set to train the autonomous navigation model. During training, it optimizes the autonomous navigation model based on the value loss and the loss value output by the supervision module.
[0022] Compared with the prior art, the present invention has the following advantages: 1. This invention innovatively introduces the concept of integral error elimination from classical PID control theory into deep reinforcement learning. Through the designed PIDM integral module (I-Module), the agent can explicitly accumulate historical errors in the feature space when it is stuck in a dead end or U-shaped trap and stagnates for a long time. Thus, even when no reward gain can be obtained, it can spontaneously generate an integral control gain sufficient to break the deadlock, significantly improving the robot's escape ability.
[0023] 2. This invention addresses the geometric deception problem caused by proximity to the target in trap environments. It employs a Twin-Critic dual-value network architecture. By conservatively estimating the value by taking the minimum of the two Critic network evaluations, it effectively cuts off false high-value signals induced by local geometric features, preventing the policy network from being misled into seemingly high-value but actually unreachable trap regions. This significantly improves the robustness of the algorithm in non-convex environments.
[0024] 3. This invention designs self-supervised auxiliary tasks (L-Loss and G-Loss) that do not require manual annotation. By utilizing locally cached historical trajectory data for hindsight learning, the agent is forced to distinguish between physical proximity and temporal reachability in the latent space, enabling it to learn the topological connectivity of the environment even without a map. Experiments show that this mechanism significantly improves the algorithm's sample efficiency and convergence speed.
[0025] Based on the above reasons, this invention can be widely promoted in fields such as mapless navigation. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a schematic diagram of the mapless navigation environment of the present invention.
[0028] Figure 2 This is a framework diagram of a mapless autonomous navigation method for mobile robots based on a PID structure memory module according to the present invention.
[0029] Figure 3 This is a schematic diagram of the encoder module of the present invention.
[0030] Figure 4 This is a schematic diagram illustrating the loss calculation of the present invention.
[0031] Figure 5 This is a simulation map for the present invention.
[0032] Figure 6 This is a diagram illustrating the success rate comparison of the present invention.
[0033] Figure 7 This is a schematic diagram comparing the reward values of the present invention. Detailed Implementation
[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0036] This invention provides a mapless autonomous navigation method for mobile robots based on a PID structure memory module. This method is used in a mapless navigation system running on an edge computing node of a mobile robot. Figure 2 As shown, the system mainly consists of an environmental perception module, a PIDM structured encoder module, and a Twin-Critic-based PPO decision module (e.g., Figure 2 (As shown). The method includes the following steps: S1. Acquire the robot's local perception stream data, which includes lidar ranging data and the polar coordinate vector of the target point, as the observation input of the system.
[0037] The first module is the environmental perception module, which is responsible for collecting the robot's local perception stream data. In this embodiment, the sensing stream data is structured and defined as follows: .
[0038] in, This is the original LiDAR ranging data stream. This is the polar coordinate vector of the robot's current position relative to the target point, including the relative distance and azimuth. This module does not rely on a global map or external positioning system (such as GPS), but only utilizes onboard sensor data, adapting to environments with limited communication (such as...). Figure 1 (As shown).
[0039] S2. Construct a dataset based on local perceptual streaming data, and divide the dataset into training set, validation set and test set.
[0040] S3. Construct an autonomous navigation model, which includes an encoder module, a supervision module, a sharing module, a dual-value module, and a strategy module. The encoder module includes a PIDM encoder module, which includes parallel proportional submodules, integral submodules, and micro-molecule modules.
[0041] The PIDM module (PID-Memory) is a neural network framework. Its update and data processing methods are based on the PID controller structure (which is a different field from the traditional PID controller; it borrows the idea of PID to handle problems, but is essentially a neural network).
[0042] The network architecture of the autonomous navigation model includes an encoder module, a sharing module, and a policy module connected in sequence. The data output of the encoder module is connected to the data input of the supervision module, and the data output of the sharing module is connected to the data input of the dual-value module.
[0043] The workflow of an autonomous navigation model includes: The first step is to input the local perceptual flow model into the encoder module to obtain the historical encoded vector.
[0044] The second step involves inputting the historical encoded vectors into the supervision module and the sharing module, respectively. The sharing module outputs the processed historical encoded vectors, while the supervision module outputs the temporal reachability loss and the future expected return loss.
[0045] The third step is to input the processed historical encoded vectors into the strategy module and the dual-value module respectively. The strategy module outputs the robot's motion commands, and the dual-value module outputs the dual-value loss.
[0046] The encoder module's workflow includes: The first step is to pass the lidar ranging data and the polar coordinate vector of the target point through the first linear layer to obtain the ranging data and polar coordinate vector through the linear layer.
[0047] The second step involves fusing the ranging data from the linear layer and the polar coordinate vector from the linear layer together to obtain the fused data. As a preferred embodiment of this invention, the fusion process is preferably a stitching process.
[0048] The third step is to pass the fused data through the second linear layer to obtain the encoded input data.
[0049] The fourth step is to pass the coded input data through the PIDM encoder module to obtain the historical coded vector.
[0050] Furthermore, the PIDM structured encoder module is the core component of this invention, used to address the problem that traditional RNNs lack error accumulation capabilities in local optima such as dead ends. This module simulates the physical characteristics of a PID controller and includes three parallel sub-modules: P (proportional), I (integral), and D (derivative). Figure 3 (As shown). The workflow of the PIDM encoder module includes: The first step, the Proportional Submodule (P-Module), is used to extract instantaneous state features. The encoded input data is processed by a Multilayer Perceptron (MLP). Obtain the proportional features of the current time step. :
[0051] in, The proportion of the current time step. A fully connected layer activated by ReLU. Input data for encoding.
[0052] The second step involves using the micro-molecule module (D-Module) to sense motion trends. The proportional features of the current time step and the proportional features of the previous time step are concatenated (Concat), and then processed by the network. Mapped to the differential features (discretionary features) of the current time step. :
[0053] in, The proportion of the previous time step, The differential feature at the current time step, This is a differential mapping network, consisting of a linear layer and a Tanh activation function. This module captures the rate of change of the state, similar to the derivative term in a PID controller, and is used to predict environmental dynamics.
[0054] The third step involves an integral submodule used to memorize historical navigation deviations. This embodiment designs a nonlinear gated integral mechanism. On one hand, the proportional features of the current time step and the memory of the previous time step are sequentially passed through the third linear layer and the Sigmoid function to obtain the update gate. On the other hand, the proportional features of the current time step and the memory of the previous time step are sequentially passed through the fourth linear layer and the Tanh function to obtain candidate memories.
[0055] The formula for updating the door is:
[0056] in, To update the door, It is the Sigmoid activation function. It is the third linear layer. This is a memory of the previous time step.
[0057] The formula for calculating candidate memories is:
[0058] in, For candidate memories, For the Tanh function, This is the fourth linear layer.
[0059] Simultaneously, based on the update gate, the memory of the previous time step, and candidate memories, the memory of the current time step is calculated:
[0060] in, This is the memory of the current time step.
[0061] By passing the memory of the current time step sequentially through the fifth linear layer and the Tanh function, the integral feature of the current time step is obtained:
[0062] in, The integral feature at the current time step, This is an integral mapping network, which consists of a fifth linear layer and a Tanh activation function.
[0063] Technical effect description: When the robot gets stuck in a deadlock (such as wandering at the bottom of a U-trap), the update gate tends to 0, allowing the memory of the current time step to be maintained and historical errors to be accumulated (simulating steady-state errors). This accumulation effect can generate an integral control gain, driving the policy network to perform turning or backtracking actions without an immediate increase in reward, thereby breaking local optima.
[0064] Fourth step: Pass the proportional features, differential features, and integral features of the current time step through the sixth linear layer and the Softmax function in sequence to obtain the weights corresponding to each feature. .
[0065] Fifth, the proportional features, differential features, and integral features of the current time step are used to generate fusion features based on weights. The fusion features, the memory of the current time step, and the differential features of the current time step are then passed through the seventh linear layer and the Tanh function to obtain the historical encoding vector.
[0066] The formula for calculating the fusion feature is:
[0067] in, As a feature of fusion, The weights corresponding to the proportional features. The weights corresponding to the differential features. These are the weights corresponding to the integral features.
[0068] The working process of the shared module includes: The historical encoding vector and the encoded input data are passed sequentially through the eighth linear layer, the ReLU function, the ninth linear layer, and the Tanh function to obtain the processed historical encoding vector.
[0069] The workflow of the strategy module includes: The first step is to pass the processed historical encoding vector through the twelfth linear layer to obtain the mean of the action probability distribution and the second linearly processed historical encoding vector.
[0070] The second step is to pass the historical encoding vector after the second linear processing through the thirteenth linear layer to obtain the logarithm of the standard deviation of the action probability distribution.
[0071] The third step is to perform an exponential operation on the logarithm of the standard deviation of the action probability distribution to obtain the actual standard deviation:
[0072] in, Standard deviation It is the logarithm of the standard deviation of the probability distribution of the actions.
[0073] Step 4: Construct a multidimensional Gaussian distribution (normal distribution) using the mean and standard deviation of the action probability distribution:
[0074] in, It is a multidimensional Gaussian distribution. This represents the mean of the probability distribution of the actions.
[0075] Step 5: Sample the original action vector from the Gaussian distribution. In actual calculations, noise that follows a standard normal distribution is usually introduced. The formula for calculating the original action vector is:
[0076] in, For the original action vector, It is noise.
[0077] Generated action vectors It is a continuous vector that directly corresponds to the robot's kinematic control variables. That is:
[0078] in, Let the linear velocity of the robot be . Let be the robot's angular velocity.
[0079] S4. Train the autonomous navigation model using the training and validation sets. During training, optimize the encoder module using the temporal reachability loss and future expected reward loss output by the supervision module. The dual-value module outputs two value estimates based on two Critic networks. Find the minimum value between the two value estimates and calculate the dual-value loss using the minimum value combined with a value pruning mechanism. Guide the policy gradient update of the autonomous navigation model based on the dual-value loss, temporal reachability loss, and future expected reward loss.
[0080] To train the PIDM encoder in a map-less and reward-sparse environment, this embodiment introduces a self-supervised auxiliary mechanism (such as...). Figure 4 (As shown). The workflow of the supervision module includes: The first step is to use the historical encoding vector to assist the prediction network to obtain the time-series reachability prediction value and the future expected return prediction value. The assist prediction network includes a fourteenth linear layer, a ReLU activation function, and a fifteenth linear layer connected in sequence.
[0081] First, the temporal reachability prediction task (L-Loss) constructs an auxiliary network to predict the expected time to reach the target. To address the misleading problem of early failure samples, this embodiment innovatively introduces a temporal truncation mask mechanism, establishing a temporal truncation mask. The formula for calculating the temporal truncation mask is as follows:
[0082] in, M t For timing truncation mask, t For a moment, T end For moments of failure, K To truncate the window size.
[0083] The second step is to calculate the temporal reachability loss based on the temporal truncation mask and historical coding vectors. The formula for calculating the temporal reachability loss is as follows:
[0084] in, For time-series reachability loss, For historical encoding vectors, For expectation operator, It is the Sigmoid activation function. The predicted time-series reachability value. The normalized remaining time label.
[0085] By using a mask, only the last step of the nearest collision in the failed trajectory is retained as a negative sample. This prevents the robot from being incorrectly marked as unreachable when it first enters a trap (when the geometric distance to the target is very close), thus preventing gradient misdirection and guiding the encoder to correctly learn the topological connectivity of the environment rather than just the geometric distance.
[0086] The third step is to calculate the future expected return loss based on the historical encoding vector and the generalized advantage estimate (GAE) return. The formula for calculating the future expected return loss is as follows:
[0087] in, Loss of expected future returns This is a forecast of future expected returns. Estimate the return for generalized advantage. This guides the encoder to focus on features that are highly valuable for long-term task success.
[0088] Furthermore, the generalized advantage estimates the return. The calculation process is as follows: Step 1: Calculate the Temporal Difference Error (TD Error). Based on the immediate reward from environmental feedback and the value network's estimate of the state's value, calculate the temporal difference error at the current time step. :
[0089] in, The immediate reward a robot receives after performing an action. and These are the Critic network's value estimates for the current state and the next state, respectively. This is a reward discount factor.
[0090] The second step is to calculate the generalized dominance estimate. Based on the time-series difference error, a weighted accumulation method is used to calculate the generalized dominance estimate. :
[0091] in, This is the GAE smoothing coefficient, used to balance bias and variance. The length of the sampling trajectory.
[0092] The third step is to calculate the generalized advantage estimate return. The generalized advantage estimate is added to the current state's value estimate to obtain the final monitoring target. :
[0093] To address the issue of overvaluation due to geometric deception in trap environments, this embodiment extends the PPO algorithm with a Twin-Critic value network architecture. The Twin-Critic structure consists of two identical but independently initialized Critic networks. For the same state feature, they output value estimates V1 and V2 respectively.
[0094] Figure 4 middle, This represents the action probability distribution output by the current policy network (New Policy). The old policy log probability is the action log probability output by the network at the sampling time. This is a generalized advantage estimate, used to measure the degree of advantage of the current action relative to the average level. The conservative value estimate output by the current Twin-Critic network (i.e.) ), The value estimate of the old value network at the sampling time. The Generalized Advantage Estimation Return (GAE Return) serves as the target truth value for value learning. To assist the network in outputting time-series reachability predictions (i.e., the ones mentioned earlier) ), To assist the network in outputting the predicted future returns (i.e., the values mentioned above) ), The true temporal reachability label calculated based on the temporal truncation mask (i.e., the one mentioned above) ), This is the entropy regularization coefficient, used to control the degree of exploration.
[0095] Figure 4 The process includes: First, calculate the ratio of the old and new strategies. :
[0096] in, The probability of the action output by the current policy network. The probability of the action output by the old policy network at the sampling time.
[0097] Then calculate the surrogate objective function after pruning:
[0098] Where clip is the truncation function. This is the cutoff range (usually set to 0.2). For expectation operators.
[0099] Calculate the entropy of the current policy distribution. :
[0100] in, Represents information entropy. for, for.
[0101] Calculate the squared error before trimming and the squared error after trimming, and take the larger of the two:
[0102] in, A conservative value estimate for the current state. This results in a double loss of value. Estimate returns for generalized advantage. For the valuation of the cropped product, , The value estimate of the old value network at the sampling time.
[0103] Auxiliary task loss ( The value of the learning environment topology and task is used to supervise the PIDM encoder learning environment, and consists of two parts: mean squared error (MSE).
[0104]
[0105]
[0106] in, and For balance coefficient, For time-series reachability loss, Loss of expected future returns This is a time-series reachability prediction value. For time-series reachability real labels, This represents the projected future return.
[0107] Finally, the model sums the four losses in a weighted manner to obtain the total loss function used for backpropagation.
[0108]
[0109] in, For the total loss function, For the proxy objective function, This is the weighting coefficient for value loss. This results in a double loss of value. For entropy loss, It is the sum of the expected future return loss and the time-series accessibility loss.
[0110] Furthermore, the workflow of the dual-value module includes: The first step is to pass the processed historical encoding vector through the tenth linear layer to output the first value estimate V1 and the first linearly processed historical encoding vector.
[0111] The second step is to pass the historical encoded vector after the first linear processing through the eleventh linear layer to output the second value estimate V2.
[0112] Third, when calculating the advantage function and updating the objective, always take the minimum of the two as the value estimate of the current state. Inside the U-shaped trap, a single Critic network is prone to overestimating the value as the robot gets closer to the objective. Using a minimum value strategy can effectively truncate this false optimistic estimate, forcing the policy network to recognize the low value of the current state, thereby avoiding ineffective wandering inside the trap.
[0113] The minimum value is selected from the first and second value estimates. The double value loss is calculated using the minimum value combined with a value trimming mechanism. The formula for calculating the double value loss is as follows:
[0114] in, This results in a double loss of value. For expectation operator, A conservative value estimate for the current state. For the valuation of the cropped product, Estimate returns for generalized advantage.
[0115] A total loss function is constructed based on the dual value loss, temporal reachability loss, and future expected return loss. This total loss function guides the policy gradient update of the autonomous navigation model. The formula for calculating the total loss function is as follows:
[0116] in, For the total loss function, For the proxy objective function, This is the weighting coefficient for value loss. This results in a double loss of value. For entropy loss, This is the sum of the dual value loss and the temporal reachability loss. By jointly optimizing the total loss, end-to-end policy updates are achieved.
[0117] S5. Input the test set into the trained autonomous navigation model to generate motion commands for the robot.
[0118] This invention also includes a map-free autonomous navigation system for mobile robots based on a PID structure memory module, used to implement the aforementioned map-free autonomous navigation method for mobile robots based on a PID structure memory module, comprising: The environmental perception module is used to acquire the robot's local perception stream data, which includes LiDAR ranging data and the polar coordinate vector of the target point.
[0119] The dataset construction module builds a dataset based on local perceptual streaming data and divides the dataset into training, validation, and test sets.
[0120] The autonomous navigation model comprises an encoder module, a supervision module, a sharing module, a dual-value module, and a policy module, wherein: The encoder module, which includes parallel proportional submodules, integral submodules, and micro-molecule modules, is used to construct historical encoding vectors to simulate the physical characteristics of the PID controller.
[0121] The dual-value module outputs value estimates based on two Critic networks and employs a minimum value strategy to estimate the value, outputting the value loss.
[0122] The supervision module is used to calculate the reachability difference and the expected future reward difference between the current state and the target state, and outputs the loss value.
[0123] The strategy module generates motion commands for the robot.
[0124] The training and optimization module uses the training and validation sets to train the autonomous navigation model. During training, the model is optimized based on the value loss and the loss value output by the supervision module.
[0125] This invention has been validated in a simulation environment (PyBullet).
[0126] Simulation Results: In test scenarios containing complex U-shaped traps and dead ends, the PIDM-PPO algorithm maintains a stable navigation success rate of around 55%, a significant improvement compared to the baseline methods LSTM-PPO (approximately 50%) and GRU-PPO (approximately 20%). Figure 6 (As shown in Figure 7). Especially in complex, unseen maps, the generalization success rate was improved by 15.4% (e.g., Figure 5 (As shown).
[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A map-free autonomous navigation method for mobile robots based on a PID structure memory module, characterized in that, Includes the following steps: Acquire the robot's local perception stream data, which includes lidar ranging data and the polar coordinate vector of the target point; A dataset is constructed based on the local perceptual stream data, and the dataset is divided into a training set, a validation set, and a test set; An autonomous navigation model is constructed, which includes an encoder module, a supervision module, a sharing module, a dual-value module, and a strategy module. The encoder module includes a PIDM encoder module, which includes a parallel proportional submodule, an integral submodule, and a micro-molecule module. The autonomous navigation model is trained using the training and validation sets. During training, the temporal reachability loss and future expected reward loss output by the supervision module are used to optimize the encoder module. The dual-value module outputs two value estimates based on two Critic networks, and the minimum value of the two value estimates is found. The minimum value is combined with a value pruning mechanism to calculate the dual-value loss. The policy gradient update of the autonomous navigation model is guided based on the dual-value loss, temporal reachability loss, and future expected reward loss. The test set is input into the trained autonomous navigation model to generate motion commands for the robot.
2. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 1, characterized in that, The network architecture of the autonomous navigation model includes an encoder module, a sharing module, and a policy module connected in sequence. The data output of the encoder module is connected to the data input of the supervision module, and the data output of the sharing module is connected to the data input of the dual-value module. The workflow of the autonomous navigation model includes: The local perceptual flow model is input into the encoder module to obtain the historical encoded vector; The historical encoded vectors are input into the supervision module and the sharing module respectively. The sharing module outputs the processed historical encoded vectors, and the supervision module outputs the temporal reachability loss and the future expected reward loss. The processed historical encoded vectors are input into the strategy module and the dual-value module, respectively. The strategy module outputs the robot's motion commands, and the dual-value module outputs the dual-value loss.
3. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 2, characterized in that, The workflow of the encoder module includes: The lidar ranging data and the polar coordinate vector of the target point are passed through the first linear layer to obtain the ranging data and the polar coordinate vector passing through the linear layer. The ranging data passing through the linear layer and the polar coordinate vector passing through the linear layer are fused together to obtain the fused data; The fused data is passed through a second linear layer to obtain encoded input data; The coded input data is passed through the PIDM encoder module to obtain historical encoded vectors.
4. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 3, characterized in that, The workflow of the PIDM encoder module includes: The scaling submodule processes the encoded input data through a multilayer perceptron to obtain the scaling features of the current time step; The micro-molecule module concatenates the proportional features of the current time step and the proportional features of the previous time step, and then maps the concatenated features to the differential features of the current time step through the network. The integral submodule, on the one hand, passes the proportional features of the current time step and the memory of the previous time step sequentially through the third linear layer and the Sigmoid function to obtain the update gate; on the other hand, it passes the proportional features of the current time step and the memory of the previous time step sequentially through the fourth linear layer and the Tanh function to obtain the candidate memory. At the same time, based on the update gate, the memory of the previous time step and the candidate memory, it calculates the memory of the current time step, and passes the memory of the current time step sequentially through the fifth linear layer and the Tanh function to obtain the integral features of the current time step. The proportional features, differential features, and integral features of the current time step are sequentially passed through the sixth linear layer and the Softmax function to obtain the weights corresponding to each feature. The proportional features, differential features, and integral features of the current time step are used to generate fused features based on the weights. The fused features, the memory of the current time step, and the differential features of the current time step are then passed through the seventh linear layer and the Tanh function to obtain the historical encoding vector.
5. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 2, characterized in that, The workflow of the supervision module includes: The historical encoding vector is used to assist the prediction network to obtain the time-series reachability prediction value and the future expected return prediction value. The auxiliary prediction network includes a fourteenth linear layer, a ReLU activation function, and a fifteenth linear layer connected in sequence. A timing truncation mask is established, and the formula for calculating the timing truncation mask is as follows: in, M t For timing truncation mask, t For a moment, T end For moments of failure, K To truncate the window size; Based on the temporal truncation mask and the predicted temporal reachability value, the temporal reachability loss is calculated using the following formula: in, For time-series reachability loss, For historical encoding vectors, For expectation operator, It is the Sigmoid activation function. The predicted time-series reachability value. The normalized remaining time label; Based on historical encoding vectors and generalized advantage estimation of returns, the expected future return loss is calculated. The formula for calculating the expected future return loss is as follows: in, Loss of expected future returns This is a forecast of future expected returns. Estimate returns for generalized advantage.
6. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 2, characterized in that, The working process of the shared module includes: The historical encoding vector and the encoded input data are sequentially passed through the eighth linear layer, the ReLU function, the ninth linear layer, and the Tanh function to obtain the processed historical encoding vector.
7. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 2, characterized in that, The workflow of the dual-value module includes: The processed historical encoded vector is passed through the tenth linear layer, and the first value estimate and the first linearly processed historical encoded vector are output. The historical encoded vector after the first linear processing is passed through the eleventh linear layer to output the second value estimate; The minimum value is selected from the first value estimate and the second value estimate. The minimum value is then combined with a value trimming mechanism to calculate the double value loss. The formula for calculating the double value loss is as follows: in, This results in a double loss of value. For expectation operator, A conservative value estimate for the current state. For the valuation of the cropped product, Estimate returns for generalized advantage.
8. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 2, characterized in that, The workflow of the strategy module includes: The processed historical encoding vector is passed through the twelfth linear layer to obtain the mean of the action probability distribution and the second linearly processed historical encoding vector; The historical encoded vector after the second linear processing is passed through the thirteenth linear layer to obtain the logarithm of the standard deviation of the action probability distribution; The standard deviation is obtained by exponentially calculating the logarithm of the standard deviation of the action probability distribution: in, Standard deviation This is the logarithm of the standard deviation of the probability distribution of the actions; Construct a Gaussian distribution using the mean and standard deviation of the action probability distribution; The original action vector is obtained by sampling from a Gaussian distribution, and the formula for calculating the original action vector is as follows: in, For the original action vector, Let be the mean of the probability distribution of the actions. It is noise.
9. The map-free autonomous navigation method for mobile robots based on a PID structure memory module according to claim 1, characterized in that, The policy gradient update of the autonomous navigation model, guided by dual value loss, temporal reachability loss, and future expected return loss, includes: A total loss function is constructed based on the dual value loss, temporal reachability loss, and future expected return loss. This total loss function guides the policy gradient update of the autonomous navigation model. The formula for calculating the total loss function is as follows: in, For the total loss function, The objective function for policy optimization is... This is the weighting coefficient for value loss. This results in a double loss of value. For entropy loss, It is the sum of the expected future return loss and the time-series accessibility loss.
10. A mapless autonomous navigation system for a mobile robot based on a PID structure memory module, used to implement the mapless autonomous navigation method for a mobile robot based on a PID structure memory module as described in any one of claims 1-9, characterized in that, include: An environmental perception module is used to acquire the robot's local perception stream data, which includes lidar ranging data and the polar coordinate vector of the target point. A dataset construction module, which constructs a dataset based on the local perceptual stream data and divides the dataset into a training set, a validation set, and a test set; An autonomous navigation model, comprising an encoder module, a supervision module, a sharing module, a dual-value module, and a policy module, wherein: The encoder module includes parallel proportional submodules, integral submodules, and micro-molecule modules, used to construct historical encoding vectors to simulate the physical characteristics of a PID controller; The dual-value module outputs value estimates based on two Critic networks and employs a minimum value strategy to estimate the value and output the value loss. The supervision module is used to calculate the reachability difference and the expected future return difference between the current state and the target state, and output the loss value. The strategy module generates motion commands for the robot; The training and optimization module uses the training set and validation set to train the autonomous navigation model. During training, it optimizes the autonomous navigation model based on the value loss and the loss value output by the supervision module.