A method for training a robot navigation model for an indoor environment and applications
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2026-02-06
- Publication Date
- 2026-06-09
Smart Images
Figure CN122174913A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot navigation technology, and more specifically, relates to a training method and application of a robot navigation model for indoor environments. Background Technology
[0002] In modern intelligent manufacturing, smart warehousing, and home services, the autonomous navigation capabilities of indoor consumer mobile robots play a crucial role in improving operational efficiency and reducing labor costs. Practical applications place stringent demands on navigation systems in terms of safety, efficiency, and robustness, which continuously drives the development of advanced navigation algorithms.
[0003] Indoor navigation performance is significantly constrained by the complex structure of the environment. Typical indoor spaces often contain a large number of dense static obstacles, such as walls and fixed furniture, creating narrow passages, sharp turns, and areas of visual obstruction. Robots need to perform reliable path planning and real-time obstacle avoidance within limited maneuver space. If the algorithm is not adaptable enough to these constraints, it can easily lead to path congestion, planning failures, or collisions, thus severely reducing the reliability of the navigation system.
[0004] Furthermore, the uncertainty of the target location and the robot's initial state is a fundamental challenge in real-world applications. In scenarios such as home services and warehousing logistics, task objectives are often dynamically generated, and prior global map information or predefined paths are usually unavailable. Therefore, navigation algorithms must rely solely on real-time perception information to quickly generate feasible and safe trajectories leading to randomly distributed target points.
[0005] Traditional indoor navigation methods can be broadly categorized into two types: global planning methods and local planning methods. Global planning methods rely on pre-built environmental maps and heuristic search strategies, limiting their applicability in unknown environments. While local planning methods possess a certain degree of real-time responsiveness, they are prone to getting trapped in local optima and struggle to strike a balance between quickly reaching the target and safely avoiding obstacles. Furthermore, these methods typically rely heavily on manually designed heuristic rules, resulting in limited generalization capabilities in complex indoor scenarios.
[0006] In recent years, Deep Reinforcement Learning (DRL) has gradually become an important alternative in the field of indoor robot navigation due to its end-to-end decision-making capabilities and its ability to learn through autonomous interaction with the environment. However, due to the complexity of indoor environments, existing reinforcement learning algorithms often struggle to establish an understanding of target rewards in the early stages of training, requiring extensive learning and thus exhibiting slow convergence speeds in the initial training phase. Summary of the Invention
[0007] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a training method and application for a robot navigation model for indoor environments. Its purpose is to improve the convergence speed of model training while ensuring the navigation accuracy of the robot navigation model.
[0008] To achieve the above objectives, in a first aspect, the present invention provides a method for training a robot navigation model for indoor environments, comprising: Initialize global distance threshold The robot navigation model is trained through one or more rounds; each round of training includes: S1. Initialize training rounds ; S2. Randomly initialize indoor environmental parameters; randomly initialize the robot's initial position and target position, with the distance between the initial position and target position being less than or equal to... ;make ; S3, the first robot state observations at time 1 Input into the robot navigation model to obtain the first Current robot movement commands And obtain the robot's movement instructions. Post-state observations and reward value ;Will , , and The quaternion formed as the first The data sample obtained at time is denoted as . The data is stored in the experience replay pool; the state observations include: the relative position information of the robot and the target position; S4. Determine if the termination condition is met. If yes, proceed to S5; otherwise, let... Proceed to S3; Termination conditions include: the robot executes the movement command. A collision may occur later, or the robot may execute a movement command. Then arrive at the target location, or, Reaching the preset time step; S5. When the robot executes a movement command Upon reaching the target location, ; Preset distance step size; The preset maximum distance; judge Is it an integer multiple of the preset number of rounds? If yes, proceed to S6; otherwise, proceed to S7. S6. Extract data samples from the experience replay pool to form a training sample set; based on the training sample set, use a reinforcement learning algorithm to train the robot navigation model; S7, Judgment Is it less than the preset round threshold M? If so, then let... If the current round of training ends, proceed to S2; otherwise, end the current round of training.
[0009] More preferably, S3 further includes: upon receiving the robot's movement command... Next, obtain the robot's movement instructions. Post-state observations and reward value Previously, based on the instructions for movement... Random sampling is performed using a Gaussian distribution with a mean of less than a preset threshold and a variance of less than a preset threshold to obtain new movement commands, and then... Update.
[0010] More preferably, the robot navigation model includes: a feature extraction module and a mapping module; the feature extraction module is used to extract features from the input robot state observations; the mapping module is used to map the features obtained by the feature extraction module into corresponding movement commands; wherein, the feature extraction module includes: an LSTM model.
[0011] More preferably, the LSTM model includes: multiple cascaded LSTM units; the LSTM unit includes: an input gate, an output gate, and a forget gate; Among them, regarding the time step When inputting data, the input gate, output gate, and forget gate perform the following processing respectively:
[0012]
[0013]
[0014] Time step candidate states for:
[0015] Time step Cellular state for:
[0016] Time step The intermediate hidden state for:
[0017] Time step Hidden state for:
[0018] in, For time steps Output of the lower input gate; This represents the weight matrix input to the input gate; Indicates time step The input vector of the lower LSTM unit; The weight matrix represents the weights from the hidden state to the input gate; Indicates time step The hidden state below; Indicates the input gate bias term; For time steps The output of the lower output gate; This represents the weight matrix of the input vector corresponding to the output gate; The weight matrix represents the hidden state corresponding to the output gate; This represents the bias term of the output gate; This represents the weight matrix input to the forget gate; The weight matrix representing the hidden state to the forget gate; Indicates the output of the forget gate; This represents the weight matrix input to the candidate state; The weight matrix represents the transition from the hidden state to the candidate state; Indicates the candidate state bias term; By time step Hidden state With time step Input vector of the lower LSTM unit The concatenated data is then fed into a neural network to measure the correlation between historical states and the current input.
[0019] More preferably, the state observation values also include: the minimum ranging value or its normalized result of the lidar carried on the robot in each sub-angle interval; the sub-angle interval is the sub-angle interval after the scanning angle of the lidar is equally divided; the minimum ranging value is: the minimum ranging value of the lidar in each laser beam direction in the corresponding sub-angle interval. The state observations also include: the movement commands from the previous moment; The relative position information between the robot and the target location includes: the relative distance and relative direction between the robot and the target location.
[0020] More preferably, the movement commands include the robot's linear velocity and angular velocity.
[0021] More preferably, the reward value for: , , , The weighted summation result; Among them, when the robot executes the movement command When the target location is reached, the first Sparse reward value at any given time A preset positive reward value is set; when the robot executes a movement command... When a collision occurs later, The first preset negative reward value; otherwise, =0; The minimum distance between the current robot and the nearest obstacle When the distance is less than the preset distance, the first Obstacle distance penalty value at any given time The second preset negative reward value, otherwise, =0; No. Speed bonus value at any given moment ; For the first The linear velocity of the robot at any given moment; For the first The robot's angular velocity at that moment; No. Directional reward value at any given moment ; These are preset coefficients; For the first The angle between the robot's current direction of travel and the direction the robot is pointing towards the target position.
[0022] More preferably, the first Obstacle distance penalty value at any given time for:
[0023] in, and All are coefficients; ; , , and All are preset distance thresholds; .
[0024] Secondly, the present invention provides a robot navigation method for indoor environments, comprising: The robot's state observations collected in real time are input into the robot navigation model to obtain the robot's movement commands at the current moment, so as to control the robot's movement in real time. The robot navigation model is trained using the training method provided in the first aspect of this invention; the state observations include the relative position information between the robot and the target position.
[0025] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.
[0026] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or second aspect of the present invention.
[0027] Fifthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first or second aspect of the invention.
[0028] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: 1. This invention provides a training method for a robot navigation model in indoor environments. It employs a reinforcement learning algorithm to train the robot navigation model. Considering the complexity of indoor environments, a progressive training strategy is adopted, decomposing the robot navigation task from easy to difficult. First, a relatively small global distance threshold is given, and initial and target positions with distances less than or equal to this global distance threshold are randomly selected for training, enabling the model to reach target positions at shorter distances. Then, the global distance threshold is gradually increased, progressively improving the distance the model can reach target positions. This avoids ineffective exploration processes caused by excessive task difficulty at the beginning, accelerates convergence, and enhances the adaptability of the strategy in unknown environments. It can improve the convergence speed of model training while ensuring the accuracy of the robot navigation model.
[0029] 2. Furthermore, considering that in real-world environments, external disturbances may cause deviations between the desired control commands and the actual executed outputs, thereby introducing output noise and causing performance degradation during actual deployment, the training method for robot navigation models in indoor environments provided by this invention obtains the robot's movement commands... Next, obtain the robot's movement instructions. Post-state observations and reward value Previously, based on the instructions for movement... Random sampling is performed using a Gaussian distribution with a mean of less than a preset threshold and a variance of less than a preset threshold to obtain new movement commands, and then... Update the system. By adding perturbations, the robot navigation model is made more adaptable to the uncertainties of real indoor scenarios during training, further improving the model's generalization ability.
[0030] 3. Furthermore, considering that traditional LSTMs are forced to update the hidden state at each time step when processing long-term sequence inputs, LSTMs may not be able to stably retain long-term temporal information, resulting in unstable policy updates and convergence delays. The training method for robot navigation models in indoor environments provided by this invention introduces a gating adjustment unit between the hidden state input of the previous time step and the hidden state output of the current time step in the traditional LSTM unit. Based on the correlation between the current input and past historical states, it adaptively selects to retain long-term historical states or update the state, providing stable and reliable long-term historical information for complex navigation decisions, thereby alleviating the problems of long-sequence information decay and state oscillation, and thus improving the stability of temporal modeling.
[0031] 4. The training method for robot navigation models in indoor environments provided by this invention is an end-to-end architecture that does not require a global map. It is suitable for various indoor application scenarios such as home services and warehousing logistics, and has good practical application value. Attached Figure Description
[0032] Figure 1 A flowchart illustrating a method for training a robot navigation model for indoor environments, as provided in an embodiment of the present invention. Figure 2 The strategy network structure diagram provided in the embodiments of the present invention; Figure 3 A network structure diagram of critics provided in an embodiment of the present invention; Figure 4 Detailed diagram of the GHULSTM network provided in the embodiments of the present invention; Figure 5 This is a schematic diagram of the Burn-in training mechanism provided in an embodiment of the present invention; Figure 6 A comparison chart of collision rates of robots trained using Comparison Method 1 and Comparison Method 2, provided for embodiments of the present invention; Figure 7 A comparison chart of the arrival rates of robots trained using Comparison Method 1 and Comparison Method 2, respectively, provided for embodiments of the present invention; Figure 8A comparison chart of collision rates of robots trained using comparison method 1 and comparison method 3, provided for embodiments of the present invention; Figure 9 A comparison chart of the arrival rates of robots trained using comparison method 1 and comparison method 3, respectively, provided for embodiments of the present invention; Figures 10-13 The diagram shows the navigation paths of the robot trained by comparison method 1, comparison method 2, and comparison document 3 in four test scenarios, respectively, as provided in the embodiments of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0034] To achieve the above objectives, in a first aspect, the present invention provides a method for training a robot navigation model for indoor environments, comprising: Initialize global distance threshold (In one alternative implementation, The initial value is 2m), and the robot navigation model is trained for one or more rounds (in one optional implementation, the number of training rounds is 100 rounds); wherein the training process of each round includes: S1. Initialize training rounds ; S2. Randomly initialize indoor environmental parameters; randomly initialize the robot's initial position and target position, with the distance between the initial position and target position being less than or equal to... ;make ; S3, the first robot state observations at time 1 Input into the robot navigation model to obtain the first Current robot movement commands And obtain the robot's movement instructions. Post-state observations and reward value ;Will , , and The quaternion formed as the first The data sample obtained at time is denoted as . The data is stored in the experience replay pool; the state observations include: the relative position information of the robot and the target position; S4. Determine if the termination condition is met. If yes, proceed to S5; otherwise, let... Proceed to S3; Termination conditions include: the robot executes the movement command. A collision may occur later, or the robot may execute a movement command. Then arrive at the target location, or, Reaching a preset time step (in one optional implementation, the preset time step) (Value is 300). S5. When the robot executes a movement command Upon reaching the target location, ; For a preset distance step size (in one optional implementation), (Value is 0.03m). To preset the maximum distance (in one optional implementation), (Value is 8m). judge Is it an integer multiple of the preset number of rounds? If yes, proceed to S6; otherwise, proceed to S7. S6. Extract data samples from the experience replay pool to form a training sample set; based on the training sample set, use a reinforcement learning algorithm to train the robot navigation model; S7, Judgment Is it less than a preset round threshold M (in one optional implementation, M is 300)? If so, then let... If the current round of training ends, proceed to S2; otherwise, end the current round of training.
[0035] Considering that external disturbances may cause deviations between the desired control commands and the actual executed outputs, thereby introducing output noise and causing performance degradation during actual deployment, preferably, in one optional implementation, the above S3 includes: after obtaining the robot's movement commands... Next, obtain the robot's movement instructions. Post-state observations and reward value Previously, based on the instructions for movement... Random sampling is performed using a Gaussian distribution with a mean of less than a preset threshold and a variance of less than a preset threshold to obtain new movement commands, and then... Update.
[0036] It should be noted that robot navigation models can be of various types, such as machine learning models, neural network models, and models based on path planning algorithms (such as PPO algorithm, PSO algorithm, etc.), etc., without any limitation here.
[0037] Preferably, in one optional implementation, the robot navigation model includes: a feature extraction module and a mapping module; the feature extraction module is used to extract features from the input state observations of the robot; the mapping module is used to map the features obtained by the feature extraction module into corresponding movement commands; wherein, the feature extraction module includes: an LSTM model.
[0038] It should be noted that the feature extraction module mentioned above can be an LSTM model, or a model that includes an LSTM model, such as a cascaded LSTM model or a CNN model. The number of LSTM models in the feature extraction module is also unlimited; multiple LSTM models can be used.
[0039] The mapping module can be a fully connected layer, a ReLU layer, etc., and there are no restrictions here.
[0040] Preferably, in one optional implementation, considering that traditional LSTM forces an update to the hidden state at each time step when processing long-term sequence inputs, LSTM may not be able to stably retain long-term temporal information, leading to unstable policy updates and convergence delays. To address this issue, this implementation introduces a gating adjustment unit (i.e., a gating highway unit) between the hidden state input of the previous time step and the hidden state output of the current time step in the traditional LSTM unit, constructing a GHULSTM network structure. This adaptively selects to retain long-term historical states or update the state based on the correlation between the current input and past historical states, enhancing the model's ability to model long-term temporal dependencies, thereby alleviating the problems of long-sequence information decay and state oscillation, and ultimately improving the stability of temporal modeling. Specifically, the LSTM model includes: multiple cascaded LSTM units; each LSTM unit includes: an input gate, an output gate, and a forget gate. Among them, regarding the time step When inputting data, the input gate, output gate, and forget gate perform the following processing respectively:
[0041]
[0042]
[0043] Time step candidate states for:
[0044] Time step Cellular state for:
[0045] Time step The intermediate hidden state for:
[0046] Time step Hidden state for:
[0047] in, For time steps Output of the lower input gate; This represents the weight matrix input to the input gate; Indicates time step The input vector of the lower LSTM unit; The weight matrix represents the weights from the hidden state to the input gate; Indicates time step The hidden state below; Indicates the input gate bias term; For time steps The output of the lower output gate; This represents the weight matrix of the input vector corresponding to the output gate; The weight matrix represents the hidden state corresponding to the output gate; This represents the bias term of the output gate; This represents the weight matrix input to the forget gate; The weight matrix representing the hidden state to the forget gate; Indicates the output of the forget gate; This represents the weight matrix input to the candidate state; The weight matrix represents the transition from the hidden state to the candidate state; Indicates the candidate state bias term; By time step Hidden state With time step Input vector of the lower LSTM unit The concatenated data is then fed into a neural network to measure the correlation between historical states and the current input.
[0048] The LSTM unit in this optional embodiment is an improved LSTM unit, denoted as GHULSTM. Unlike traditional LSTM, which forces an update of the hidden state at each time step, GHULSTM achieves adaptive information propagation based on input correlation through a gated highway mechanism. In long sequence scenarios, it effectively balances "state preservation" and "state update", alleviates the problem of temporal information decay, and enhances the stability and robustness of long-term dependency modeling.
[0049] In one optional implementation, the state observation value further includes: the minimum ranging value or its normalized result of the lidar carried on the robot in each sub-angle interval; the sub-angle interval is the sub-angle interval after the scanning angle of the lidar is equally divided; the minimum ranging value is: the minimum ranging value of the lidar in each laser beam direction in the corresponding sub-angle interval. The state observations also include: the movement commands from the previous moment; The relative position information between the robot and the target location includes: the relative distance and relative direction between the robot and the target location.
[0050] In one alternative implementation, the movement commands include the robot's linear velocity and angular velocity.
[0051] In one alternative implementation, the reward value for: , , , The weighted summation result; Among them, when the robot executes the movement command When the target location is reached, the first Sparse reward value at any given time A preset positive reward value is set; when the robot executes a movement command... When a collision occurs later, The first preset negative reward value; otherwise, =0; The minimum distance between the current robot and the nearest obstacle When the distance is less than the preset distance, the first Obstacle distance penalty value at any given time The second preset negative reward value, otherwise, =0; No. Speed bonus value at any given moment ; For the first The linear velocity of the robot at any given moment; For the first The robot's angular velocity at that moment; No. Directional reward value at any given moment ; The preset coefficient is preferably 0.3 in one optional implementation. For the first The angle between the robot's current direction of travel and the direction the robot is pointing towards the target position.
[0052] Preferably, in one optional implementation, the first The obstacle distance penalty value at any given time is measured in three stages, the first stage being... Obstacle distance penalty value at any given time for:
[0053] in, and All are coefficients; ; , , and All are preset distance thresholds. Preferably, in one optional implementation, The value is 2. The value is 0.5. The value is 0.5m. The value is 0.3m. The value is 1m.
[0054] It should be noted that the reinforcement learning algorithm mentioned above can be the SAC algorithm, the Proximal Policy Optimization (PPO) algorithm, the Q-Learning algorithm, the DQN algorithm, etc., and is not limited here. Preferably, in one optional implementation, the reinforcement learning algorithm mentioned above is the SAC algorithm.
[0055] To further illustrate the training method for a robot navigation model in an indoor environment provided by the first aspect of the present invention, a detailed description is provided below with reference to a specific embodiment: like Figure 1 As shown, this embodiment constructs a reinforcement learning model comprising a policy network and a critic network. The policy network and critic network are built based on the SAC algorithm, and a gated high-speed unit-enhanced Long Short-Term Memory (LSTM) network (GHULSTM) is introduced into both networks to model the temporal state information during the mobile robot's navigation process. Figure 2 The diagram shown is a strategy network structure diagram; as follows: Figure 3 The diagram shown is a network structure diagram of critics.
[0056] During the training phase, the mobile robot is controlled to interact with the indoor environment based on the current policy network. Motion noise is introduced when the action is performed to simulate the impact of control execution deviation on the robot's movement in the actual environment. The robot's state, action, reward and next state at each moment are obtained and the corresponding state transition data is stored in the round buffer. After the training round ends, a fixed-length sequence is extracted and stored in the experience replay pool.
[0057] A fixed-length temporal subsequence is randomly sampled from the experience replay pool to update the parameters of the policy network and the critic network; During the parameter update process, a Burn-in hidden state warm-up step is first performed on each sampled time-series subsequence. The hidden state of GHULSTM is updated forward using the data from the beginning of the time-series subsequence. Then, loss calculation and parameter backpropagation are performed on the remaining part of the time-series subsequence based on the warmed-up hidden state. During training, a progressive training mechanism is adopted, which controls the difficulty of navigation tasks by limiting the maximum distance between the robot's initial position and the target position, and gradually relaxes the maximum distance limit when the robot successfully reaches the target position.
[0058] In the progressive training mechanism, the maximum distance between the robot's initial position and the target position is limited to an initial distance threshold in the early stages of training, and gradually increased according to a preset distance increment when the robot successfully reaches the target position, until the preset maximum distance threshold is reached.
[0059] Furthermore, during the training phase, random noise is introduced into the actions output by the policy network before they are used as actual actions by the robot. The random noise is used to simulate control deviations caused by execution errors or external disturbances in the real environment, thereby improving the robustness of the navigation policy in the real environment.
[0060] In this embodiment, an indoor simulation environment is constructed based on the ROS2 and Gazebo platforms. The environment includes walls and fixed obstacles, supports randomized configuration of obstacle positions, and also supports random generation of the robot's initial and target positions. During training, the distribution of obstacles in the indoor environment is randomly generated to enhance the adaptability of the navigation strategy under different environmental configurations.
[0061] This embodiment mainly involves four core modules that work together: the GHULSTM temporal modeling module, the SAC reinforcement learning decision-making module, the progressive training scheduling module, and the Burn-in experience replay mechanism. These modules collaborate in their structural design, training process, and optimization objectives, collectively forming a complete, closed-loop indoor navigation learning and decision-making framework. The following is a description of each module: 1) GHULSTM timing modeling module: In indoor navigation tasks, robots need to make long-term decisions based on continuous perception data, and the task itself has significant long-term temporal dependencies. Traditional LSTM is prone to problems such as frequent oscillations of hidden states when processing long-sequence navigation data, resulting in unstable policy updates and slow convergence speed. To address this, this embodiment introduces a Gate Highway Unit (GHU) to the traditional LSTM to construct a GHULSTM network structure, thereby enhancing the model's ability to model long-term temporal dependencies.
[0062] like Figure 4As shown, the GHULSTM module adopts a multi-layer stacked structure, with each layer retaining the complete computational flow of a traditional LSTM unit. At each time step, the output hidden state of the traditional LSTM unit is first calculated based on the input gate, forget gate, output gate, and cell state update mechanism. With cell state This allows it to inherit the mature capabilities of LSTM in time series modeling.
[0063] Building upon the traditional LSTM output, a gated highway path is introduced for each layer. The input to the highway path is the hidden state from the previous time step. Compared with the original input at the current time By splicing together, through and The concatenated data is then input into a neural network to generate gated variables. It is used to measure the correlation between historical states and current inputs.
[0064] Final hidden state Output via traditional LSTM With historical hidden state The weighted fusion is obtained, and its calculation formula is:
[0065] The fused hidden states are then subjected to layer normalization to stabilize the feature distribution and enhance the numerical stability of the training process.
[0066] when When the value is close to 1, it indicates that the current input is highly consistent with the historical state, and the network tends to directly retain historical information through high-speed pathways, avoiding unnecessary updates to the already established stable time-series representation; when... When the value is close to 0, it indicates that there is a significant difference between the current input and the historical pattern. The network will prioritize using the LSTM output calculated based on the current input to quickly adapt to environmental changes.
[0067] Unlike traditional LSTM, which forcibly updates the hidden state at every time step, GHULSTM introduces a gated high-speed path on top of the traditional Long Short-Term Memory (LSTM) network. This allows the current hidden state to be obtained by weighted fusion of the LSTM output and the previous hidden state according to gating weights. The gated high-speed path adaptively adjusts the fusion ratio of the previous hidden state information and the LSTM output in the hidden state update based on the correlation between the current input information and historical hidden state information. This gated high-speed mechanism enables adaptive information propagation based on input correlation, effectively balancing "state preservation" and "state update" in long-sequence scenarios, mitigating the problem of temporal information decay, and enhancing the stability and robustness of long-term dependency modeling.
[0068] 2) SAC Reinforcement Learning Decision Module: This embodiment uses the Soft Actor-Critic (SAC) algorithm as the core reinforcement learning framework. By introducing a maximum entropy regularization term, SAC maximizes the cumulative reward while maintaining policy randomness, exhibiting good exploration capabilities and robustness.
[0069] (a) Strategy Network Design Policy network observes state With hidden state As input, the input features are processed sequentially through a GHULSTM layer, a linear layer, and a ReLU activation function, and finally output as the mean of the action distribution through two parallel linear layers. With log standard deviation A Squashed Normal distribution is constructed for action sampling.
[0070] To explicitly model control noise during the execution of a real robot, a secondary sampling mechanism is introduced after the initial action sampling: the initial sampled action is used as the mean, and a fixed small variance is applied for resampling, which is then used as the final execution action, thereby improving the robustness of the strategy in real deployment scenarios.
[0071] (II) Critics' Network Design The critic network uses a double-Q network structure, with the input being the observed state. ,action With hidden state The splicing features, after being processed by the GHULSTM layer and the fully connected layer, output two independent Q-value estimates. and The minimum of the two values is taken as the final value estimate to reduce overestimation bias and improve decision reliability.
[0072] 3) Progressive training scheduling module: In complex indoor environments, if the initial position of the robot is far from the target position and the obstacles are randomly distributed, the agent may have difficulty obtaining effective reward signals in the early stages of training, which may lead to ineffective exploration and slow convergence.
[0073] This embodiment proposes a progressive training strategy centered on increasing target distance. By dynamically adjusting the difficulty of training tasks, the model gradually learns navigation capabilities from easy to difficult.
[0074] Define the upper limit parameter of distance Its value range is: Initial training setup:
[0075] robot's initial position With the target location satisfy:
[0076] When the robot successfully reaches the goal without collision at the current difficulty level, the difficulty is increased according to the following rules:
[0077] This mechanism ensures that the model only moves to a higher difficulty level after mastering the current difficulty, thus avoiding training instability.
[0078] 4) Burn-in experience replay mechanism Under the experience replay framework, hidden states cannot be stored effectively, which disrupts GHULSTM's dependency on long-term historical states and weakens its long-term time-series modeling performance.
[0079] like Figure 5 As shown, the Burn-in mechanism works as follows: The length is The sequence is divided into a burn-in segment and an effective training segment:
[0080] The burn-in phase only performs forward propagation to update the hidden state, without calculating the loss; the effective training phase performs backpropagation and parameter updates based on the warmed-up hidden state.
[0081] In the Burn-in hidden state warm-up step, only the data from the first part of the time series subsequence is used to perform forward propagation updates on the hidden state of GHULSTM, and no loss calculation or parameter updates are performed in this step. After completing the Burn-in hidden state warm-up step, based on the warmed-up hidden state, loss calculation is performed on the remaining part of the time series subsequence, and backpropagation updates are performed on the policy network and the commentator network.
[0082] This mechanism can simultaneously satisfy the decorrelation requirement of experience replay and the temporal constraint of hidden states during training. While maintaining the decorrelation advantage of experience replay, it ensures the normal updating of the policy network and the commentator network.
[0083] The following are the detailed implementation steps for the training phase: (I) Observation State Construction and Preprocessing Methods The robot's observation state is formed by fusing environmental perception information from the lidar carried by the robot, target relative position information, and historical action information, as detailed below: 1) LiDAR data processing: Angle range discretization: The original scanning angle range of the lidar is divided into equal parts. Each fixed angle interval contains multiple laser beams.
[0084] Minimum distance extraction: For the For each angle interval, the minimum distance value within that interval is extracted using the following formula:
[0085] in, Indicates the angle of the lidar The distance measured at that location.
[0086] Normalization process: The minimum distance value of each angle interval is normalized to the interval using the following formula. : in, This is the maximum detection range of the laser.
[0087] Thus, the feature vector of the lidar is obtained:
[0088] 2) Construction of target relative position features Relative distance calculation: Calculate the robot's current position With the target location The Euclidean distance between them is normalized using the following formula:
[0089] in, This represents the maximum diagonal distance of the environment.
[0090] Relative orientation representation: Calculate the angle between the robot's current heading vector and the "robot-target position" direction vector. and extract and As a directional feature, it is used to characterize the orientation relationship of the target relative to the robot.
[0091] 3) Fusion of historical action information Introduce the robot's linear velocity from the previous moment. With angular velocity This data is then concatenated with the lidar features and the target's relative position features to construct the final observation state vector.
[0092] This embodiment employs a hybrid reward mechanism that combines sparse and dense rewards to simultaneously ensure task completion, motion safety, and navigation efficiency.
[0093] 1. Sparse Reward Items Sparse rewards are triggered only when the task is terminated, and are defined as follows:
[0094] in, A positive reward is given to indicate that the robot has successfully reached the target location. This indicates a collision has occurred and a penalty will be imposed.
[0095] 2. Dense Reward Items Obstacle distance penalty: Based on the minimum distance between the robot and the nearest obstacle Design a piecewise penalty function:
[0096] Basic speed bonus items: Used to encourage robots to maintain high linear velocity and suppress excessive angular velocity:
[0097] Directional reward items: Guide the robot to move in the target direction:
[0098] 3. Total Reward Function Robot at all times The total reward is defined as follows:
[0099] (III) Training Process 1. Training round initialization Set the total number of training rounds to [number]. Each training round contains Each independent training round; initialize the target distance upper limit parameter. An experience replay buffer is established. The maximum number of interaction steps per training round is set to [value missing]. .
[0100] 2. Interaction flow of a single training round At the beginning of each training round, the robot agent interacts with the environment in the following steps: (1) Round initialization Reset the simulation environment, including: Randomly generate the spatial distribution of obstacles; Under the current upper limit constraint of target distance, the initial position of the randomly sampled robot is determined. With the target location ,satisfy:
[0101] Meanwhile, the hidden state of the GHULSTM network is initialized to a zero vector to eliminate the interference of cross-round historical information on the current round of training.
[0102] (2) Construction of environmental perception and observation status Within each control time step, the robot acquires the observed state vector at the current moment. .
[0103] (3) Policy reasoning and hidden state update based on GHULSTM Current observation status The input policy network is processed through a GHULSTM layer, a fully connected layer, and a nonlinear activation function to output the mean and standard deviation parameters of the action distribution. Action sampling is then performed based on the Squashed Normal distribution to obtain the robot's current linear velocity command. With angular velocity command .
[0104] (4) Action execution and environmental state recursion The robot executes actual movements according to the control commands, and the environment returns to a new observation state. and instant reward signals .
[0105] (5) Interactive data recording The interaction samples at the current moment Recorded in the current round's round buffer for subsequent experience segmentation and replay training.
[0106] (6) Round termination condition judgment Determine the round termination condition after each time step: If the robot successfully reaches the target location, it is considered a successful return and the process terminates. If the robot collides with an obstacle, it is considered a failed attempt and the operation is terminated immediately. If the number of interaction steps reaches the maximum limit If so, the round will be forcibly terminated; If none of the above conditions are met, proceed to the next time step to continue the interaction.
[0107] (7) Progressive training trigger When the training round ends with the "target successfully reached" status, the target distance upper limit parameter is updated according to the progressive training rules. This introduces more challenging navigation tasks for subsequent training rounds.
[0108] 3. Experience processing after a round ends After each training round, the continuous interaction trajectories collected in that round are divided into fixed lengths. The time series is segmented into multiple subsequences and stored uniformly in an experience replay buffer to support time series training based on recurrent neural networks.
[0109] 4. Network parameter updates within training rounds When the number of independent training rounds is an integer multiple of a preset number of rounds, random sampling is performed from the experience replay buffer. Each time series subsequence is used to update the parameters of the policy network and the commentator network, respectively.
[0110] In each time series sample, firstly, the previous... A burn-in warm-up is performed at each time step, used only for the recursive update of the GHULSTM hidden state and not involved in the loss calculation. Subsequently, the loss function is calculated based on the remaining time steps, and backpropagation is performed to optimize the network parameters. This update process is repeated within each training epoch. Second-rate.
[0111] 5. Testing and Performance Evaluation Process After parameter updates are completed in each training epoch, the testing and evaluation phase begins. During the testing phase: The target distance is uniformly set to the maximum distance limit. ; The obstacle distribution rules remain consistent with those used in the training phase. Navigation tasks were performed in 10 pre-set independent test scenarios, and the percentage of robots successfully reaching the target and the percentage of collisions were counted respectively, serving as the arrival rate and collision rate indicators for the current training round.
[0112] 6. Training cycle progression After the test and evaluation are completed, proceed to the next training round and repeat the above process until all tests are completed. One training round.
[0113] To address the common problems of existing indoor consumer mobile robot navigation algorithms in complex environments, such as slow training convergence speed, insufficient ability to retain long-sequence temporal information, large performance fluctuations in the stable phase, and limited robustness and generalization ability, this embodiment proposes a SAC-GHULSTM indoor robot navigation method based on progressive training.
[0114] This embodiment improves the training convergence efficiency, target arrival rate, and long-term stability of the robot navigation strategy simultaneously without relying on a global map or manual rules, through the coordinated design of temporal modeling structure optimization, innovative training strategy design, and robustness enhancement mechanism. It effectively reduces the collision rate and enhances the algorithm's adaptability and generalization performance in unknown and complex indoor environments.
[0115] To further illustrate the performance of this invention, performance verification and result analysis are presented below. Three sets of comparison methods were designed to verify the effectiveness of each core module (progressive training and GHULSTM): Comparison Method 1: GHULSTM + Progressive Training; Comparison Method 2: Method without GHULSTM: Traditional LSTM + Progressive Training; Comparison Method 3: Non-progressive training method: GHULSTM + fixed target distance training.
[0116] All methods were independently repeated 10 times under the same environment and parameter conditions, and the results were averaged for statistical analysis. In this experiment, the total number of training epochs (training rounds) was... 100; each training round contains Each epoch consists of two independent training rounds; the robot navigation model is trained every two training rounds within each epoch; the maximum number of steps per round. 300; sequence length 32; Burn-in steps The batch size is 16; when training the robot navigation model using a reinforcement learning algorithm, the batch size is... The number of robot navigation model update iterations is 50. The initial distance is set to 100; the number of test scenarios is 10; for progressive training, the initial distance upper limit is... The maximum distance is 2. The distance increment is 8. The value is 0.03; the capacity of the experience replay buffer is 5000; the state dimension of the network structure is 25, and the action dimension is 2.
[0117] The average performance metrics of each method throughout the training process are shown in Table 1: Table 1
[0118] The comparison results above show that, compared to the scheme without a progressive training strategy, the scheme using a progressive training strategy gradually increases the difficulty of tasks from easy to hard, enabling the robot to quickly establish an effective navigation strategy in the early stages of training and gradually adapt to navigation tasks in complex scenarios in subsequent stages. This avoids ineffective exploration and performance degradation caused by excessively difficult initial tasks. Furthermore, compared to the scheme without the GHULSTM structure, the scheme with the GHULSTM structure improves the robot's safety and stability in complex indoor environments by enhancing its ability to model and retain historical temporal information.
[0119] Specifically, Figure 6 A comparison chart of collision rates of robots trained using Comparison Method 1 and Comparison Method 2, provided for embodiments of the present invention; Figure 7 A comparison chart of the arrival rates of robots trained using Comparison Method 1 and Comparison Method 2, respectively, provided for embodiments of the present invention; Figure 8 A comparison chart of collision rates of robots trained using comparison method 1 and comparison method 3, provided for embodiments of the present invention; Figure 9 The arrival rate comparison chart of the robot trained by comparison method 1 and comparison method 3 is provided for the embodiments of the present invention; wherein the shaded area represents the fluctuation range of 10 experiments.
[0120] The following are examples of navigation efficiency and path visualization verification. The method according to the embodiment After completing a full training cycle using comparison methods 1, 2, and 3 (i.e., a total of 100 epochs), the trained model parameters were saved. Subsequently, the navigation paths and efficiency of the robot trained using different comparison methods were verified in a fixed testing environment.
[0121] In the verification embodiment, four pre-defined indoor navigation test scenarios (Scenario 1 to Scenario 4) were selected as test objects. To ensure the repeatability and consistency of the comparison results, the robot's initial position, target position, and obstacle distribution remained unchanged in each test scenario, and the same environmental configuration was used across different comparison methods.
[0122] During the testing process, comparison method 1, comparison method 2, and comparison method 3 will be compared.
[0123] In each test scenario, the robot performs a navigation task until it successfully reaches the target location, collides with another object, or reaches the maximum number of steps.
[0124] In four test scenarios, the robot's motion trajectory from its initial position to its target position is recorded and visualized.
[0125] Test results show that the robot trained using Comparison Method 1 successfully reached the target location in all four test scenarios. The robot trained using Comparison Method 2 also reached the target location in all four test scenarios, and its path was basically the same as that of the robot trained using Comparison Method 1. The robot trained using Comparison Method 3 reached the target location in scenarios 1, 2, and 4, but failed to reach the target location in scenario 3.
[0126] To quantitatively analyze the execution efficiency of the different methods in navigation tasks, statistics were compiled on the number of control steps required for the robot to complete the navigation task and the average movement speed in each test scenario.
[0127] For navigation step counting: In scenarios 1 to 4, the number of control steps required for the robot trained using Comparison Method 1 to complete the navigation task are 64, 79, 67, and 62 steps, respectively.
[0128] The robots trained using Comparison Method 2 required 75, 84, 116, and 79 control steps to complete navigation tasks in the same scenario, respectively.
[0129] The robot trained using Comparison Method 3 required 78, 198, and 94 control steps to complete the navigation task in Scenario 1, Scenario 2, and Scenario 4, respectively, but failed to reach the target location in Scenario 3.
[0130] Statistics on average motion speed: In scenarios 1 to 4, the average movement speeds of the robots trained using Comparison Method 1 were 0.642 m / s, 0.591 m / s, 0.633 m / s, and 0.589 m / s, respectively.
[0131] The average movement speeds of the robots trained using Comparison Method 2 in the same scenario were 0.524 m / s, 0.515 m / s, 0.462 m / s, and 0.467 m / s, respectively.
[0132] The robot trained using Comparison Method 3 had average movement speeds of 0.526 m / s, 0.427 m / s, and 0.388 m / s in scenarios 1, 2, and 4, respectively, but failed to complete the navigation task in scenario 3.
[0133] Specifically, Figures 10-13 The diagram shows the navigation paths of the robot trained by comparison method 1, comparison method 2, and comparison document 3 in four test scenarios, respectively, as provided in the embodiments of the present invention.
[0134] In summary, the training method provided by this invention can improve the convergence speed of model training while ensuring the navigation accuracy of the robot navigation model. In particular, compared with method 1, it achieves improved efficiency and performance at both the path selection and motion execution levels by constructing a more efficient time-dependent relationship model. Therefore, the robot exhibits superior motion stability and execution efficiency in complex environments. Under the same training rounds and task configuration conditions, the proposed model not only shortens the task completion time but also improves the reliability of target arrival, demonstrating excellent overall navigation performance.
[0135] Secondly, the present invention provides a robot navigation method for indoor environments, comprising: The robot's state observations collected in real time are input into the robot navigation model to obtain the robot's movement commands at the current moment, so as to control the robot's movement in real time. The robot navigation model is trained using the training method provided in the first aspect of this invention; the state observations include the relative position information between the robot and the target position.
[0136] The related technical solutions are the same as the training method provided in the first aspect of this invention, and are not limited here.
[0137] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the method provided in the first or second aspect of the present invention.
[0138] The related technical solutions are the same as the training method provided in the first aspect and the navigation method provided in the second aspect of this invention, and are not limited here.
[0139] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed by a processor, controls the device in which the storage medium is located to perform the method provided in the first or second aspect of the present invention.
[0140] The related technical solutions are the same as the training method provided in the first aspect and the navigation method provided in the second aspect of this invention, and are not limited here.
[0141] Fifthly, the invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first or second aspect of the invention.
[0142] The related technical solutions are the same as the training method provided in the first aspect and the navigation method provided in the second aspect of this invention, and are not limited here.
[0143] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A training method for a robot navigation model in an indoor environment, characterized in that, include: Initialize global distance threshold The robot navigation model is trained through one or more rounds; each round of training includes: S1. Initialize training rounds ; S2. Randomly initialize indoor environmental parameters; randomly initialize the robot's initial position and target position, with the distance between the initial position and target position being less than or equal to... ;make ; S3, the first robot state observations at time 1 Input into the robot navigation model to obtain the first Current robot movement commands And obtain the robot's movement instructions. Post-state observations and reward value ;Will , , and The quaternion formed as the first The data sample obtained at time is denoted as . The data is stored in the experience replay pool; the state observations include: the relative position information of the robot and the target position; S4. Determine if the termination condition is met. If yes, proceed to S5; otherwise, let... Proceed to S3; the termination condition includes: the robot executes a movement command. A collision may occur later, or the robot may execute a movement command. Then arrive at the target location, or, Reaching the preset time step; S5. When the robot executes a movement command Upon reaching the target location, ; Preset distance step size; The preset maximum distance; judge Is it an integer multiple of the preset number of rounds? If yes, proceed to S6; otherwise, proceed to S7. S6. Extract data samples from the experience replay pool to form a training sample set; based on the training sample set, use a reinforcement learning algorithm to train the robot navigation model; S7, Judgment Is it less than the preset round threshold M? If so, then let... If the current round of training ends, proceed to S2; otherwise, end the current round of training.
2. The training method according to claim 1, characterized in that, S3 further includes: upon receiving the robot's movement command. Next, obtain the robot's movement instructions. Post-state observations and reward value Previously, based on the instructions for movement... Random sampling is performed using a Gaussian distribution with a mean of less than a preset threshold and a variance of less than a preset threshold to obtain new movement commands, and then... Update.
3. The training method according to claim 1, characterized in that, The robot navigation model includes a feature extraction module and a mapping module; the feature extraction module is used to extract features from the input state observations of the robot; the mapping module is used to map the features obtained by the feature extraction module into corresponding movement commands; wherein, the feature extraction module includes an LSTM model.
4. The training method according to claim 3, characterized in that, The LSTM model includes: multiple cascaded LSTM units; each LSTM unit includes: an input gate, an output gate, and a forget gate; For time steps When inputting data, the input gate, output gate, and forget gate perform the following processing respectively: Time step candidate states for: Time step Cellular state for: Time step The intermediate hidden state for: Time step Hidden state for: in, For time steps Output of the lower input gate; This represents the weight matrix input to the input gate; Indicates time step The input vector of the lower LSTM unit; The weight matrix represents the weights from the hidden state to the input gate; Indicates time step The hidden state below; Indicates the input gate bias term; For time steps The output of the lower output gate; This represents the weight matrix of the input vector corresponding to the output gate; The weight matrix represents the hidden state corresponding to the output gate; This represents the bias term of the output gate; This represents the weight matrix input to the forget gate; The weight matrix representing the hidden state to the forget gate; Indicates the output of the forget gate; This represents the weight matrix input to the candidate state; The weight matrix represents the transition from the hidden state to the candidate state; Indicates the candidate state bias term; By time step Hidden state With time step Input vector of the lower LSTM unit The concatenated data is then fed into a neural network to measure the correlation between historical states and the current input.
5. The training method according to any one of claims 1-4, characterized in that, The state observation values also include: the minimum ranging value or its normalized result of the lidar carried on the robot in each sub-angle interval; the sub-angle interval is the sub-angle interval after the scanning angle of the lidar is equally divided; the minimum ranging value is: the minimum ranging value of the lidar in each laser beam direction in the corresponding sub-angle interval. The state observation values also include: the movement command from the previous moment; The relative position information between the robot and the target location includes: the relative distance and relative direction between the robot and the target location; The movement commands include the robot's linear velocity and angular velocity.
6. The training method according to any one of claims 1-4, characterized in that, Reward Value for: , , , The weighted summation result; Among them, when the robot executes the movement command When the target location is reached, the first Sparse reward value at any given time A preset positive reward value is set; when the robot executes a movement command... When a collision occurs later, The first preset negative reward value; otherwise, =0; The minimum distance between the current robot and the nearest obstacle When the distance is less than the preset distance, the first Obstacle distance penalty value at any given time The second preset negative reward value, otherwise, =0; No. Speed bonus value at any given moment ; For the first The linear velocity of the robot at any given moment; For the first The robot's angular velocity at that moment; No. Directional reward value at any given moment ; These are preset coefficients; For the first The angle between the robot's current direction of travel and the direction the robot is pointing towards the target position.
7. The training method according to claim 6, characterized in that, No. Obstacle distance penalty value at any given time for: in, and All are coefficients; ; , , and All are preset distance thresholds; .
8. A robot navigation method for indoor environments, characterized in that, include: The robot's state observations collected in real time are input into the robot navigation model to obtain the robot's movement commands at the current moment, so as to control the robot's movement in real time. The robot navigation model is trained using the training method described in any one of claims 1-7; the state observations include the relative position information between the robot and the target position.
9. An electronic device, characterized in that, include: A memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the method according to any one of claims 1-7.
10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method described in any one of claims 1-7.