Scene generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning

Abstract

The application discloses a scene generalization end-to-end autonomous navigation method based on imitation-reinforcement learning, and belongs to the technical field of robots. In view of the problems of unstable action output and easy falling into local optimization of the existing robot navigation method, a TD3 network architecture is constructed, experience is obtained through continuous interaction with the environment in a Gazebo simulation environment, and network parameters are iteratively optimized and trained by using an experience replay buffer; the trained neural network outputs continuous linear speed and angular speed instructions of the robot according to an input state, drives the robot to move after scaling, and realizes obstacle avoidance and target point navigation; a target approaching reward, a collision penalty and a motion process reward are designed, the policy network is guided to improve the ability of stable navigation to the target point while ensuring safe obstacle avoidance; and imitation learning is introduced to optimize the motion posture of the robot, an Actor network is fine-tuned by collecting an expert data set, and the robot keeps the posture stable in motion.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a scenario-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence and automation technologies, mobile robots are increasingly being used in industrial inspection, disaster relief, household services, and warehousing and logistics. In these application scenarios, robots need to achieve autonomous movement and task execution in complex, ever-changing, and unknown environments, making autonomous navigation technology a research hotspot and core technology in the field of robotics.

[0003] Robot navigation is one of the core functions of autonomous mobile robots, requiring real-time obstacle avoidance and target tracking without prior environmental information. Traditional navigation methods rely on predefined rules or path planning algorithms, which have poor adaptability to complex and dynamic environments. While existing deep reinforcement learning methods can learn navigation strategies through data-driven approaches, they suffer from unstable action outputs and are prone to getting trapped in local optima, making it difficult to meet the high-precision and high-reliability navigation requirements of real-world scenarios. Therefore, a reinforcement learning-based navigation method is needed to improve the robot's obstacle avoidance capabilities and target navigation efficiency in unknown environments by optimizing the network architecture and reward mechanism. Summary of the Invention

[0004] To address the problems of unstable action output and susceptibility to local optima in existing robot navigation methods, this invention provides a scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning, which is suitable for obstacle avoidance and target tracking of robots in unknown environments.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] A scenario-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning includes the following steps:

[0007] Step 1, State Construction and Perception Fusion: The robot is equipped with a laser sensor to collect surrounding environmental data as laser information. At the same time, it obtains the polar coordinates of the target point relative to the robot's current position and the robot's motion state at the previous moment. The laser information, the polar coordinates of the target point relative to the robot's current position, and the motion state information of the robot at the previous moment are concatenated to form a state vector, which is used as the input of the dual-delay deep deterministic policy gradient policy network.

[0008] In step 1, the robot is equipped with a laser sensor to collect environmental distance data within a 180° range in front of the robot as laser information. The laser information is evenly divided into 20 groups, each containing laser readings within a continuous angular range. The minimum value of each group's readings is taken as the feature value of that group to ensure effective capture of obstacle information. Simultaneously, the polar coordinates of the target point relative to the robot's current position and the robot's motion state information at the previous moment are acquired. The laser information, the polar coordinates of the target point relative to the robot's current position, and the robot's motion state information at the previous moment are concatenated to form an input state vector. The polar coordinates of the target point relative to the robot's current position include distance. ,angle The robot's motion state information at the previous moment includes linear velocity and angular velocity.

[0009] Step 2, Construction and training of the dual-delay deep deterministic policy gradient policy network (TD3): Construct the dual-delay deep deterministic policy gradient policy network architecture, continuously interact with the environment in the Gazebo simulation environment to obtain experience, and use the experience replay buffer to iteratively train and optimize the network parameters.

[0010] The dual-delay deep deterministic policy gradient policy network architecture constructed in step 2 is an Actor-Critic network structure. The Actor network consists of multiple fully connected layers and ReLU activation functions. The output uses the Tanh activation function to generate continuous actions in the (-1,1) interval, corresponding to the robot's linear velocity and angular velocity. The Critic network adopts a dual-Q network parallel structure to evaluate the value of state-action pairs and takes the minimum Q value of the two Critic outputs as the final evaluation result.

[0011] The specific operation of step 2 is as follows:

[0012] Step 2.1, the Actor network uses state vectors As input, two fully connected layers are sequentially connected and nonlinear mapping is performed using the ReLU activation function. The output layer uses the Tanh activation function to restrict the action to the interval (-1, 1), and finally outputs two action parameters. and , This is the original value of the linear velocity. This is the original value of the angular velocity;

[0013] Step 2.2, the Critic network adopts a dual Q-network structure (Critic1, Critic2), with state vectors... With action As a joint input, the state vector is first linearly mapped and then activated by ReLU to obtain an intermediate representation. Then the action With intermediate representation Two fully connected transformation layers are input separately, and feature fusion is achieved using the following formula: ;

[0014] in, Represents a fully connected layer of composition. , This is the weight matrix. For bias;

[0015] Step 2.3: Continuously interact with the environment in the Gazebo simulation environment to gain experience, and use the experience replay buffer to iteratively train and optimize the network parameters; during the training phase, a target network and a soft update mechanism are introduced, and Gaussian noise is added to the target action to improve the smoothness and stability of the strategy. After feature fusion, the Q value is output by ReLU activation. During the training process, the minimum Q value between the two Critic networks is selected as the evaluation result.

[0016] Step 3, Action Generation and Execution: The trained policy network outputs continuous linear velocity and angular velocity commands for the robot based on the input state to scale the actions. After the actions are scaled, the robot is driven to move, achieving obstacle avoidance and target point navigation.

[0017] In step 3, motion scaling maps the original motion output by the Actor network to actual motion commands. The motion scaling formula is: ;

[0018] in, This represents the raw value of the linear velocity output by the Actor network. This represents the raw angular velocity value output by the Actor network. Indicates the maximum linear velocity. This represents the maximum angular velocity; after scaling, the linear velocity only takes positive values.

[0019] Step 4, Reward Design and Strategy Guidance: Design rewards for approaching the target, penalties for collisions, and rewards for the movement process to guide the strategy network to improve its ability to stably navigate to the target point while ensuring safe obstacle avoidance;

[0020] In step 4, the reward function is set as follows: ;

[0021] in, Indicates the robot at a certain moment The following state; Indicates that the robot is The following actions were taken; This indicates the current distance between the robot and the target. This represents the distance threshold to reach the target. Indicates positive goal reward. This indicates a negative reward for collisions. Indicates linear velocity. Indicates angular velocity;

[0022] When the distance between the robot and the target Give positive rewards at that time. ;

[0023] When a collision is detected, a negative reward is given. ;

[0024] In other cases, the reward value is defined as The calculation formula is: ;

[0025] in, Indicates the robot at a certain moment The instant reward received; Indicates the robot at a certain moment linear velocity; Indicates the robot at a certain moment The absolute value of the angular velocity; Indicates laser diopters;

[0026] When laser diopters The reward is When the laser diopter When the robot is less than a distance threshold from an obstacle, an additional penalty is applied. The closer the robot is to the obstacle, the greater the penalty, in order to encourage the robot to move quickly and smoothly.

[0027] Step 5: Use a course-based learning approach: Design training courses in ascending order of navigation distance to gradually expand the robot's navigation range and improve the adaptability of the policy network in a wide range of scenarios.

[0028] The specific steps for learning the course in step 5 are as follows:

[0029] Step 5.1, Construct a hierarchical training environment container: Construct N training environments with increasing difficulty in a continuous simulation space, and add the currently unlocked environment to the environment container; the training environments are divided into basic, intermediate and advanced stages according to difficulty, corresponding to static regular obstacle environment, complex narrow passage and dead end environment, and dynamic obstacle and high noise environment, respectively.

[0030] Step 5.2, Set stage switching conditions: Monitor the robot's task success rate in the current stage environment in real time. When the task success rate is continuously higher than a preset threshold within a preset number of rounds, the switch will be initiated. At that time, the next difficulty level environment will be unlocked and added to the environment container;

[0031] Step 5.3, Perform periodic review sampling: During training, randomly sample environments from the environment container according to probability for training; assuming the current state is the... Phase 1, extracting the first environmental container The probability of an environment probability Satisfy the following formula: ,in, , Index for the environment phase;

[0032] Step 5.4: Iterate until convergence, repeating steps 5.2 to 5.3 until the current stage. Training is considered complete when the advancement conditions are met or the preset total number of training steps is reached.

[0033] Step 6: Introduce imitation learning to optimize robot motion posture: By collecting expert datasets and fine-tuning the Actor network, the policy network is guided to learn a motion pattern with high linear velocity and low angular velocity, so that the robot can maintain a stable posture during movement.

[0034] The specific implementation of imitation learning in step 6 is as follows:

[0035] Step 6.1, Construct an expert dataset: Collect navigation samples containing 22-dimensional state data and 4-dimensional motion data, and divide the navigation samples into a training set and a test set according to a preset ratio; wherein the state data includes 20-dimensional laser sensor readings and 2-dimensional waypoint polar coordinate data;

[0036] Step 6.2, Construct the Actor network for the imitation learning stage: The Actor network structure is consistent with that in Step 2, with an input dimension of 24 and an output dimension of 2, and two hidden fully connected layers in between; the hidden layers all use the ReLU activation function, and the output layer uses the Tanh activation function to constrain the range of action output;

[0037] Step 6.3, Configure training parameters: Use the AdamW optimizer, the HuberLoss loss function, and introduce a cosine annealing learning rate scheduler;

[0038] Step 6.4, Model Training: Supervised training of the Actor network is performed using the training set. The network parameters are updated through backpropagation. After training, the final Actor policy network parameters are saved for subsequent decision-making and control.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] Compared with existing technologies, the significant advantages of this invention are that the trained navigation model (an Actor network trained based on a dual-delay deep deterministic policy gradient algorithm and imitation learning) can dynamically generate robot linear velocity and angular velocity control commands based on real-time perception information, achieving real-time obstacle avoidance in unknown environments with stable and reliable obstacle avoidance performance; it also improves the robot's navigation capabilities in a wide range of complex scenarios; furthermore, by integrating imitation learning to optimize the navigation strategy, it guides the robot to form a motion pattern with a large linear velocity and a small change in angular velocity, thereby improving the stability of motion posture during navigation and reducing unnecessary swaying. Attached Figure Description

[0041] Figure 1 This is a flowchart of the overall method of the present invention;

[0042] Figure 2 Experimental testing environment 1;

[0043] Figure 3 Experimental testing environment 2;

[0044] Figure 4 Experimental testing environment 3;

[0045] Figure 5 This is a comparison diagram of the basic TD3 and the model path described in this invention under experimental environment 1;

[0046] Figure 6 This is a comparison diagram of the basic TD3 and the model path described in this invention under experimental environment 2;

[0047] Figure 7 A comparison diagram of the path between the basic TD3 and the model described in this invention under experimental environment 3;

[0048] Figure 8 A comparison chart of the angular velocities of the TD3 model and the model described in this invention. Detailed Implementation

[0049] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0050] A scenario-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning includes the following steps:

[0051] Step 1, State Construction and Perception Fusion: The robot is equipped with a laser sensor to collect surrounding environmental data as laser information. At the same time, it obtains the polar coordinates of the target point relative to the robot's current position and the robot's motion state at the previous moment. The laser information, the polar coordinates of the target point relative to the robot's current position, and the motion state information of the robot at the previous moment are concatenated to form a state vector, which is used as the input of the dual-delay deep deterministic policy gradient policy network.

[0052] In step 1, the robot is equipped with a laser sensor to collect environmental distance data within a 180° range in front of the robot as laser information. The laser information is evenly divided into 20 groups, each containing laser readings within a continuous angular range. The minimum value of each group's readings is taken as the feature value of that group to ensure effective capture of obstacle information. Simultaneously, the polar coordinates of the target point relative to the robot's current position and the robot's motion state information at the previous moment are acquired. The laser information, the polar coordinates of the target point relative to the robot's current position, and the robot's motion state information at the previous moment are concatenated to form an input state vector. The polar coordinates of the target point relative to the robot's current position include distance. ,angle The robot's motion state information at the previous moment includes linear velocity and angular velocity.

[0053] Step 2, Construction and training of the dual-delay deep deterministic policy gradient policy network (TD3): Construct the dual-delay deep deterministic policy gradient policy network architecture, continuously interact with the environment in the Gazebo simulation environment to obtain experience, and use the experience replay buffer to iteratively train and optimize the network parameters.

[0054] The dual-delay deep deterministic policy gradient policy network architecture constructed in step 2 is an Actor-Critic network structure. The Actor network consists of multiple fully connected layers and ReLU activation functions. The output uses the Tanh activation function to generate continuous actions in the (-1,1) interval, corresponding to the robot's linear velocity and angular velocity. The Critic network adopts a dual-Q network parallel structure to evaluate the value of state-action pairs and takes the minimum Q value of the two Critic outputs as the final evaluation result.

[0055] The specific operation of step 2 is as follows:

[0056] Step 2.1, the Actor network uses state vectors As input, two fully connected layers are sequentially connected and nonlinear mapping is performed using the ReLU activation function. The output layer uses the Tanh activation function to restrict the action to the interval (-1, 1), and finally outputs two action parameters. and , This is the original value of the linear velocity. This is the original value of the angular velocity;

[0057] Step 2.2, the Critic network adopts a dual Q-network structure (Critic1, Critic2), with state vectors... With action As a joint input, the state vector is first linearly mapped and then activated by ReLU to obtain an intermediate representation. Then the action With intermediate representation Two fully connected transformation layers are input separately, and feature fusion is achieved using the following formula: ;

[0058] in, Represents a fully connected layer of composition. , This is the weight matrix. For bias;

[0059] Step 2.3: Continuously interact with the environment in the Gazebo simulation environment to gain experience, and use the experience replay buffer to iteratively train and optimize the network parameters; during the training phase, a target network and a soft update mechanism are introduced, and Gaussian noise is added to the target action to improve the smoothness and stability of the strategy. After feature fusion, the Q value is output by ReLU activation. During the training process, the minimum Q value between the two Critic networks is selected as the evaluation result.

[0060] The simulation experiment was conducted in the Gazebo environment, constructing a 10×10 meter scene. The robot's initial pose and target points were randomly generated. The maximum number of steps per round was 500. Robustness and generalization capabilities were improved by incorporating noise enhancement strategies into the motion and sensor inputs.

[0061] Step 3, Action Generation and Execution: The trained policy network outputs continuous linear velocity and angular velocity commands for the robot based on the input state to scale the actions. After the actions are scaled, the robot is driven to move, achieving obstacle avoidance and target point navigation.

[0062] In step 3, motion scaling maps the original motion output by the Actor network to actual motion commands. The motion scaling formula is: ;

[0063] in, This represents the raw value of the linear velocity output by the Actor network. This represents the raw angular velocity value output by the Actor network. Indicates the maximum linear velocity. This represents the maximum angular velocity; after scaling, the linear velocity only takes positive values.

[0064] Step 4, Reward Design and Strategy Guidance: Design rewards for approaching the target, penalties for collisions, and rewards for the movement process to guide the strategy network to improve its ability to stably navigate to the target point while ensuring safe obstacle avoidance;

[0065] In step 4, the reward function is set as follows: ;

[0066] in, Indicates the robot at a certain moment The following state; Indicates that the robot is The following actions were taken; This indicates the current distance between the robot and the target. This represents the distance threshold to reach the target. Indicates positive goal reward. This indicates a negative reward for collisions. Indicates linear velocity. Indicates angular velocity;

[0067] When the distance between the robot and the target Give positive rewards at that time. ;

[0068] When a collision is detected, a negative reward is given. ;

[0069] In other cases, the reward value is defined as The calculation formula is: ; ;

[0070] in, Indicates the robot at a certain moment The instant reward received; Indicates the robot at a certain moment linear velocity; Indicates the robot at a certain moment The absolute value of the angular velocity; Indicates laser diopters;

[0071] When laser diopters The reward is When the laser diopter When the robot is less than a distance threshold from an obstacle, an additional penalty is applied. The closer the robot is to the obstacle, the greater the penalty, in order to encourage the robot to move quickly and smoothly.

[0072] Step 5: Use a course-based learning approach: Design training courses in ascending order of navigation distance to gradually expand the robot's navigation range and improve the adaptability of the policy network in a wide range of scenarios.

[0073] The specific steps for learning the course in step 5 are as follows:

[0074] Step 5.1, Construct a tiered training environment container: Construct N training environments with increasing difficulty in a continuous simulation space, and add the currently unlocked environment to the environment container; construct an environment library with 6 difficulty gradients based on the ROS-Gazebo simulation platform. Define an environment container to manage all unlocked environments in the current training stage. The stages to which the environments belong are as follows:

[0075] (1) Basic stage (levels 1-2): Set to a simple static environment, containing only a few regular geometric obstacles, with no dead ends, focusing on training the basic ability to move from the starting point to the target point in a straight line;

[0076] (2) Advanced stage (levels 3-4): Set up a complex static environment, introduce irregular obstacles, narrow passages and dead ends, and train path planning and escape ability;

[0077] (3) Advanced Stage (Levels 5-6): A dynamic and complex environment is set up, with randomly moving obstacles and sensor noise introduced to train dynamic obstacle avoidance and generalization abilities. Initially, the course... The environment container contains only the Level 1 environment.

[0078] Step 5.2, Set stage switching conditions: Monitor the robot's task success rate in the current stage environment in real time. When the task success rate is continuously higher than a preset threshold within a preset number of rounds, the switch will be initiated. At that time, the next difficulty level environment will be unlocked and added to the environment container;

[0079] Step 5.3, Perform periodic review sampling: During training, randomly sample environments from the environment container according to probability for training; assuming the current state is the... Phase 1, extracting the first environmental container The probability of an environment probability Satisfy the following formula: ,in, , Index for the environment phase;

[0080] Before each round of training, a weighted probability-based environment sampling strategy is used to randomly select an environment from the environment container according to probability. Provides interaction for robots, rather than directly using the current highest difficulty environment. Sampling probability From the current stage and the number of the sampled environment The decision was made jointly, and the calculation formula is as follows: ,in The design logic of this formula lies in: as the course progresses... The upgrade involves extracting the current highest difficulty environment. The probability is always highest, ensuring that the learning process mainly focuses on overcoming new challenges; at the same time, extracting from the old environment The probability is low but always greater than 0 (i.e., The remaining part), thereby forcing the robot to "regularly review" old knowledge during the training process, overcoming catastrophic forgetting;

[0081] Step 5.4, Collection and Update of Training Data. The robot is placed in the selected environment. Navigation task training is conducted within the system. At each time step, the current environmental state (LiDAR and odometry data) is acquired, an action is output, and a reward is obtained. Regardless of whether an old or new environment is selected, the generated trajectory data is used to update the policy network parameters, thereby enabling the network parameters to adapt to the new environment while retaining the memory of the characteristics of the old environment.

[0082] Step 5.5, Stage Advancement Assessment and Environment Expansion: To determine when to increase the course difficulty, the system needs to assess the robot's current highest stage. The robot's mastery of the target is assessed independently. The primary metric is the success rate, defined as the percentage of times the robot successfully reaches the target in 100 independent runs within the current environment, requiring no collisions and completion within a specified number of steps. A preset success rate threshold is set during this process. The success rate is 95%, meaning the robot's success rate consistently exceeds 95% for three consecutive rounds (i.e., 3 x 100 runs). When this occurs, a stage switching operation is triggered. The course stage is updated to... ; will the first The level environment is loaded into the environment container; the sampling probability of all unlocked environments is recalculated. ;

[0083] Step 5.6: Iterate until convergence, repeating steps 5.2 to 5.5. As the number of environments increases from 1 to 6, the environments contained within the container become increasingly diverse, and the sampling probability distribution continuously tilts towards higher-order environments while always covering lower-order environments. During this stage... When the advancement conditions are met (or the preset total number of training steps is reached), the training is considered complete, and the final navigation strategy with full-scene generalization capability is output.

[0084] Step 6: Introduce imitation learning to optimize robot motion posture: By collecting expert datasets and fine-tuning the Actor network, the policy network is guided to learn a motion pattern with high linear velocity and low angular velocity, so that the robot can maintain a stable posture during movement.

[0085] The specific implementation of imitation learning in step 6 is as follows:

[0086] Step 6.1: Read the expert dataset. The state data consists of 20-dimensional LiDAR readings and 2-dimensional polar coordinates of the robot relative to the target point, totaling 22-dimensional feature vectors. The motion data includes the robot's 4-dimensional control commands, namely the linear velocity and angular velocity of the previous and current actions. Then, all extracted data are concatenated to construct the overall dataset, which is then randomly divided into a training set and a test set in an 8:2 ratio.

[0087] Step 6.2, Constructing the Actor network for the imitation learning stage: The Actor network structure is [24, 800, 600, 2], where the input dimension is 24, passing through two hidden fully connected layers, and the output layer dimension is 2; the hidden layers all use the ReLU activation function, and the output layer uses the Tanh activation function to constrain the action output range; the Actor (policy) network built in the imitation learning stage has the same structure as the network used in TD3 training, which is a three-layer fully connected multilayer perceptron (MLP) structure with an output dimension of 2 (corresponding to the robot's linear velocity and angular velocity).

[0088] Step 6.3, Configure training parameters: The AdamW optimizer is used, the HuberLoss loss function is used, and a cosine annealing learning rate scheduler is introduced; in order to ensure the stability and efficiency of training, the following parameters are configured: (1) Optimizer: The AdamW optimizer is selected, the initial learning rate is set to 1e-7, and the weight decay coefficient is set to 1e-7 to prevent overfitting; (2) Loss function: HuberLoss is selected (parameter delta=1.0). Compared with MSE, HuberLoss is more robust to outliers; (3) Learning rate scheduling: The cosine annealing (Cosine AnnealingLR) strategy is introduced, and the scheduling period is set. The training time is set to 2000 epochs, which allows the learning rate to smoothly decrease to 1e-8 of its minimum value in the later stages of training. This is beneficial for the optimization process to converge to a better local optimum.

[0089] Step 6.4, Model Training: Supervised training of the Actor network is performed using the training set. The network parameters are updated through backpropagation. After training, the final Actor policy network parameters are saved for subsequent decision-making and control.

[0090] The training process consisted of 2000 rounds. In each round, state data was input into the Actor network in batches to obtain predicted actions, and the HuberLoss loss between the predicted actions and the real expert actions was calculated. Gradients were then calculated using backpropagation, and the network weights were updated using the optimizer. After training, the model was switched to Evaluation Mode, and the average loss was calculated and output on the test set to verify the model's generalization performance.

[0091] Experimental results, combined with Figure 2 , Figure 3 The environment shown and the corresponding Figure 5 , Figure 6 As shown in the navigation path, the model proposed in this invention significantly improves obstacle avoidance capabilities compared to the basic method. Specifically, this model can not only stably pass through narrow passages in complex environments, but also generates a more compact navigation path with an overall path length shorter than the basic TD3. In contrast, the basic TD3 is prone to collisions or significant detours under the same conditions, resulting in path redundancy, lower driving efficiency, and difficulty in achieving efficient and stable autonomous passage.

[0092] Further integration Figure 4 , Figure 7 As can be seen from the experimental environment and corresponding navigation path shown, when facing complex structures such as L-shaped obstacles that easily form closed or semi-closed spaces, the model of the present invention can complete reasonable turning in advance and successfully bypass the obstacles, while the basic TD3 will collide and thus fail to complete effective obstacle avoidance, thus verifying the reliability of the method of the present invention in complex obstacle scenarios.

[0093] In addition, by Figure 8 As shown in the comparison results of the angular velocity output, the model proposed in this invention has a significantly smaller angular velocity variation range than the basic TD3, and its angular velocity fluctuation range is significantly narrowed. This can effectively suppress the problem of frequent robot swaying during navigation, thereby significantly improving the robot's attitude stability during autonomous navigation.

[0094] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A scenario-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning, characterized in that, The method includes the following steps: Step 1, State Construction and Perception Fusion: The robot is equipped with a laser sensor to collect surrounding environmental data as laser information. At the same time, it obtains the polar coordinates of the target point relative to the robot's current position and the robot's motion state at the previous moment. The laser information, the polar coordinates of the target point relative to the robot's current position, and the motion state information of the robot at the previous moment are concatenated to form a state vector, which is used as the input of the dual-delay deep deterministic policy gradient policy network. Step 2, Construction and training of a dual-delay deep deterministic policy gradient policy network: Construct a dual-delay deep deterministic policy gradient policy network architecture, continuously interact with the environment in the Gazebo simulation environment to gain experience, and use the experience replay buffer to iteratively train and optimize the network parameters. The dual-delay deep deterministic policy gradient policy network architecture constructed in step 2 is an Actor-Critic network structure. The Actor network consists of multiple fully connected layers and ReLU activation functions. The output is generated by the Tanh activation function, which generates continuous actions in the interval (-1,1), corresponding to the robot's linear velocity and angular velocity. The Critic network adopts a dual-Q network parallel structure to evaluate the value of state-action pairs and takes the minimum Q value of the two Critic outputs as the final evaluation result. Step 3, Action Generation and Execution: The trained policy network outputs continuous linear velocity and angular velocity commands for the robot based on the input state to scale the actions. After the actions are scaled, the robot is driven to move, achieving obstacle avoidance and target point navigation. Step 4, Reward Design and Strategy Guidance: Design rewards for approaching the target, penalties for collisions, and rewards for the movement process to guide the strategy network to improve its ability to stably navigate to the target point while ensuring safe obstacle avoidance; Step 5: Use a course-based learning approach: Design training courses in ascending order of navigation distance to gradually expand the robot's navigation range and improve the adaptability of the policy network in a wide range of scenarios. Step 6: Introduce imitation learning to optimize robot motion posture: By collecting expert datasets and fine-tuning the Actor network, the policy network is guided to learn a motion pattern with high linear velocity and low angular velocity, so that the robot can maintain a stable posture during motion. The specific implementation of imitation learning in step 6 is as follows: Step 6.1, Construct an expert dataset: Manually collect expert example data of the robot in navigation tasks as navigation samples. Each sample contains 22-dimensional state data and 4-dimensional motion data. Divide the navigation samples into a training set and a test set according to a preset ratio. The state data consists of 20-dimensional laser sensor readings and the two-dimensional coordinates of the target point in Euclidean space. The motion data consists of the linear velocity and angular velocity of the previous and current actions. Step 6.2, Construct the Actor network for the imitation learning stage: The network structure of the Actor network is consistent with that of the Actor network in Step 2, wherein the input dimension is 24, the output dimension is 2, and two hidden fully connected layers are set in between; the hidden layers all use the ReLU activation function, and the output uses the Tanh activation function to constrain the range of action output; Step 6.3, Configure training parameters: Use the AdamW optimizer, the HuberLoss loss function, and introduce a cosine annealing learning rate scheduler; Step 6.4, Model Training: Supervised training of the Actor network is performed using the training set. The network parameters are updated through backpropagation. After training, the final Actor policy network parameters are saved for subsequent decision-making and control.

2. The scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning according to claim 1, characterized in that, In step 1, the robot is equipped with a laser sensor to collect environmental distance data within a 180° range in front of the robot as laser information. The laser information is evenly divided into 20 groups, each containing laser readings within a continuous angular range. The minimum value of each group's readings is taken as the feature value of that group. Simultaneously, the polar coordinates of the target point relative to the robot's current position and the robot's motion state information at the previous moment are acquired. The laser information, the polar coordinates of the target point relative to the robot's current position, and the robot's motion state information at the previous moment are concatenated to form an input state vector. The polar coordinates of the target point relative to the robot's current position include distance. ,angle The robot's motion state information at the previous moment includes linear velocity and angular velocity.

3. The scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning according to claim 2, characterized in that, The specific operation of step 2 is as follows: Step 2.1, the Actor network uses state vectors As input, two fully connected layers are sequentially connected and nonlinear mapping is performed using the ReLU activation function. The output uses the Tanh activation function to restrict the action to the interval (-1, 1), and finally outputs two action parameters. and , This is the original value of the linear velocity. This is the original value of the angular velocity; Step 2.2, the Critic network adopts a dual-Q network structure, with state vectors... As input, first the state vector Perform a linear mapping and then activate with ReLU to obtain an intermediate representation. Then the action With intermediate representation Two fully connected transformation layers are input separately, and feature fusion is achieved using the following formula: ; in, Represents a fully connected layer of composition. , This is the weight matrix. For bias; Step 2.3: Continuously interact with the environment in the Gazebo simulation environment to gain experience, and use the experience replay buffer to iteratively train and optimize the network parameters; during the training phase, a target network and a soft update mechanism are introduced, and Gaussian noise is added to the target action to improve the smoothness and stability of the strategy. After feature fusion, the Q value is output by ReLU activation. During the training process, the minimum Q value between the two Critic networks is selected as the evaluation result.

4. The scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning according to claim 3, characterized in that, In step 3, motion scaling maps the original motion output by the Actor network to actual motion commands. The motion scaling formula is: ; in, This represents the raw value of the linear velocity output by the Actor network. This represents the raw angular velocity value output by the Actor network. Indicates the maximum linear velocity. This represents the maximum angular velocity; after scaling, the linear velocity only takes positive values.

5. The scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning according to claim 4, characterized in that, In step 4, the reward function is set as follows: ; in, Indicates the robot at a certain moment The following state; Indicates the robot's state The following actions were taken; This indicates the current distance between the robot and the target. This represents the distance threshold to reach the target. Indicates positive goal reward. This indicates a negative reward for collisions. Indicates linear velocity. Indicates angular velocity; When the distance between the robot and the target Give positive rewards at that time. ; When a collision is detected, a negative reward is given. ; In other cases, the reward value is defined as The calculation formula is: ; ; in, Indicates the robot at a certain moment The instant reward received; Indicates the robot at a certain moment linear velocity; Indicates the robot at a certain moment The absolute value of the angular velocity; Indicates laser diopters; When laser diopters The reward is When the laser diopter When the robot is less than a distance threshold from an obstacle, an additional penalty is applied. .

6. The scene-generalizable end-to-end autonomous navigation method based on imitation-reinforcement learning according to claim 5, characterized in that, The specific steps for learning the course in step 5 are as follows: Step 5.1, Construct a hierarchical training environment container: Construct N training environments with increasing difficulty in a continuous simulation space, and add the currently unlocked environment to the environment container; the training environments are divided into basic, intermediate and advanced stages according to difficulty, corresponding to static regular obstacle environment, complex narrow passage and dead end environment, and dynamic obstacle and high noise environment, respectively. Step 5.2, Set stage switching conditions: Monitor the robot's task success rate in the current stage environment in real time. When the task success rate is continuously higher than a preset threshold within a preset number of rounds, the switch will be initiated. At that time, the next difficulty level environment will be unlocked and added to the environment container; Step 5.3, Perform periodic review sampling: During training, randomly sample environments from the environment container according to probability for training; assuming the current state is the... Phase 1, extracting the first environmental container The probability of an environment probability Satisfy the following formula: ,in, , Index for the environment phase; Step 5.4: Iterate until convergence, repeating steps 5.2 to 5.3 until the current stage. Training is considered complete when the advancement conditions are met or the preset total number of training steps is reached.

Images