A robot unknown environment autonomous exploration navigation method based on ctsac

Abstract

The application discloses a robot unknown environment autonomous exploration navigation method based on CTSAC, which combines the powerful sequence perception ability of the Transformer with the Soft Actor-Critic deep reinforcement learning framework, and introduces a periodic review mechanism, course learning, laser radar partition optimization processing and refined reward setting. The state of the robot is represented by the laser radar data and the gyroscope data after laser radar partition optimization processing. Meanwhile, the state information is used to obtain the corresponding reward value after refined reward setting, and the corresponding experience sequence is stored in the experience pool. The data sampled from the experience pool is used to train the neural network. The application is suitable for robots to perceive the surrounding environment through sensors, plan paths to reach the specified target point or establish an environment map in an unknown environment, and is used for complex tasks such as space exploration, search and rescue and reconnaissance.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for autonomous exploration and navigation of robots in unknown environments based on CTSAC. Background Technology

[0002] Autonomous exploration by robots in unknown environments is an extremely challenging task. It requires robots to perceive their surroundings using their onboard sensors, plan paths to reach designated target points or create environmental maps, and avoid collisions. This process is essentially a sequential decision-making problem, considered a nondeterministic multinomial-time (NP-Hard) problem due to its complexity. To address this challenge, reinforcement learning, a method that allows agents to autonomously learn skills through interaction with their environment, has been widely applied to solve complex robotic problems where humans struggle to pre-define rules. However, in the field of robot exploration, reinforcement learning still faces many challenges, such as poor perceptual reasoning capabilities, slow training convergence, and difficulty in achieving real-world simulations. Particularly noteworthy is that most existing methods rely solely on sensor information from the current frame for decision-making, lacking reasoning and comprehensive judgment based on historical information. This leads to robots easily getting stuck in a state of indecision during exploration and struggling to escape local optima. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a CTSAC-based autonomous exploration and navigation method for robots in unknown environments, enabling robots to explore and navigate more efficiently and accurately in unknown environments, thereby improving the overall system's exploration efficiency and generalization.

[0004] Technical Solution: A CTSAC-based autonomous exploration and navigation method for robots in unknown environments. The CTSAC module receives surrounding environment information from LiDAR and gyroscope and outputs robot motion actions. The robot performs motion actions in the environment, and the environment provides the CTSAC module with certain rewards. The CTSAC module executes the CTSAC algorithm, which mainly includes a periodic review mechanism for course learning, a Transformer-based SAC reinforcement learning algorithm, LiDAR partition optimization processing, and refined reward settings.

[0005] The course learning with a regular review mechanism serves as a training method for overall deep reinforcement learning, and is responsible for switching between environments.

[0006] Transformer-based SAC reinforcement learning is a SAC reinforcement learning paradigm used to train models, in which the neural network is composed of Transformer-based networks.

[0007] The robot's state is represented by LiDAR data and gyroscope data after partition optimization. Simultaneously, the state information is refined through reward settings to obtain the corresponding reward value r. t The corresponding experience sequences are stored in the experience pool, and the data sampled from the experience pool is used to train the neural network.

[0008] Furthermore, the implementation steps of the CTSAC algorithm are as follows:

[0009] S1, initialize the parameters of the Transformer-based SAC reinforcement learning neural network, and initialize the current training level j, success rate β, and environment container EC;

[0010] Environment E is obtained by sampling in the environment container EC. In Gazebo, the model in environment E is loaded through spawn_model, and the robot's current position, target position, and position of the i-th obstacle are randomly generated.

[0011] S3, the robot acquires 360° environmental information of its surroundings at the current moment through LiDAR, and then performs LiDAR partitioning optimization processing to obtain d. lidar 3D lidar information;

[0012] S4, the robot estimates its attitude using the inertial measurement unit to obtain its current position (x). robot ,y robot ), velocity and angular velocity (v) robot ,ω robot );

[0013] S5, based on the robot's current position (x) robot ,y robot ) and target location (x) target ,y target Using the Euclidean distance formula and the arctangent formula, the relative distance and angle (d,θ) of the target are calculated.

[0014] S6 combines n-dimensional data from the lidar, robot speed, and angular velocity (v). robot ,ω robot The relative distance and angle (d, θ) to the target are concatenated into a state vector, which is then normalized using the Min-Max normalization method to form the robot state S. t ;

[0015] S7, change robot state S t The Actor network in the Transformer-based SAC algorithm is input to obtain the robot's motion (v, ω). After obtaining the motion, the robot moves in the environment.

[0016] S8 refines the reward settings for the robot's movements based on its motion in the environment, and simultaneously stores continuous experience data in the experience pool. This experience data consists of a continuous quintuple experience sequence {s}. t ,a t ,r t ,s t+1 ,c t}composition;

[0017] S9, Calculate the robot's success rate: When the success rate is less than or equal to β, repeat steps S3 to S8; when the success rate is greater than β, add new environmental elements to the environmental container EC. (n) At the same time, the training level is increased by j+1, the success rate is reset to zero, and the experience pool is emptied.

[0018] When the training level reaches the training limit, all training is stopped, the neural network model is saved, and the training process ends.

[0019] S10, when the number of experience sequences in the experience pool exceeds the set value S min At that time, the training of SAC reinforcement learning is started; the parameters of the Actor network, the dual Q network, and the dual V network are optimized, and step S9 is run simultaneously to acquire experience and store the experience sequence in the experience pool.

[0020] Furthermore, in step S1, a dual Q network and a dual V network are used to combine the Transformer with a fully connected layer and serve as the neural network structure in the deep reinforcement learning model.

[0021] The Actor network consists of a policy selection network and a policy boosting network, which share network parameters. In the policy selection phase, the network input consists of all the robot's states at each step in the environment. The input is then passed through a fully connected layer for dimensionality upscaling before being fed into a Transformer layer to generate correlation information between each step and merge it with the current step information. Subsequently, the mean and variance of the actions are generated through two fully connected layers. In the policy boosting phase, the network receives experience sequences from the experience replay pool. The sequence data processed by the Transformer layer does not need to be averaged for dimensionality downscaling but is directly concatenated with the original data after dimensionality upscaling.

[0022] Furthermore, in step S3, the lidar zoning optimization process is as follows:

[0023] The LiDAR area is divided into several blocks according to its dimensions, and then the LiDAR data in each block is clustered; the angular resolution expression for the block is as follows:

[0024]

[0025] Where, Δθ md represents the angular resolution of the m-th block; the block number m indicates the position of the current block within the lidar coverage area; d represents the total number of blocks within the lidar coverage area.

[0026] Furthermore, the reward settings R are refined to include a turning penalty r1(a) r ), target proximity reward r2(d t Obstacle approach penalty r3 (mind) t ), wandering punishment r p The steps penalty is -λ7, the goal reward is λ1, and the collision penalty is -λ2, expressed as follows:

[0027]

[0028] Turning penalty r1(a r The expression for ) is as follows:

[0029]

[0030] Among them, a r Angular velocity;

[0031] Target proximity reward r2(d) t The expression for ) is as follows:

[0032]

[0033] When the distance d between the robot and the obstacle t A reward will be given for distances less than 10 meters.

[0034] Obstacle approach penalty r3(mind) t The expression for ) is as follows:

[0035]

[0036] mind t The laser radar information is converted into a minimum distance value to the obstacle; when the minimum distance value is less than 1 meter, a certain penalty will be imposed.

[0037] Wandering punishment r p By statistically calculating the current position (x, y) and the stored historical position (x, y), i ,y i The number of Manhattan distances less than δ is penalized; the expression is as follows:

[0038] d m (x,y,x i ,y i )=|xx i |+|yy i |

[0039]

[0040] Where, d m () represents the current position (x, y) and the historical position (x, y) stored in this set. i ,y i The relative distance between ); 1(·) is an indicator function that returns 1 if the condition is true, otherwise returns 0;

[0041] Step penalty -λ7: The purpose is to encourage the robot to reach the target point as quickly as possible and reduce unnecessary detours; a constant step penalty is added to each step to motivate the robot to move along a more direct path;

[0042] Reward λ1 for reaching the target: When the robot successfully reaches the target, a positive reward is given and the current training round is terminated;

[0043] Collision Penalty -λ2: When the robot collides with an obstacle, a negative reward will be given and the current training round will be terminated.

[0044] Compared with the prior art, the significant advantages of this invention are as follows:

[0045] 1. This invention employs the Transformer-based SAC (Soft Actor-Critic) reinforcement learning algorithm, combining the powerful sequence perception capability of Transformer with SAC reinforcement learning based on uncertain entropy to enhance the decision-making and reasoning ability of the agent; by using the robot's historical state as the input of the neural network, it effectively solves the problem of the robot wandering and getting stuck in a dead end with a single frame input.

[0046] 2. The periodic review mechanism proposed in this invention designs a periodic review mechanism based on course learning, which avoids the problems of low efficiency and divergent learning caused by learning directly in a complex environment, and at the same time solves the problem of catastrophic forgetting in course learning.

[0047] 3. The LiDAR partitioning optimization process proposed in this invention optimizes the LiDAR partitioning for the robot's forward direction, while performing clustering to extract useful information and reduce misjudgments caused by noise or abnormal data. It also reduces the dimensionality of sensor data and narrows the gap between simulation and reality, thereby improving sim-to-real performance.

[0048] 4. The refined reward settings proposed in this invention achieve a balance between exploration and utilization through seven specific mechanisms. Through these reward and punishment mechanisms, the robot can efficiently complete autonomous exploration tasks, avoid over-reliance on certain strategies (such as over-exploration or wandering), and effectively improve the stability and convergence speed of training. Attached Figure Description

[0049] Figure 1 This is a diagram of the overall framework of the robot and reinforcement learning used in this invention;

[0050] Figure 2 This is a schematic diagram of the overall framework of the CTSAC algorithm;

[0051] Figure 3 This is a schematic diagram of the LiDAR partitioning optimization process;

[0052] Figure 4a It is the Actor policy in CTSAC that boosts the network;

[0053] Figure 4b It is the Actor policy selection network in CTSAC;

[0054] Figure 5 This is a diagram of the dual-V network architecture in CTSAC;

[0055] Figure 6 This is a diagram of the dual-Q network architecture in CTSAC;

[0056] Figure 7 It is a training and testing map in course learning based on a review mechanism. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0058] This invention provides a method for autonomous exploration and navigation of robots in unknown environments based on CTSAC (Curriculum Transformer Soft Actor-Critic). It overcomes the limitations of traditional reinforcement learning algorithms, such as slow training convergence, susceptibility to local optima, and difficulty adapting to complex environmental changes. By introducing the SAC reinforcement learning algorithm based on the Transformer architecture, the robot's reasoning ability based on historical information and the environment is enhanced. Periodic learning mechanisms improve training efficiency and accelerate convergence. Partition optimization using LiDAR reduces the discrepancy between simulated and real-world sensor information, improving simulation-to-real transfer performance. Refined reward settings, through these reward and punishment mechanisms, enable the robot to efficiently complete autonomous exploration tasks. Ultimately, this allows the robot to more efficiently and accurately explore and navigate in unknown environments, improving the overall system's exploration efficiency and generalization ability.

[0059] The course learning with a regular review mechanism aims to address the catastrophic forgetting problem common in course learning. In each training stage of the course learning, the learning environment of the previous stages is reproduced at a certain frequency. This ensures successful learning in the new environment and effectively retains the knowledge gained in the previous environment. It also speeds up the convergence speed, improves sampling efficiency, and avoids the agent getting stuck in local optima, which can lead to training divergence.

[0060] The Transformer-based SAC (Soft Actor-Critic) reinforcement learning algorithm is a combination of maximum entropy reinforcement learning and Transformer neural networks. Employing the SAC framework, it aims to maximize both the expected reward and the policy entropy, balancing exploration and exploitation. The algorithm designs three neural networks: Actor, Critic_Q, and Critic_V. A double-Q network is used to avoid Q-value overestimation, and a double-V network is used to stabilize the training process and avoid overestimation bias. The Transformer structure is used to extract robot state sequence information, enhancing the model's reasoning ability on sequential data.

[0061] LiDAR partitioning optimization is a preprocessing technique for LiDAR data designed to enhance a robot's perception of its surroundings. When processing LiDAR data, the LiDAR area is first divided into several blocks based on its dimensions, and the data within each block is then clustered. Traditional partitioning methods typically divide the area into several equal parts, but this may not meet the robot's need for more detailed perception of the area ahead. Therefore, optimization processing is performed to enable the robot to perceive the area ahead more precisely when facing the same obstacle, thereby improving the robot's navigation and exploration capabilities. This optimization is of great significance for achieving end-to-end target-oriented exploration.

[0062] To refine the reward system for autonomous exploration tasks, seven specific mechanisms were designed: turning penalty, target proximity reward, obstacle approach penalty, wandering penalty, step count penalty, target arrival reward, and collision penalty. These mechanisms help the robot explore autonomously in complex and unknown environments while achieving a balance between exploration and utilization. Through these reward and penalty mechanisms, the robot can complete tasks efficiently, avoid over-reliance on certain strategies (such as overexploration or wandering), and effectively improve the stability and convergence speed of training.

[0063] like Figure 1 The diagram shown illustrates the overall framework of the robot and reinforcement learning employed in this invention. The CTSAC module receives environmental information from the lidar and gyroscope and outputs robot motion actions. The robot performs these actions within the environment, while the environment provides the CTSAC module with a reward. Figure 2The diagram shows the flowchart of the CTSAC algorithm, which mainly includes a periodic review mechanism for course learning, a Transformer-based SAC reinforcement learning algorithm, LiDAR partition optimization processing, and refined reward settings. The periodic review mechanism serves as the overall deep reinforcement learning training method, responsible for tasks such as environment switching. The Transformer-based SAC reinforcement learning is the SAC reinforcement learning paradigm used to train the model, where the neural network consists of a Transformer-based network. The robot's state is represented by LiDAR data after partition optimization processing and gyroscope data. Simultaneously, the state information, after refining the reward settings, yields the corresponding reward value r. t The corresponding experience sequences are stored in an experience pool, and the data used for training the neural network is sampled from the experience pool. The implementation of the CTSAC algorithm includes the following steps:

[0064] Step 1: Initialize the parameters of the Transformer-based SAC reinforcement learning neural network (Actor network, dual Critic Q network, dual Critic V network), initialize the current training level j to 0, initialize the success rate to 0, and initialize the environment container EC to e. (0) , where e (0) For environment number 0;

[0065] The Transformer-based SAC reinforcement learning algorithm includes the following:

[0066] We employ Soft-Actor-Critic reinforcement learning as the training method, and use the off-policy actor-critic framework based on maximum entropy reinforcement learning to maximize the policy entropy while maximizing the expected reward.

[0067]

[0068] Where, π * For target strategy; To find the expectation of the function within parentheses under the probability distribution of the trajectory τ generated by strategy π; r(s) t ,a t Let be the reward function, evaluating action a. t In state s t The value is γ; γ is the discount factor, weighing short-term and long-term returns; t is the time step. Let be the entropy of policy π, representing the randomness of the policy; α1 is the temperature coefficient, controlling the influence of randomness on the optimization objective; π(·|s t ) is in state s t The strategy π below.

[0069] By employing dual Q-networks and dual V-networks, overestimation is avoided while improving the stability of the training process; the Transformer is combined with fully connected layers as the neural network structure in the deep reinforcement learning model to enhance the time-series inference capability of deep reinforcement learning.

[0070] like Figure 4a , 4b As shown, the Actor network consists of two parts: a policy selection network and a policy boosting network, which share network parameters. In the policy selection phase, the network input consists of all the robot's states at each step in the environment. The input is boosted through fully connected layers and then fed into the Transformer layer to generate correlation information between steps, which is then merged with the current step information. Subsequently, two fully connected layers are used to generate the mean and variance of the actions. In the policy boosting phase, the network receives experience sequences from the experience replay pool. The sequence data processed by the Transformer layer does not require averaging down; instead, it is directly concatenated with the boosted original data. This design aims to generate action data with sequence dimensions for the Q-network to learn. The objective function of the Actor network is J. π (φ) is shown in the following formula:

[0071]

[0072] Among them, J π (φ) represents strategy π φ The state objective function is given by φ, where φ is the parameter of the policy network; α2 is the hyperparameter controlling the randomness of the exploration; π φ (a t s t ) indicates according to strategy π φ In state s t Take action a t The probability of Q; θ (s t ,a t ) is in state s t Take action a t The objective function of the action; Indicates the experience replay pool Sampling of s t and from The sampled Gaussian noise ε t Given the joint distribution of , find the expectation of the function within the parentheses.

[0073] like Figure 5 As shown, a dual-V network: one for estimating the current state value V(s) t A V-network, used to compute the objective value V(s) t+1The V-target network is a V-target network whose parameters are obtained through soft updates of the V network. The structure of the dual-V network is similar to that of the Actor network for policy selection. The input of the dual-V network is s, but unlike s, the dual-V network outputs only a one-dimensional numerical value V(s). The objective function of the V network is J. V (ψ) is shown in the following formula:

[0074]

[0075] Among them, J v (ψ) is a value function V ψ (s t The objective function of ) Indicates the experience replay pool Medium sampling t Find the expectation of the function within parentheses under the probability distribution; ψ is the parameter of the value function V, V ψ (s t ) is the current value function V ψ For state s t The predicted value; Indicates from strategy π φ Sample a t The expected value of the function within the parentheses is calculated under the probability distribution, where α3 represents the hyperparameter controlling the randomness of the exploration.

[0076] like Figure 6 As shown, the dual-Q network consists of two Q-networks with identical structures, but their parameters are independent and updated independently. The final Q-value is the minimum of the outputs of these two networks. The structure of the dual-Q network is very similar to that of the dual-V network, with the main difference being the input data: the input to the Q network is the superposition of s and a, while the other network structures remain consistent with the V network. The objective function J of the Q network... Q (θ) is shown in the following formula:

[0077]

[0078]

[0079]

[0080] Among them, J Q (θ) is the objective function of the Q-value function, used to minimize the difference between the predicted value and the target value; Indicates the experience replay pool Medium sampling (s) t ,a t Find the expectation of the function within the parentheses under the probability distribution of ). The target value for Q is determined by combining immediate rewards and a value function; γ is a discount factor that weighs the importance of short-term versus long-term returns. The value function estimates the expected cumulative reward for the next state; α0 is the initial temperature coefficient, setting the initial value for the entropy regularization weights; τ is the adjustment rate, controlling the decay rate of α; and n is the number of training steps.

[0081] Step 2: When the success rate is less than the set value, initialize the training environment E, according to P. (i,j) The training environment E is obtained by sampling from the environment container EC. In the Gazebo software, the model from the training environment E is loaded using `spawn_model`, and the robot's current position (x) is randomly generated. robot ,y robot ), target location (x) target ,y target ) and the position of the i-th obstacle (x) i obstacle ,y i obstacle ).

[0082] Step 3: The robot acquires 360° environmental information around it at the current moment using LiDAR, and then performs LiDAR partitioning optimization processing to obtain d. lidar 3D lidar information;

[0083] like Figure 3 As shown, the LiDAR zonal optimization process includes the following:

[0084] The LiDAR area is divided into several blocks according to its dimensions, and then the LiDAR data in each block is clustered. For a robot moving forward, the perception of the area in front is more important. Therefore, the LiDAR block segmentation method is optimized to make the blocks in front of the robot denser and the blocks behind it sparser. Under the same LiDAR dimension, facing the same obstacle, the optimized segmentation method allows the robot to perceive the area in front more precisely. The angular resolution expression of the block is as follows:

[0085]

[0086] Where, Δθ m This represents the angular resolution (or angular step size) of the m-th block; the smaller the angular resolution, the denser the blocks, and the higher the corresponding perception accuracy; the block number m represents the position of the current block within the lidar coverage area; d represents the total number of blocks within the lidar coverage area.

[0087] Step 4: The robot performs attitude estimation using an IMU (Inertial Measurement Unit) to obtain its current position (x).robot ,y robot ), velocity and angular velocity (v) robot ,ω robot );

[0088] Step 5, based on the robot's current position (x robot ,y robot ) and target location (x) target ,y target Using the Euclidean distance formula and the arctangent formula, the relative distance and angle (d,θ) of the target are calculated.

[0089] Step 6: Combine the n-dimensional data from the lidar, the robot's velocity, and its angular velocity (v). robot ,ω robot The relative distance and angle (d, θ) to the target are concatenated into a state vector, which is then normalized using the Min-Max normalization method to form the robot state S. t .

[0090] Step 7, set the robot state S t The Actor network in the Transformer-based SAC algorithm is input to obtain the robot's motion (v, ω). After obtaining the motion, the robot moves in the environment.

[0091] Step 8: Based on the robot's movement in the environment, refine the reward settings for the robot's movement, and simultaneously store continuous experience data in the experience pool. The experience data consists of a continuous quintuple experience sequence {s}. t ,a t ,r t ,s t+1 ,c t}composition;

[0092] The reward settings have been refined, including the following:

[0093] The detailed reward settings consist of 7 parts, namely the turning penalty r1(a r ), target proximity reward r2(d t Obstacle approach penalty r3 (mind) t ), wandering punishment r p The steps penalty is -λ7, the goal reward is λ1, and the collision penalty is -λ2, expressed as follows:

[0094]

[0095] Where R represents refined reward settings; the specific description is as follows:

[0096] Step 81, turn penalty r1(a) rTo reduce the frequency of robot turning, its angular velocity a r A punishment mechanism has been imposed.

[0097]

[0098] Step 82, target proximity reward r2(d) t To encourage the robot to move towards the target location, when the robot is at a distance d from the obstacle... t A reward will be given for distances less than 10 meters.

[0099]

[0100] Step 83, obstacle approach penalty r3 (mind t To prevent the robot from getting too close to obstacles, this invention converts LiDAR information into a minimum distance value (mind) to the obstacle. t When the minimum distance is less than 1 meter, a penalty will be imposed.

[0101]

[0102] Step 84, lingering punishment r p To prevent the robot from getting stuck in a local optimum (preventing it from spinning or lingering at a certain position) and to incentivize it to explore more extensively, this invention designs a lingering penalty mechanism. This mechanism calculates the difference between the current position (x, y) and the stored historical position (x, y). i ,y i The number of Manhattan distances less than δ is penalized.

[0103] d m (x,y,x i ,y i )=|xx i |+|yy i | (12)

[0104]

[0105] Where d m The current position (x, y) and the historical position stored in this set (x, y) are used to determine the relative positions of the current position and the historical position stored in this set. i ,y i The relative distance is 1(·), which is an indicator function that returns 1 if the condition is true and 0 otherwise.

[0106] Step 85, Step Penalty -λ7: In order to encourage the robot to reach the target point as soon as possible and reduce unnecessary detours, this invention adds a constant step penalty to each step, which motivates the robot to move forward along a more direct path, thereby effectively reducing ineffective exploration behavior.

[0107] Step 86, Reward λ1 for reaching the target: When the robot successfully reaches the target, give a positive reward and terminate the current training round.

[0108] Step 87, Collision Penalty -λ2: When the robot collides with an obstacle, a negative reward will be given and the current training round will be terminated.

[0109] Step 9: Calculate the robot's success rate;

[0110] When the success rate is less than or equal to β, repeat steps 3 to 8;

[0111] When the success rate is greater than β, a new environment e is added to the environment container EC. (n) At the same time, the training level is increased by j+1, the success rate is reset to zero, and the experience pool is emptied.

[0112] If the current training level reaches the training limit, stop all training, save the neural network model, and end the training process.

[0113] Step 10: When the number of experience sequences in the experience pool exceeds the set value S min At that time, the training of SAC reinforcement learning is started; during this process, SAC is trained, the parameters of the Actor network, the dual Q network, and the dual V network are optimized, and step 9 is run at the same time to acquire experience and store the experience sequence in the experience pool.

[0114] The pseudocode for the CTSAC algorithm is shown in Table 1.

[0115] Table 1 CTSAC Pseudocode

[0116]

[0117]

[0118] In Table 1, `success_rate` represents the success rate, and `success history[k]` indicates whether the k-th experiment was successful; 1 indicates success, and 0 indicates failure. Figure 7 The image shows training maps at different levels in the course learning process of the periodic review mechanism. The first row is the training map, and the second row is the test map.

Claims

1. A method for autonomous exploration and navigation of robots in unknown environments based on CTSAC, characterized in that, The CTSAC module receives environmental information from the LiDAR and gyroscope and outputs robot motion actions. The robot performs motion actions in the environment, and the environment gives the CTSAC module a certain reward. The CTSAC module executes the CTSAC algorithm, which mainly includes a periodic review mechanism for course learning, a Transformer-based SAC reinforcement learning algorithm, LiDAR partition optimization processing, and refined reward settings. The course learning with a regular review mechanism serves as a training method for overall deep reinforcement learning, and is responsible for switching between environments. Transformer-based SAC reinforcement learning is a SAC reinforcement learning paradigm used to train models, in which the neural network is composed of Transformer-based networks. The robot's state is represented by LiDAR data and gyroscope data after partition optimization. Simultaneously, the state information is used to obtain corresponding reward values ​​after refining the reward settings. The corresponding experience sequences are stored in the experience pool, and the data sampled from the experience pool is used to train the neural network. The implementation steps of the CTSAC algorithm are as follows: S1, Initialize the parameters of the Transformer-based SAC reinforcement learning neural network, and initialize the current training level. Success rate Environmental containers (EC); Environment E is obtained by sampling in the environment container EC. In Gazebo, the model in environment E is loaded via spawn_model, and the robot's current position, target position, and position are randomly generated. The location of the obstacle; S3, the robot acquires 360° environmental information of its surroundings at the current moment through LiDAR, and then performs LiDAR partitioning optimization processing to obtain... 3D lidar information; S4, the robot estimates its attitude using the inertial measurement unit to obtain its current position. velocity and angular velocity ; S5, based on the robot's current position and target location Using the Euclidean distance formula and the arctangent formula, the relative distance and angle to the target are calculated. ; S6 combines n-dimensional data from LiDAR, robot speed, and angular velocity. Target relative distance and angle The states are concatenated into a single state vector and then normalized using the Min-Max normalization method to form the robot's state. ; S7, change robot status The robot's motion is obtained by inputting the Actor network from the Transformer-based SAC algorithm. After acquiring the action, the robot moves in the environment; S8 refines the reward settings for the robot's movements based on its actions in the environment, and simultaneously stores continuous experience data in the experience pool. The experience data consists of a continuous sequence of five-tuple experience data. composition; S9, Calculate the robot's success rate: When the success rate is less than or equal to... When the success rate is greater than 10%, repeat steps S3 to S8; When adding new environmental elements to the environmental container EC Simultaneously training level Success rate resets to zero, experience pool emptied; When the training level reaches the training limit, all training is stopped, the neural network model is saved, and the training process ends. S10, when the number of experience sequences in the experience pool exceeds a set value At that time, the training of SAC reinforcement learning is started; the parameters of the Actor network, the dual Q network, and the dual V network are optimized, and step S9 is run simultaneously to acquire experience and store the experience sequence in the experience pool.

2. The CTSAC-based autonomous exploration and navigation method for robots in unknown environments according to claim 1, characterized in that, In step S1, a dual Q network and a dual V network are used to combine the Transformer with a fully connected layer and serve as the neural network structure in the deep reinforcement learning model. The Actor network consists of a policy selection network and a policy boosting network, which share network parameters. In the policy selection phase, the network input consists of all the robot's states at each step in the environment. The input is then passed through a fully connected layer for dimensionality upscaling before being fed into the Transformer layer to generate correlation information between steps and merge it with the current step information. Subsequently, the mean and variance of the actions are generated through two fully connected layers. In the policy boosting phase, the network receives experience sequences from the experience replay pool. The sequence data processed by the Transformer layer does not need to be averaged for dimensionality downscaling; instead, it is directly concatenated with the original data after dimensionality upscaling.

3. The CTSAC-based autonomous exploration and navigation method for robots in unknown environments according to claim 1, characterized in that, In step S3, the lidar zoning optimization process is as follows: The LiDAR area is divided into several blocks according to its dimensions, and then the LiDAR data in each block is clustered; the angular resolution expression for the block is as follows: ； in, Indicates the first Angular resolution of each block; block number This indicates the current location of the block within the lidar coverage area; This indicates the total number of blocks within the lidar coverage area.

4. The CTSAC-based autonomous exploration and navigation method for robots in unknown environments according to claim 1, characterized in that, Refine the reward settings, including turning penalties. Target proximity reward Obstacle approach penalty , lingering punishment Step count penalty Rewards for reaching the target Collision penalty The expression is as follows: ； Turning to punishment The expression is as follows: ； in, Angular velocity; Target proximity reward The expression is as follows: ； When the robot is at a distance from the obstacle A reward will be given for distances less than 10 meters. Obstacle approach penalty The expression is as follows: ; The laser radar information is converted into a minimum distance value to the obstacle; when the minimum distance value is less than 1 meter, a certain penalty will be imposed. Wandering punishment Calculate the current location using statistics. With stored historical location Manhattan distance is less than The quantity is used as a penalty; the expression is as follows: ； ； in, ( ) represents the current position With stored historical location The relative distance; It is an indicator function that returns 1 if the condition is true, otherwise it returns 0; Step penalty The goal is to encourage the robot to reach the target point as quickly as possible and reduce unnecessary detours; a constant step penalty is added to each step to motivate the robot to move along a more direct path. Reward for reaching the target When the robot successfully reaches the goal, a positive reward is given, and the current training round is terminated; Collision Penalty When the robot collides with an obstacle, a negative reward will be given and the current training round will be terminated.

Images