Robot unknown environment autonomous exploration navigation method based on CTSAC

Abstract

The invention discloses a robot unknown environment autonomous exploration navigation method based on CTSAC, and the method combines the powerful sequence perception capability of Transform and a Soft Actor-Critic deep reinforcement learning framework, and introduces curriculum learning of a regular review mechanism, laser radar partition optimization processing, and refined reward setting. The state of the robot is represented by laser radar data and gyroscope data which are subjected to laser radar partition optimization processing, meanwhile, after state information is subjected to refined reward setting, corresponding reward values are obtained, corresponding experience sequences are stored in an experience pool, and data obtained through sampling of the experience pool are supplied to a neural network for training. The method is suitable for the robot to perceive the surrounding environment through the sensor in an unknown environment to perform path planning so as to reach a specified target point or establish an environment map, and is used for complex tasks such as space exploration, search and rescue, reconnaissance and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a method for autonomous exploration and navigation of a robot in an unknown environment based on CTSAC. Background Art

[0002] Autonomous exploration of a robot in an unknown environment is a highly challenging task, which requires the robot to sense the surrounding environment through its equipped sensors, perform path planning to reach a specified target point or build an environmental map, and at the same time avoid collisions with the environment. This process essentially belongs to a sequential decision-making problem, which is regarded as a non-deterministic polynomial time hard (NP-Hard) problem due to its complexity. To address this challenge, reinforcement learning, as a method of autonomously learning skills through the interaction between an agent and the environment, has been widely used to solve complex robot problems that are difficult for humans to preset rules. However, in the field of robot exploration, reinforcement learning still faces many problems, such as poor perception and reasoning ability, slow training convergence, and difficulty in sim-to-real. In particular, most of the existing methods only rely on the sensor information of the current frame for decision-making, lacking the reasoning and comprehensive judgment of historical information, which causes the robot to easily fall into a wandering state during the exploration process and is difficult to jump out of the local optimal solution. Summary of the Invention

[0003] Object of the Invention: The object of the present invention is to provide a method for autonomous exploration and navigation of a robot in an unknown environment based on CTSAC, so as to enable the robot to more efficiently and accurately explore and navigate in an unknown environment, and improve the exploration efficiency and generalization of the overall system.

[0004] Technical Solution: A method for autonomous exploration and navigation of a robot in an unknown environment based on CTSAC, the CTSAC module receives the surrounding environment information obtained from a lidar and a gyroscope, and outputs the motion actions of the robot. The robot executes the motion actions in the environment, and at the same time, the environment gives a certain reward to the CTSAC module; the CTSAC module executes the CTSAC algorithm, and the CTSAC algorithm mainly includes curriculum learning with a regular review mechanism, SAC reinforcement learning algorithm based on Transformer, lidar partition optimization processing, and refined reward setting;

[0005] Curriculum learning with a regular review mechanism, as the training method of the overall deep reinforcement learning, is responsible for performing the switching task of the environment;

[0006] SAC reinforcement learning based on Transformer is the SAC reinforcement learning paradigm used to train the model, and the neural network therein is composed of a network based on Transformer;

[0007] The state of the robot is represented by the lidar data and gyroscope data after lidar partition optimization. Meanwhile, after refining the reward settings, the state information obtains the corresponding reward value r t , and stores the corresponding experience sequence into the experience pool. The data sampled from the experience pool is supplied to the neural network for training.

[0008] Furthermore, the implementation steps of the CTSAC algorithm are as follows:

[0009] S1, Initialize the parameters of the SAC reinforcement learning neural network based on Transformer, initialize the current training level j, success rate β, and environment container EC;

[0010] Sample the environment E from the environment container EC, load the model in E into Gazebo through spawn_model, and randomly generate the current position of the robot, the target position, and the position of the i-th obstacle;

[0011] S3, The robot obtains the 360° surrounding environment information at the current moment through the lidar, and then performs lidar partition optimization to obtain the d lidar -dimensional lidar information;

[0012] S4, The robot performs attitude estimation through the inertial measurement unit to obtain the position (x robot , y robot ), speed, and angular velocity (v robot , ω robot ) of the robot at the current moment;

[0013] S5, According to the current position (x robot , y robot ) and the target position (x target , y target ) of the robot, use the Euclidean distance calculation formula and the arctangent formula to calculate the target relative distance and angle (d, θ);

[0014] S6, Concatenate the lidar n-dimensional data, the speed and angular velocity (v robot , ω robot ) of the robot, and the target relative distance and angle (d, θ) into a state vector, and normalize it through the Min-Max normalization method to form the robot state S t ;

[0015] S7, Input the robot state S t into the Actor network in the SAC algorithm based on Transformer to obtain the actions (v, ω) of the robot. After the robot obtains the actions, it moves in the environment;

[0016] S8. Refine the reward settings for the robot's motion according to the robot's motion in the environment, and at the same time store the continuous experience data in the experience pool. The experience data consists of a continuous quintuple experience sequence {s t , a t , r t , s t+1 , c t}.

[0017] S9. Calculate the success rate of the robot: When the success rate is less than or equal to β, repeat steps S3 to S8; when the success rate is greater than β, add a new environment e to the environment container EC (n) , and at the same time train level j + 1, clear the success rate, and empty the experience pool;

[0018] When the training level reaches the training limit, stop all training, save the neural network model, and end the training process;

[0019] S10. When the number of experience sequences in the experience pool exceeds the set value S min , start the training of SAC reinforcement learning; optimize the parameters of the Actor network, the double Q network, and the double V network, and at the same time run step S9 to obtain experiences and store the experience sequences in the experience pool.

[0020] Furthermore, in step S1, a double Q network and a double V network are adopted, and Transformer is combined with a fully connected layer and used as the neural network structure in the deep reinforcement learning model;

[0021] The Actor network includes a policy selection network and a policy improvement network, and the policy selection network and the policy improvement network share network parameters; in the policy selection stage, the network input is all the states of each step of the robot in the environment; after the input is dimensionally increased by the fully connected layer, it is passed into the Transformer layer to generate the correlation information between each step and merged with the current step information; subsequently, the mean and variance of the actions are generated through two fully connected layers respectively; in the policy improvement stage, the network receives the experience sequence from the experience replay pool, and the sequence data processed by the Transformer layer does not need to be averaged and dimensionally reduced, but is directly concatenated with the original data after dimensional increase.

[0022] Furthermore, in step S3, the lidar zoning optimization is processed as follows:

[0023] Divide the area of the lidar into several blocks according to its dimension, and then perform clustering processing on the lidar data in each block; the angular resolution expression for block division is as follows:

[0024]

[0025] Among them, Δθ mRepresents the angular resolution of the m-th block; the block number m indicates the position of the current block in the lidar coverage; d represents the total number of blocks in the lidar coverage.

[0026] Furthermore, the refined reward setting R includes a steering penalty r 1 (a r ), a target proximity reward r 2 (d t ), an obstacle approach penalty r 3 (mind t ), a wandering penalty r p , a step penalty -λ 7 , a target arrival reward λ 1 , a collision penalty -λ 2 , and the expression is as follows:

[0027]

[0028] The steering penalty r 1 (a r ) has the following expression:

[0029]

[0030] where a r is the angular velocity;

[0031] The target proximity reward r 2 (d t ) has the following expression:

[0032]

[0033] When the distance d t between the robot and the obstacle is less than 10 meters, a certain reward is given;

[0034] The obstacle approach penalty r 3 (mind t ) has the following expression:

[0035]

[0036] mind t is the minimum distance value converted from the lidar information to the obstacle; when the minimum distance value is less than 1 meter, a certain penalty is given;

[0037] The wandering penalty r p is calculated by counting the number of Manhattan distances between the current position (x,y) and the stored historical position (x i ,y i ) that are less than δ as a penalty; the expression is as follows:

[0038] d m (x, y, x i , y i ) = |x - x i | + |y - y i |

[0039]

[0040] where d m () is the relative distance between the current position (x, y) and the historical position (x i , y i ) stored in this set; 1(·) is an indicator function that returns 1 if the condition is true and 0 otherwise;

[0041] Step penalty -λ 7 : The purpose is to prompt the robot to reach the target point as soon as possible and reduce unnecessary detours; a constant step penalty is added in each step to encourage the robot to move forward along a more direct path;

[0042] Reaching target reward λ 1 : When the robot successfully reaches the target, a positive reward is given and the current training episode is terminated;

[0043] Collision penalty -λ 2 : When the robot collides with an obstacle, a negative reward is given and the current training episode is terminated.

[0044] Compared with the prior art, the remarkable effects of the present invention are as follows:

[0045] 1. The present invention adopts a Transformer-based SAC (Soft Actor-Critic) reinforcement learning algorithm, which combines the powerful sequence perception ability of Transformer with SAC reinforcement learning based on uncertain entropy to enhance the decision-making and reasoning ability of the agent; using the historical state of the robot as the input of the neural network effectively solves the situation where the robot wanders into a dead end with single-frame input;

[0046] 2. The curriculum learning with a regular review mechanism proposed by the present invention designs a regular review mechanism on the basis of curriculum learning, avoiding the problems of low efficiency and learning divergence caused by directly learning in a complex environment, and at the same time solving the problem of catastrophic forgetting in curriculum learning;

[0047] 3. The lidar partition optimization processing proposed by the present invention optimizes the lidar partition according to the forward direction of the robot, and at the same time performs clustering to extract useful information while reducing misjudgment caused by noise or abnormal data, reducing the sensor data dimension and reducing the gap between simulation and reality, and improving the sim-to-real performance;

[0048] 4. The refined reward setting proposed by the present invention achieves a balance between exploration and exploitation through 7 specific mechanisms; through these reward and punishment mechanisms, the robot can efficiently complete autonomous exploration tasks, avoid over-reliance on certain strategies (such as over-exploration or wandering), and effectively improve the stability and convergence speed of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is the overall framework diagram of the robot and reinforcement learning adopted by the present invention;

[0050] Figure 2 is the overall framework schematic diagram of the CTSAC algorithm;

[0051] Figure 3 is the schematic diagram of the optimized processing of lidar zoning;

[0052] Figure 4a is the Actor policy improvement network in CTSAC;

[0053] Figure 4b is the Actor policy selection network in CTSAC;

[0054] Figure 5 is the dual V network architecture diagram in CTSAC;

[0055] Figure 6 is the dual Q network architecture diagram in CTSAC;

[0056] Figure 7 is the training and test map in curriculum learning based on the review mechanism. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings of the specification and the specific embodiments.

[0058] The present invention provides a method for autonomous exploration and navigation of a robot based on CTSAC (Curriculum Transformer Soft Actor-Critic), which breaks through the limitations of traditional reinforcement learning algorithms in robot exploration, such as slow training convergence, easy to fall into local optimum, and difficult to adapt to complex environmental changes. By introducing the SAC reinforcement learning algorithm with the Transformer architecture, the reasoning ability of the robot for historical information and the environment is enhanced; by introducing curriculum learning with a regular mechanism, the training efficiency is improved and the convergence is accelerated; by introducing optimized processing of lidar partitioning, the difference between simulation and real sensor information is reduced, and the sim-to-real transfer performance is improved; the reward setting is refined. Through these reward and punishment mechanisms, the robot can efficiently complete the autonomous exploration task. Finally, the robot can explore and navigate more efficiently and accurately in an unknown environment, improving the exploration efficiency and generalization of the overall system.

[0059] For curriculum learning with a regular review mechanism, to solve the common catastrophic forgetting problem in curriculum learning, the learning environment of the previous few stages is reproduced at a certain frequency in each training stage of curriculum learning. Finally, successful learning in the new environment is ensured, and the knowledge obtained in the previous environment can be effectively retained, accelerating the convergence speed, improving the sampling efficiency, and preventing the agent from falling into local optimum and causing training divergence.

[0060] The SAC (Soft Actor-Critic) reinforcement learning algorithm based on Transformer is an algorithm that combines maximum entropy reinforcement learning and Transformer neural network. Using the SAC framework, it aims to maximize the entropy of the policy while maximizing the expected reward to balance exploration and exploitation. The algorithm designs three neural networks, Actor, Critic_Q, and Critic_V, uses a double Q-network to avoid overestimation of Q values, uses a double V-network to stabilize the training process and avoid overestimation bias. The Transformer structure is used to extract the robot state sequence information, enhancing the model's reasoning ability for sequence data.

[0061] Optimized processing of lidar partitioning is a technology for preprocessing lidar data, aiming to improve the robot's perception ability of the surrounding environment. When processing lidar data, the area of the lidar is first divided into several blocks according to its dimension, and the data in each block is clustered. The traditional partitioning method usually divides the area into several equal parts on average, but this method may not meet the robot's more detailed perception requirements for the front area. Therefore, optimized processing is carried out to make the robot's perception of the front area more detailed when facing the same obstacle, thereby improving the robot's navigation and exploration ability. This optimized processing is of great significance for achieving end-to-end goal-oriented exploration.

[0062] The reward settings are refined. For the robot's autonomous exploration task, 7 specific mechanisms are designed: turning penalty, target proximity reward, obstacle proximity penalty, wandering penalty, step penalty, reaching target reward, and collision penalty. These mechanisms help the robot conduct autonomous exploration in complex and unknown environments while achieving a balance between exploration and exploitation. Through these reward and punishment mechanisms, the robot can efficiently complete tasks, avoid over-reliance on certain strategies (such as excessive exploration or wandering), and effectively improve the stability and convergence speed of training.

[0063] As Figure 1 shown, it is the overall framework diagram of the robot and reinforcement learning adopted by the present invention. The CTSAC module receives the surrounding environment information obtained from the lidar and gyroscope and outputs the robot's motion actions. The robot executes the motion actions in the environment, and at the same time, the environment gives a certain reward to the CTSAC module. As Figure 2 shown, it is the flowchart of the CTSAC algorithm, which mainly includes curriculum learning of the regular review mechanism, SAC reinforcement learning algorithm based on Transformer, lidar partition optimization processing, refined reward settings. The curriculum learning of the regular review mechanism, as the training method of the overall deep reinforcement learning, is responsible for tasks such as environment switching; the SAC reinforcement learning based on Transformer is the SAC reinforcement learning paradigm used to train the model, and the neural network therein consists of a network based on Transformer. The state of the robot is represented by the lidar data and gyroscope data after lidar partition optimization processing. At the same time, after the state information passes through the refined reward settings, the corresponding reward value r t is obtained, and the corresponding experience sequence is stored in the experience pool. The data for training the neural network is sampled from the experience pool. The implementation of the CTSAC algorithm includes the following steps:

[0064] Step 1, initialize the parameters of the SAC reinforcement learning neural network based on Transformer (Actor network, dual Critic Q network, dual Critic V network), initialize the current training level j to 0, initialize the success rate to 0, and initialize the environment container EC (Environment Container) to e (0) where e (0) is the environment numbered 0;

[0065] The SAC reinforcement learning algorithm based on Transformer includes the following content:

[0066] Soft-Actor-Critic reinforcement learning is adopted as the training method, based on the off-policy actor-critic framework of maximum entropy reinforcement learning. While maximizing the expected reward, the entropy of the policy is maximized;

[0067]

[0068] Among them, π * is the target policy; is to take the expectation of the function in the brackets under the probability distribution of the trajectory τ generated by the policy π; r(s t , a t ) is the reward function, evaluating the value of the action a t in the state s t ; γ is the discount factor, weighing short-term and long-term rewards; t is the time step, is the entropy of the policy π, representing the randomness of the policy; α 1 is the temperature coefficient, controlling the influence of randomness on the optimization objective; π(·|s t ) is the policy π in the state s t .

[0069] The double Q-network and double V-network are adopted to avoid overestimation and improve the stability of the training process; the Transformer is combined with the fully connected layer and used as the neural network structure in the deep reinforcement learning model to improve the time series inference ability of deep reinforcement learning:

[0070] As Figure 4a , 4b shown, the Actor network is divided into two parts: the policy selection network and the policy improvement network, and these two parts of the network share network parameters. In the policy selection stage, the network input is all the states of each step of the robot in the environment. After being dimensionally elevated by the fully connected layer, the input is passed into the Transformer layer to generate the correlation information between each step and merged with the current step information. Subsequently, the mean and variance of the action are generated through two layers of fully connected layers respectively. In the policy improvement stage, the network receives the experience sequence from the experience replay pool. The sequence data processed by the Transformer layer does not need to be averaged and dimensionally reduced, but is directly concatenated with the original data after dimensional elevation. This design aims to generate action data with sequence dimensions for the Q-network to learn. The objective function J π (φ) of the Actor network is shown as follows:

[0071]

[0072] Among them, J π (φ) is the state objective function of the policy π φ , and φ is the parameter of the policy network; α2 is a hyperparameter for controlling exploration randomness; π φ (a t s t ) represents the probability of taking action a φ in state s t ; Q t (s θ ,a t ) is the action objective function for taking action a t in state s t ; t The function in the brackets is expected under the joint distribution of s sampled from the experience replay pool and the Gaussian noise ε t sampled from . t As shown in

[0073] such as Figure 5 shown, the double V network: one V network for estimating the current state value V(s t ), and one V-target network for calculating the target value V(s t+1 ). The parameters of V-target are obtained by soft-updating the V network. The structure of the double V network is similar to the Actor network for policy selection. The input of the double V network is s. However, the double V network only outputs a one-dimensional value V(s) at the end. The objective function J V (ψ) of the V network is as follows:

[0074]

[0075] where J v (ψ) is the objective function of the value function V ψ (s t ); represents the expectation of the function in the brackets under the probability distribution of s sampled from the experience replay pool t ; ψ is the parameter of the value function V, and V ψ (s t ) is the predicted value of the current value function V ψ for state s t ; represents the expectation of the function in the brackets under the probability distribution of a φ sampled from the policy π t , and α 3 is a hyperparameter for controlling exploration randomness.

[0076] As Figure 6As shown, the double Q network: The structures of the two Q networks are exactly the same, but their parameters are independent and are updated independently. Finally, when selecting the Q value, the minimum value of the outputs of these two networks is taken as the final Q value. The structure of the double Q network is very similar to that of the double V network, and the main difference lies in the input data: the input of the Q network is the superposition of s and a, and the other network structures are the same as those of the V network. The objective function J Q of (θ) is shown as follows:

[0077]

[0078]

[0079]

[0080] Among them, J Q (θ) is the objective function of the Q-value function, which is used to minimize the difference between the predicted value and the target value; represents taking the expectation of the function in the brackets under the probability distribution of sampling (s , a t ) in the experience replay pool t ; is the target value of the Q value, which combines the immediate reward and the value function; γ is the discount factor, which weighs the importance of short-term and long-term rewards; is the value function, which estimates the expected cumulative reward of the next state; α 0 is the initial temperature coefficient, which sets the initial value of the entropy regularization weight; τ is the adjustment rate, which controls the decay speed of α, and n is the number of training steps.

[0081] Step 2, when the success rate is less than the set value, initialize the training environment E, and sample the training environment E from the environment container EC according to P (i,j) . Load the model in the training environment E through spawn_model in the Gazebo software, and randomly generate the current position (x robot , y robot ) of the robot, the target position (x target , y target ) and the position of the i-th obstacle (x i obstacle , y i obstacle ).

[0082] Step 3, the robot obtains the surrounding 360° environmental information at the current moment through the lidar, and then performs lidar partition optimization processing to obtain d lidar -dimensional lidar information;

[0083] As Figure 3 shown, the lidar partition optimization processing includes the following contents:

[0084] The area of the lidar is divided into several blocks according to its dimension, and then the lidar data in each block is clustered. For a robot moving forward, the perception requirement in the front is more important. Therefore, the way of dividing the lidar is optimized so that the partition in the front of the robot is denser and the partition in the back is sparser. Under the same lidar dimension, in the face of the same obstacle, through the optimized division method, the robot can perceive the front area more carefully. The angular resolution expression of the block is as follows:

[0085]

[0086] where Δθ m represents the angular resolution (or angular step) of the m-th block; the smaller the angular resolution, the denser the block, and the higher the corresponding perception accuracy; the block number m represents the position of the current block in the lidar coverage; d represents the total number of blocks in the lidar coverage.

[0087] Step 4, the robot estimates its pose through the IMU (Inertial Measurement Unit) to obtain the robot's position (x robot , y robot ), speed and angular velocity (v robot , ω robot ) at the current moment;

[0088] Step 5, according to the robot's current position (x robot , y robot ) and the target position (x target , y target ), use the Euclidean distance calculation formula and the arctangent formula to calculate the relative distance and angle (d, θ) of the target.

[0089] Step 6, splice the lidar n-dimensional data, the robot's speed and angular velocity (v robot , ω robot ), and the relative distance and angle (d, θ) of the target into a state vector, and normalize it through the Min-Max normalization method to form the robot state S t .

[0090] Step 7, input the robot state S t into the Actor network in the Transformer-based SAC algorithm to obtain the robot's actions (v, ω). After the robot obtains the actions, it moves in the environment.

[0091] Step 8: According to the movement of the robot in the environment, refine the reward settings for the robot's movement, and at the same time store the continuous experience data in the experience pool. The experience data consists of a continuous five-tuple experience sequence {s t , a t , r t , s t+1 , c t};

[0092] The refined reward settings include the following:

[0093] The refined reward settings consist of 7 parts, namely the turning penalty r 1 (a r ), the target proximity reward r 2 (d t ), the obstacle approach penalty r 3 (mind t ), the wandering penalty r p , the step penalty -λ 7 , the target arrival reward λ 1 , and the collision penalty -λ 2 . The expression is as follows:

[0094]

[0095] where R represents the refined reward settings; the specific description is as follows:

[0096] Step 81, the turning penalty r 1 (a r ): To reduce the frequent turning of the robot, a penalty mechanism is imposed on its angular velocity a r .

[0097]

[0098] Step 82, the target proximity reward r 2 (d t ): To encourage the robot to move towards the target position, when the distance d t between the robot and the obstacle is less than 10 meters, a certain reward is given.

[0099]

[0100] Step 83, the obstacle approach penalty r 3 (mind t ): To prevent the distance between the robot and the obstacle from being too close, the present invention converts the lidar information into the minimum distance value mind t to the obstacle. When the minimum distance value is less than 1 meter, a certain penalty is given.

[0101]

[0102] Step 84, Wandering Penalty r p : To prevent the robot from falling into a local optimal state (preventing it from spinning or wandering in a certain position) and to encourage it to conduct a more extensive exploration, the present invention designs a wandering penalty mechanism. This mechanism calculates the number of historical positions (x i , y i ) stored whose Manhattan distance from the current position (x, y) is less than δ as the penalty.

[0103] d m (x, y, x i , y i ) = |x - x i | + |y - y i | (12)

[0104]

[0105] where d m is the relative distance between the current position (x, y) and the historical position (x i , y i ) stored in this set, and 1(·) is an indicator function that returns 1 if the condition is true and 0 otherwise.

[0106] Step 85, Step Penalty -λ 7 : To prompt the robot to reach the target point as soon as possible and reduce unnecessary detours, the present invention adds a constant step penalty in each step to encourage the robot to move forward along a more direct path, thereby effectively reducing ineffective exploration behaviors.

[0107] Step 86, Target Reached Reward λ 1 : When the robot successfully reaches the target, a positive reward is given and the current training round is terminated.

[0108] Step 87, Collision Penalty -λ 2 : When the robot collides with an obstacle, a negative reward is given and the current training round is terminated.

[0109] Step 9, Calculate the success rate of the robot;

[0110] When the success rate is less than or equal to β, repeat Steps 3 to 8;

[0111] When the success rate is greater than β, add a new environment e (n) to the environment container EC, while incrementing the training level j by 1, clearing the success rate, and emptying the experience pool;

[0112] If the current training level reaches the training upper limit, stop all training, save the neural network model, and end the training process.

[0113] Step 10, when the number of experience sequences in the experience pool exceeds the set value S min start the training of SAC reinforcement learning; during this process, conduct the training of SAC, optimize the parameters of the Actor network, the dual Q network, and the dual V network, and at the same time run Step 9 to obtain experiences and store the experience sequences in the experience pool.

[0114] The pseudocode of the CTSAC algorithm is shown in Table 1.

[0115] Table 1 CTSAC Pseudocode

[0116]

[0117]

[0118] In Table 1, success_rate represents the success rate, and success_history[k] represents the flag indicating whether the k-th experiment is successful. 1 means success and 0 means failure. As Figure 7 shown, the training maps of different levels in the curriculum learning of the regular review mechanism, the first row is the training map, and the second row is the test map.

Claims

1. A robot autonomous exploration and navigation method in an unknown environment based on CTSAC, characterized in that: The CTSAC module receives the surrounding environment information obtained from the laser radar and gyroscope, and outputs the robot's motion. The robot performs the motion in the environment, and the environment gives the CTSAC module a certain reward. The CTSAC module executes the CTSAC algorithm, which mainly includes course learning with a regular review mechanism, a Transformer-based SAC reinforcement learning algorithm, laser radar partition optimization processing, and detailed reward settings. The course learning with regular review mechanism is the overall deep reinforcement learning training method, responsible for the task of switching the environment; Transformer-based SAC reinforcement learning is a SAC reinforcement learning paradigm used to train models, where the neural network consists of a Transformer-based network; The robot's state is represented by the lidar data and gyroscope data after lidar partition optimization. At the same time, the state information is refined through reward settings to obtain the corresponding reward value r t , and store the corresponding experience sequence into the experience pool, and the data sampled from the experience pool is used to train the neural network.

2. According to claim 1, the robot's autonomous exploration and navigation method in an unknown environment based on CTSAC is characterized in that: The implementation steps of the CTSAC algorithm are as follows: S1, initialize the parameters of the Transformer-based SAC reinforcement learning neural network, initialize the current training level j, success rate β, and environment container EC; The environment E is sampled from the environment container EC, and the model in the environment E is loaded in Gazebo through spawn_model to randomly generate the robot's current position, target position, and the position of the i-th obstacle; S3, the robot obtains the 360° environment information around it at the current moment through the laser radar, and then performs laser radar partition optimization processing to obtain d lidar dimensional lidar information; S4, the robot estimates the posture through the inertial measurement unit and obtains the current robot position (x robot ,y robot ), velocity and angular velocity (v robot ,ω robot ); S5, according to the current position of the robot (x robot ,y robot ) and the target position (x target ,y target ), use the Euclidean distance calculation formula and the inverse tangent formula to calculate the target relative distance and angle (d, θ); S6, the n-dimensional data of the laser radar, the robot speed and angular velocity (v robot ,ω robot ), the target relative distance and angle (d, θ) are concatenated into a state vector, and normalized using the Min-Max normalization method to form the robot state S t ; S7, change the robot state to S t The Actor network in the Transformer-based SAC algorithm is passed in to obtain the robot's action (v, ω). After the robot obtains the action, it moves in the environment. S8, according to the movement of the robot in the environment, the robot's movement is refined and reward settings are set. At the same time, the continuous experience data is stored in the experience pool. The experience data consists of a continuous five-tuple experience sequence {s t ,a t ,r t ,s t+1 ,c t }composition; S9, calculate the success rate of the robot: when the success rate is less than or equal to β, repeat steps S3 to S8; when the success rate is greater than β, add a new environment e to the environment container EC. (n) , at the same time, the training level is j+1, the success rate is reset to zero, and the experience pool is cleared; When the training level reaches the upper limit of training, all training is stopped, the neural network model is saved, and the training process ends; S10, when the number of experience sequences in the experience pool exceeds the set value S min , start the SAC reinforcement learning training; optimize the parameters of the Actor network, the dual Q network, and the dual V network, and simultaneously run step S9 to acquire experience, and store the experience sequence in the experience pool.

3. According to claim 2, the robot's autonomous exploration and navigation method in an unknown environment based on CTSAC is characterized in that: In step S1, a double Q network and a double V network are used to combine the Transformer with the fully connected layer as the neural network structure in the deep reinforcement learning model; The Actor network includes a strategy selection network and a strategy improvement network, which share network parameters. In the strategy selection stage, the network input is all the states of the robot at each step in the environment. The input is passed to the Transformer layer after dimension upgrading through the fully connected layer to generate the association information between each step and merge it with the current step information. Subsequently, the mean and variance of the action are generated respectively through two fully connected layers. In the strategy improvement stage, the network receives the experience sequence from the experience replay pool. The sequence data processed by the Transformer layer does not need to be averaged and dimensionally reduced, but is directly spliced ​​with the original data after dimension upgrading.

4. According to claim 2, the robot's autonomous exploration and navigation method in an unknown environment based on CTSAC is characterized in that: In step S3, the laser radar partition optimization process is as follows: The laser radar area is divided into several blocks according to its dimension, and then the laser radar data in each block is clustered; the angular resolution expression of the block is as follows: Among them, Δθ m represents the angular resolution of the mth block; the block number m represents the position of the current block in the laser radar coverage; d represents the total number of blocks in the laser radar coverage.

5. According to claim 2, the robot's autonomous exploration and navigation method in an unknown environment based on CTSAC is characterized in that: The refined reward setting R includes the turn penalty r1(a r ), target proximity reward r2(d t ), obstacle approach penalty r3(mind t ), wandering penalty p , step penalty -λ7, target reward λ1, collision penalty -λ2, the expressions are as follows: Turn penalty r1(a r ) is as follows: Among them, a r is the angular velocity; The target proximity reward r2(d t ) is as follows: When the distance d between the robot and the obstacle t If it is less than 10 meters, a certain reward will be given; Obstacle approach penalty r3(mind t ) is as follows: mind t The minimum distance value to the obstacle converted from the laser radar information; when the minimum distance value is less than 1 meter, a certain penalty will be given; Wandering Penalty p By statistically calculating the current position (x, y) and the stored historical position (x i ,y i ) whose Manhattan distance is less than δ, as a penalty; the expression is as follows: d m (x,y,x i ,and i )=|xx i |+|yy i | Among them, d m () is the current position (x, y) and the stored historical position (x i ,y i ) is the relative distance of the condition; 1(·) is an indicator function that returns 1 if the condition is true, otherwise it returns 0; Step penalty - λ7: The purpose is to encourage the robot to reach the target point as quickly as possible and reduce unnecessary detours. A constant step penalty is added to each step to encourage the robot to move forward in a more direct path. Reaching the target reward λ1: When the robot successfully reaches the target, a positive reward is given and the current training round is terminated; Collision penalty - λ2: When the robot collides with an obstacle, a negative reward will be given and the current training round will be terminated.

Images