Mobile robot path planning method based on improved long short-term memory network

Abstract

The invention discloses a mobile robot path planning method based on an improved long short-term memory network. The method comprises the following steps: step 1, carrying out partial observable Markov decision process modeling on a mobile robot in a time state; 2, adding a long-short-term memory network optimization reinforcement learning path planning algorithm and a preliminary optimization algorithm to complete basic navigation path planning, and 3, providing probability compensation for the long-short-term memory network; 4, applying a self-adaptive compensation function aiming at the strategy function so that the model can more effectively utilize the existing information and improve the decision and navigation capability of the robot in a complex environment in a changeable environment, and 5, training the modified SAC algorithm model, and deploying the finally trained model to the mobile robot, so as to improve the navigation capability of the robot in a complex environment. And path planning can be better completed. According to the method, the long-short-term memory network is introduced into the reinforcement learning algorithm, so that the performance of path planning is effectively improved.

Description

Technical Field

[0001] The invention belongs to the field of mobile robots and computer processing, and in particular relates to a mobile robot path planning method based on an improved long short-term memory network. Background Art

[0002] With the rapid development of artificial intelligence technology, reinforcement learning (RL) has been increasingly used in complex tasks, especially in the fields of autonomous driving, robot navigation, and intelligent control. Reinforcement learning continuously learns the optimal strategy through interaction with the environment to achieve the predetermined goal. However, in practical applications, traditional reinforcement learning algorithms face many challenges, including high-dimensional state space, long-term dependency problems, and training stability.

[0003] Long Short-Term Memory (LSTM) is a special type of recurrent neural network (RNN) that has attracted widespread attention for its ability to effectively capture and process long-term dependencies in sequence data. By introducing memory units and gate mechanisms, LSTM solves the gradient vanishing and gradient exploding problems that are prone to occur in traditional RNNs when processing long sequence data, thereby being able to better model dependencies in time series data. In reinforcement learning, the training of value networks and policy networks often relies on a large amount of environmental interaction data, which contains not only immediate feedback but also estimates of long-term rewards. Therefore, the introduction of LSTM can help reinforcement learning algorithms better capture long-term dependencies in time series and improve the stability and efficiency of strategies. Summary of the invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a mobile robot path planning method based on an improved long short-term memory network, aiming to solve the challenges faced by traditional reinforcement learning algorithms in complex environments. By introducing a probability compensation mechanism in the SAC (Soft Actor-Critic) algorithm, the method of the present invention can effectively enhance the robustness and stability of the value network. Specifically, the probability compensation mechanism compensates for the update of the value network by adjusting the probability distribution of the strategy, thereby improving the accuracy and stability of the value estimation. In addition, the method of the present invention further enhances the policy network's ability to process time series data by introducing an LSTM network.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A mobile robot path planning method based on an improved long short-term memory network comprises the following steps:

[0007] Step 1: Model the partially observable Markov decision process of the mobile robot under time state;

[0008] Step 2: Add the long short-term memory network to optimize the reinforcement learning path planning algorithm, preliminarily optimize the algorithm, and complete the basic navigation path planning;

[0009] Step 3: Propose probability compensation for long short-term memory networks;

[0010] Step 4: Apply an adaptive compensation function to the policy function so that the model can more effectively use existing information in a changing environment and improve the robot's decision-making and navigation capabilities in complex environments;

[0011] Step 5: Train the modified SAC algorithm model and deploy the final trained model to the mobile robot so that it can better complete path planning.

[0012] Furthermore, in step 1, the partially observable Markov decision process POMDP describes the state of the mobile robot when modeling it, and defines a complex interactive system, including state space, action space, observation space, and corresponding transfer, observation and reward functions. The model is described by a six-tuple (S, O, A, T, O, R), as follows:

[0013] 1.1) State space S = {s1, s2, …, s n}: This space contains all possible states that the system can be in. For a mobile robot, each state s t ={o t ,h t}Including the observable part o t and the unobservable part h t ,The observable part is the robot’s current sensor reading, and the unobservable part includes the speed of dynamic obstacles and external environmental change factors;

[0014] 1.2) Observation space O = {o1,o2,…,o n}: This space describes all possible observation data obtained by the robot. In mobile robot modeling, the observation space is composed of It consists of three parts: represents the real-time local map generated by the lidar at time t, including the grid map information centered on the robot; Represents the obstacle information obtained through the point cloud target detection algorithm and its coordinates under the global map; It includes the robot's posture information such as speed and acceleration; mobile robot posture information where d t represents the distance between the robot and the obstacle at the current time t, at Indicates the angle between the current robot and the obstacle;

[0015]

[0016] Action space A = {α1, α2, …, α n}: defines the set of all possible actions that the mobile robot can take. Since the mobile robot adopts differential drive, its speed command α in the action space must be satisfied. t = {v t ,ω t}, where v t represents the linear velocity of the mobile robot at time t, ω t represents the angular velocity of the mobile robot at time t. In order to limit the direction and field of view of the robot, v is set t The range is [0,0.9] m / s, ω t The range is [-0.9, 0.9] rad / s;

[0017] State transition function T = T(s′|s,a): T defines the probability distribution of the system transitioning to the next state s′ given the current state s and the action a taken;

[0018] Observation function Ω = Ω(z|s,a): defines the probability distribution of the observed information z given the current state s and the action a taken;

[0019] Reward function R=R(s,a,z): defines the immediate reward or cost obtained by the mobile robot when it is in a specific state s, takes action a and based on observation information z. The design of the reward function is directly related to the efficiency of the learning process, the training speed of the model and the convergence of the final strategy. In the path planning task of the mobile robot, designing an effective reward function is particularly critical because it needs to meet multiple conditions at the same time: ensuring the safety of the robot, the timeliness of the task, and avoiding the strategy from falling into a local optimal solution.

[0020] Furthermore, in step 2, the reinforcement learning algorithm is based on the actor-critic SAC algorithm, and the loss function and value network of the SAC algorithm are defined as follows:

[0021] The network structures of Actor and Critic both start from the first state s of the current sequence t Start initialization and set s t After inputting the Actor network, output action a t and get rewarded t , then transfer to the next state s t+1 ;

[0022] For the Critic network, the goal of the Critic network is to minimize the mean square error between the actual Q value and the target Q value, and to convert the s output by the Actor network into t+1 With the initial state s t Together as input, the current Q value is calculated as follows:

[0023] y t =r t +γ(minQ φ (s,a)-αlogπ θ (a|s)

[0024] Where γ represents the attenuation coefficient, r t represents the return, α is the entropy regularization coefficient, π θ (a|s) represents the policy network, Q φ (s,a) represents the estimated state;

[0025] Therefore, the loss function of Critic is defined as:

[0026]

[0027] The value network is defined as:

[0028]

[0029] For the loss function of the Actor network, the goal of the Actor network is to maximize the expected reward and entropy. The loss function is defined as:

[0030]

[0031] After completing the definition of the loss function and value network, the SAC algorithm can be trained so that the mobile robot can complete the basic path planning steps.

[0032] In step 2, a long short-term memory network is added to the SAC algorithm. Considering that the input state sequence and output information are both time series, the state action of the Q value network is optimized first, and the traditional fully connected network is replaced by an LSTM network to process time series data. The process is as follows:

[0033] 2.1) For Actor network: LSTM is used to process input state sequence and output action;

[0034] First, given a state sequence s = {s1, s2, ..., s t}, the output of the LSTM layer is the hidden state sequence h = {h1,h2,…,h t}, and finally output action a={a1,a2,…,a t};

[0035] h t ,c t =LSTM(s,h t -1,c t -1)

[0036] a t =FC(h t )

[0037] Among them, h t and c t They are the hidden state and cell state of LSTM, respectively, and FC represents the fully connected layer;

[0038] 2) For the Critic network: LSTM is used to process the state-action sequence and output the Q value;

[0039] Given a state-action sequence (s,a) = {(s1,a1),(s2,a2),…,(s t ,a t )}, the output of the LSTM layer is the hidden state sequence h={h1,h2,…,h t}, and output the Q value Q = {Q1, Q2, ..., Q t};

[0040] h t ,c t =LSTM((s,a),h t -1,c t -1)

[0041] Q t =FC(h t )

[0042] During the training process, it is necessary to process sequence data. Each training sample is expanded into a sequence containing multiple time steps. When updating the network parameters, ensure that the state of the LSTM can be correctly transferred;

[0043] For the loss function: similar to the traditional SAC, it is optimized according to the characteristics of the sequence data;

[0044] The goal of the Critic network is still to minimize the mean square error between the actual Q value and the target Q value, but now the Q value is generated by the LSTM network. Assume that the hidden state of the LSTM is h and the sequence length is T.

[0045] y t =r t +γ(minQ φ (s t+1 ,a t+1 )-αlogπ θ (a t+1∣s t+1 ))

[0046] The loss function of Critic becomes:

[0047]

[0048] The loss function of Actor becomes:

[0049]

[0050] The original loss function is for a single-step state-action pair, but now it is used to calculate the average loss for the entire time series. At the same time, the calculation of the Q value depends on the hidden state h of the LSTM. t .

[0051] Furthermore, in step 3, the probability compensation mechanism is explained as follows:

[0052] In reinforcement learning, the value network is used to estimate the value of a state, that is, the cumulative reward that the state can obtain in the future. In order to improve the accuracy and stability of value estimation, a probability compensation mechanism is introduced in the soft actor-critic SAC algorithm. This mechanism compensates for the update of the value network by adjusting the probability distribution of the strategy, thereby enhancing its robustness and stability.

[0053] The probability compensation mechanism introduces a compensation term related to the strategy probability into the objective function of the value network to adjust the impact of the strategy probability on the value estimation, thereby improving its accuracy and stability. According to the strategy network π in the SAC algorithm θ , the compensation term is defined as follows

[0054] C(s,a)=βlogπ θ (a|s)

[0055] Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

[0056] In step 4, the probability compensation mechanism is substituted into the algorithm as follows:

[0057] In the SAC algorithm, the value network V is known ψ The goal of (s) is to estimate the value of state s. SAC uses a dual Q-value network to reduce the value estimation bias. Based on the previous formula, the design compensation term C(s,a) is further introduced. The objective function of the value network can be modified as follows:

[0058]

[0059] Substituting the compensation term C(s,a) and solving it, we finally get:

[0060]

[0061] Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

[0062] In step 5, the steps of training the model are as follows:

[0063] 5.1) State modeling, initializing the environment and state space, and setting the hyperparameters of the SAC algorithm;

[0064] 5.2) Replace the state network and observation network with long short-term memory network;

[0065] 5.3) Initialize the policy network and Q value network, as well as the target Q value network, and initialize the experience replay buffer. In the training loop, sample actions based on the current policy network, execute these actions in the environment, record the next state, reward and termination flag, and store these experiences in the experience replay buffer;

[0066] 5.4) Randomly sample a batch of experience from the buffer, calculate the Q-value target using the target Q-value network, and use these target values ​​to update the Q-value network;

[0067] 5.5) Update the policy network based on the current Q-value network and perform a soft update on the target Q-value network. During the entire training process, regularly evaluate the performance of the policy in the evaluation environment and record the results.

[0068] 5.6) Save the parameters of the trained policy network and Q-value network.

[0069] Preferably, in step 5, the process of deploying the model to the mobile robot to complete path planning is: the trained model is packaged and encapsulated in a module device, and configured in the mobile robot. When driving the robot, it is first necessary to access the device for path planning. The device uses the output result to drive the mobile robot, and the final path planning of the mobile robot can be completed.

[0070] The beneficial effects of the present invention are mainly manifested in: introducing a long short-term memory network into a reinforcement learning algorithm, effectively improving the performance of path planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is a partially observable Markov decision process modeling diagram of the present invention.

[0072] Figure 2 It is a framework diagram of the SAC algorithm of the present invention.

[0073] Figure 3 This is the framework diagram of the LSTM algorithm of the present invention.

[0074] Figure 4It is a schematic diagram of the loss updating principle of the probability compensation mechanism proposed by the present invention.

[0075] Figure 5 This is a flow chart of a mobile robot path planning method based on an improved long short-term memory network. DETAILED DESCRIPTION

[0076] The present invention will be further described below in conjunction with the accompanying drawings.

[0077] Reference Figure 1 to Figure 5 ,A mobile robot path planning method based on an improved long short-term memory network comprises the following steps;

[0078] Step 1: Model the partially observable Markov decision process of the mobile robot under time state;

[0079] Reference Figure 1 , firstly, by describing the state of the mobile robot modeled by the partially observable Markov decision process POMDP, a complex interactive system is defined, including the state space, action space, observation space, and the corresponding transfer, observation, and reward functions. The model can be described by a six-tuple (S, O, A, T, O, R), as follows:

[0080] 1.1) State space S = {s1, s2, …, s n}: This space contains all possible states that the system can be in. For a mobile robot, each state s t ={o t ,h t}Including the observable part o t and the unobservable part h t The observable part may be the robot’s current sensor readings, and the unobservable part may include factors such as the speed of dynamic obstacles and changes in the external environment.

[0081] 1.2) Observation space O = {o1,o2,…,o n}: This space describes all possible observation data that the robot can obtain. In the mobile robot modeling of the present invention, the observation space is composed of It consists of three parts: represents the real-time local map generated by the lidar at time t, including the grid map information centered on the robot; Represents the obstacle information obtained through the point cloud target detection algorithm and its coordinates under the global map; It includes the robot's posture information such as speed and acceleration. Specifically, the mobile robot posture information where d t represents the distance between the robot and the obstacle at the current time t, a tIndicates the angle between the current robot and the obstacle.

[0082]

[0083] 1.3) Action space A = {α1, α2, …, α n}: defines the set of all possible actions that the mobile robot can take. Since the mobile robot adopts differential drive, it needs to satisfy its speed command α in the action space. t = {v t ,ω t}, where v t represents the linear velocity of the mobile robot at time t, ω t represents the angular velocity of the mobile robot at time t. In order to limit the direction and field of view of the robot, set v t The range is [0,0.9] m / s, ω t The range is [-0.9,0.9]rad / s.

[0084] 1.4) State transition function T = T(s′|s,a): T defines the probability distribution of the system transitioning to the next state s′ given the current state s and the action a taken.

[0085] 1.5) Observation function Ω = Ω(z|s,a): defines the probability distribution of the observed information z given the current state s and the action a taken.

[0086] 1.6) Reward function R = R (s, a, z): defines the immediate reward or cost obtained by the mobile robot in a specific state s, after taking action a and based on observation information z. The design of the reward function is directly related to the efficiency of the learning process, the training speed of the model, and the convergence of the final strategy. In the path planning task of the mobile robot, designing an effective reward function is particularly critical because it needs to meet multiple conditions at the same time: ensuring the safety of the robot, the timeliness of the task, and avoiding the strategy from falling into the local optimal solution.

[0087] Step 2: Add the long short-term memory network to optimize the reinforcement learning path planning algorithm, preliminarily optimize the algorithm, and complete the basic navigation path planning;

[0088] Reference Figure 2 , further define the SAC reinforcement learning algorithm framework based on this six-tuple (S, O, A, T, O, R):

[0089] The network structures of Actor and Critic both start from the first state s of the current sequence t Start initialization. t After inputting the Actor network, output action a tand get rewarded t , then transfer to the next state s t+1 .

[0090] For the Critic network, the goal of the Critic network is to minimize the mean square error between the actual Q value and the target Q value, and to convert the output of the Actor network into t+1 With the initial state s t Together as input, the current Q value is calculated as follows:

[0091] y t =r t +γ(minQ φ (s,a)-αlogπ θ (a|s)

[0092] Where γ represents the attenuation coefficient, r t represents the return, α is the entropy regularization coefficient, π θ (a|s) represents the policy network, Q φ (s,a) represents the estimated state.

[0093] Therefore, the loss function of Critic is defined as:

[0094]

[0095] The value network is defined as:

[0096]

[0097] For the loss function of the Actor network, the goal of the Actor network is to maximize the expected reward and entropy. The loss function is defined as:

[0098]

[0099] Reference Figure 3 , we choose to use the long short-term memory network LSTM to sequence the SAC algorithm.

[0100] Considering that both the input state sequence and the output information are time-series, the present invention first optimizes the state action of the Q-value network and replaces the traditional fully connected network with an LSTM network to process time series data.

[0101] 2.1) For Actor network: LSTM is used to process input state sequence and output action.

[0102] First, given a state sequence s = {s1, s2, ..., s t}, the output of the LSTM layer is the hidden state sequence h = {h1,h2,…,h t}, and finally output action a={a1,a2,…,a t}.

[0103] h t ,c t =LSTM(s,h t -1,c t -1)

[0104] a t =FC(h t )

[0105] Among them, h t and c t They are the hidden state and cell state of LSTM respectively, and FC represents the fully connected layer

[0106] 2.2) For the Critic network: LSTM is used to process the state-action sequence and output the Q value.

[0107] Given a state-action sequence (s,a) = {(s1,a1),(s2,a2),…,(s t ,a t )}, the output of the LSTM layer is the hidden state sequence h={h1,h2,…,h t}, and output the Q value Q = {Q1, Q2, ..., Q t}.

[0108] h t ,c t =LSTM((s,a),h t -1,c t -1)

[0109] Q t =FC(h t )

[0110] During the training process, it is necessary to process sequence data. Each training sample is expanded into a sequence containing multiple time steps. When updating the network parameters, ensure that the state of the LSTM can be correctly transferred.

[0111] For the loss function: Similar to the traditional SAC, it is optimized according to the characteristics of the sequence data.

[0112] The goal of the Critic network is still to minimize the mean square error between the actual Q value and the target Q value, but now the Q value is generated by the LSTM network. Assume that the hidden state of the LSTM is h and the sequence length is T

[0113] y t =r t +γ(minQ φ (st+1 ,a t+1 )-αlogπ θ (a t+1 ∣s t+1 ))

[0114] The loss function of Critic becomes:

[0115]

[0116] The loss function of Actor becomes:

[0117]

[0118] The original loss function is for a single-step state-action pair, but now it is used to calculate the average loss for the entire time series. At the same time, the calculation of the Q value depends on the hidden state h of the LSTM t .

[0119] Step 3: Propose probability compensation for long short-term memory networks;

[0120] Reference Figure 4 , the probability compensation mechanism introduces a compensation term related to the strategy probability into the objective function of the value network to adjust the impact of the strategy probability on the value estimation, thereby improving its accuracy and stability. According to the strategy network π in the SAC algorithm θ , the present invention defines the compensation term as follows

[0121] C(s,a)=βlogπ θ (a|s)

[0122] Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

[0123] Step 4: Apply an adaptive compensation function to the policy function so that the model can more effectively use existing information in a changing environment and improve the robot's decision-making and navigation capabilities in complex environments;

[0124] In the SAC algorithm, the value network V is known ψ The goal of (s) is to estimate the value of state s. SAC uses a dual Q value network to reduce the value estimation deviation. Based on the previous formula, the present invention further introduces the design compensation term C(s,a). The objective function of the value network can be modified as follows:

[0125]

[0126] Substituting the compensation term C(s,a) and solving it, we finally get:

[0127]

[0128] Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

[0129] Finally, each function is substituted into the computing network, the network automatically trains parameters and weights, and the returned model can be loaded and used.

[0130] Step 5: Train the modified SAC algorithm model and deploy the final trained model to the mobile robot so that it can better complete path planning.

[0131] Reference Figure 5 , the steps to establish the SAC algorithm and improve the long short-term memory network are as follows:

[0132] 5.1) State modeling, initializing the environment and state space, and setting the hyperparameters of the SAC algorithm.

[0133] 5.2) Replace the state network and observation network with long short-term memory networks.

[0134] 5.3) Initialize the policy network and Q value network, as well as the target Q value network, and initialize the experience replay buffer. In the training loop, sample actions based on the current policy network, execute these actions in the environment, record the next state, reward and termination flag, and store these experiences in the experience replay buffer.

[0135] 5.4) Randomly sample a batch of experiences from the buffer, calculate Q-value targets using the target Q-value network, and update the Q-value network with these target values.

[0136] 5.5) Update the policy network based on the current Q-value network and perform a soft update of the target Q-value network. Throughout the training process, regularly evaluate the performance of the policy in the evaluation environment and record the results.

[0137] 5.6) Save the parameters of the trained policy network and Q-value network.

[0138] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A mobile robot path planning method based on an improved long short-term memory network, characterized in that: The method comprises the following steps: Step 1: Model the partially observable Markov decision process of the mobile robot under time state; Step 2: Add the long short-term memory network to optimize the reinforcement learning path planning algorithm, preliminarily optimize the algorithm, and complete the basic navigation path planning; Step 3: Propose probability compensation for long short-term memory networks; Step 4: Apply an adaptive compensation function to the policy function so that the model can more effectively use existing information in a changing environment and improve the robot's decision-making and navigation capabilities in complex environments; Step 5: Train the modified SAC algorithm model and deploy the final trained model to the mobile robot so that it can better complete path planning.

2. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 1, characterized in that: In step 1, the partially observable Markov decision process POMDP describes the state of the mobile robot when modeling, and defines a complex interactive system, including state space, action space, observation space, and corresponding transfer, observation and reward functions. The model is described by a six-tuple (S, O, A, T, O, R), as follows: 1.1) State space S = {s1, s2, ..., s n }: This space contains all possible states that the system can be in. For a mobile robot, each state s t ={o t ,h t }Including the observable part o t and the unobservable part h t ,The observable part is the robot’s current sensor reading, and the unobservable part includes the speed of dynamic obstacles and external environmental change factors; 1.2) Observation space O = {o1, o2, ..., o n }: This space describes all possible observation data obtained by the robot. In mobile robot modeling, the observation space is composed of It consists of three parts: represents the real-time local map generated by the lidar at time t, including the grid map information centered on the robot; Represents the obstacle information obtained through the point cloud target detection algorithm and its coordinates under the global map; It includes the robot's posture information such as speed and acceleration; mobile robot posture information where d t represents the distance between the robot and the obstacle at the current time t, a t Indicates the angle between the current robot and the obstacle; Action space A = {α1, α2, ..., α n }: defines the set of all possible actions that the mobile robot can take. Since the mobile robot adopts differential drive, its speed command α in the action space must be satisfied. t = {v t ,ω t }, where v t represents the linear velocity of the mobile robot at time t, ω t represents the angular velocity of the mobile robot at time t. In order to limit the direction and field of view of the robot, v is set t The range is [0, 0.9] m / s, ω t The range is [-0.9, 0.9] rad / s; State transition function T = T(s′|s, a): T defines the probability distribution of the system transitioning to the next state s′ given the current state s and the action a taken; Observation function Ω = Ω(z|s, a): defines the probability distribution of the observed information z given the current state s and the action a taken; Reward function R=R(s, a, z): defines the immediate reward or cost obtained by the mobile robot when it is in a specific state s, takes action a and based on observation information z. The design of the reward function is directly related to the efficiency of the learning process, the training speed of the model and the convergence of the final strategy. In the path planning task of the mobile robot, designing an effective reward function is particularly critical because it needs to meet multiple conditions at the same time: ensuring the safety of the robot, the timeliness of the task, and avoiding the strategy from falling into a local optimal solution.

3. A mobile robot path planning method based on an improved long short-term memory network as described in claim 1 or 2, characterized in that: In step 2, the reinforcement learning algorithm is based on the actor-critic SAC algorithm. The loss function and value network of the SAC algorithm are defined as follows: The network structures of Actor and Critic both start from the first state s of the current sequence t Start initialization and set s t After inputting the Actor network, output action a t and get rewarded t , then transfer to the next state s t+1 ; For the Critic network, the goal of the Critic network is to minimize the mean square error between the actual Q value and the target Q value, and to convert the s output by the Actor network into t+1 With the initial state s t Together as input, the current Q value is calculated as follows: y t =r t +γ(minQ φ (s, a)-αlogπ θ (a|s)) Where γ represents the attenuation coefficient, r t represents the return, α is the entropy regularization coefficient, π θ (a|s) represents the policy network, Q φ (s, a) represents the estimated state; Therefore, the loss function of Critic is defined as: The value network is defined as: For the loss function of the Actor network, the goal of the Actor network is to maximize the expected reward and entropy. The loss function is defined as: After completing the definition of the loss function and value network, the SAC algorithm can be trained so that the mobile robot can complete the basic path planning steps.

4. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 3, characterized in that: In step 2, a long short-term memory network is added to the SAC algorithm. Considering that the input state sequence and output information are both time series, the state action of the Q value network is optimized first, and the traditional fully connected network is replaced by an LSTM network to process time series data. The process is as follows: 2.1) For Actor network: LSTM is used to process input state sequence and output action; First, given a state sequence s = [s1, s2, ..., s t }, the output of the LSTM layer is the hidden state sequence h = [h1, h2, ..., h t }, and finally output action a={a1, a2, ..., a t }; h t ,c t =LSTM(s,h t -1,c t -1) and t =FC(h t ) Among them, h t and c t They are the hidden state and cell state of LSTM, respectively, and FC represents the fully connected layer; 2.2) For the Critic network: LSTM is used to process the state-action sequence and output the Q value; Given a state-action sequence (s, a) = {(s1, a1), (s2, a2), ..., (s t , a t )}, the output of the LSTM layer is the hidden state sequence h = {h1, h2, ..., h t }, output Q value Q = {Q1, Q2, ..., Q t }; h t ,c t =LSTM((s,a),h t -1,c t -1) Q t =FC(h t ) During the training process, it is necessary to process sequence data. Each training sample is expanded into a sequence containing multiple time steps. When updating the network parameters, ensure that the state of the LSTM can be correctly transferred. For the loss function: similar to the traditional SAC, it is optimized according to the characteristics of the sequence data; The goal of the Critic network is still to minimize the mean square error between the actual Q value and the target Q value, but now the Q value is generated by the LSTM network. Assume that the hidden state of the LSTM is h and the sequence length is T. y t =r t +γ(min Q φ (s t+1 ,a t+1 )-alogπ θ (a t+1 |s t+1 )) The loss function of Critic becomes: The loss function of Actor becomes: The original loss function is for a single-step state-action pair, but now it is used to calculate the average loss for the entire time series. At the same time, the calculation of the Q value depends on the hidden state h of the LSTM. t .

5. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 1 or 2, characterized in that: In step 3, the probability compensation mechanism is explained as follows: In reinforcement learning, the value network is used to estimate the value of a state, that is, the cumulative reward that the state can obtain in the future. In order to improve the accuracy and stability of value estimation, a probability compensation mechanism is introduced in the soft actor-critic SAC algorithm. This mechanism compensates for the update of the value network by adjusting the probability distribution of the strategy, thereby enhancing its robustness and stability. The probability compensation mechanism introduces a compensation term related to the strategy probability into the objective function of the value network to adjust the impact of the strategy probability on the value estimation, thereby improving its accuracy and stability. According to the strategy network π in the SAC algorithm θ , the compensation term is defined as follows C(s,a)=βlogπ θ (a|s) Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

6. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 1 or 2, characterized in that: In step 4, the probability compensation mechanism is substituted into the algorithm as follows: In the SAC algorithm, the known value network V ψ The goal of (s) is to estimate the value of state s. SAC uses a dual Q-value network to reduce the value estimation bias. Based on the previous formula, the design compensation term C(s, a) is further introduced. The objective function of the value network can be modified as follows: Substituting the compensation term C(s,a) and solving it, we finally get: Among them, β is the weight coefficient, which is used to balance the impact of the compensation term.

7. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 1 or 2, characterized in that: In step 5, the steps of training the model are as follows: 5.1) State modeling, initializing the environment and state space, and setting the hyperparameters of the SAC algorithm; 5.2) Replace the state network and observation network with long short-term memory network; 5.3) Initialize the policy network and Q value network, as well as the target Q value network, and initialize the experience replay buffer. In the training loop, sample actions based on the current policy network, execute these actions in the environment, record the next state, reward and termination flag, and store these experiences in the experience replay buffer; 5.4) Randomly sample a batch of experience from the buffer, calculate the Q-value target using the target Q-value network, and use these target values ​​to update the Q-value network; 5.5) Update the policy network based on the current Q-value network and perform a soft update on the target Q-value network. During the entire training process, regularly evaluate the performance of the policy in the evaluation environment and record the results. 5.6) Save the parameters of the trained policy network and Q-value network.

8. A mobile robot path planning method based on an improved long short-term memory network as claimed in claim 7, characterized in that: In step 5, the process of deploying the model to the mobile robot to complete path planning is: the trained model is packaged and encapsulated in a module device, and configured in the mobile robot. When driving the robot, it is first necessary to access the device for path planning. The device uses the output result to drive the mobile robot to complete the final path planning of the mobile robot.

Images