Mobile robot autonomous navigation planning method based on brain-like pulse deep reinforcement learning

Abstract

The invention discloses a mobile robot autonomous navigation planning method based on brain-like pulse deep reinforcement learning. Firstly, a brain-like pulse neural network (SNN) is constructed as a Q function approximator, and the energy efficiency of path planning is improved by using the time sequence processing capability and biological rationality of the SNN. Secondly, designing a double-priority experience playback mechanism, dividing experience samples into a common experience pool and an elite experience pool according to a reward threshold value, adjusting the sampling proportion of the two experience pools by adopting a dynamic self-adaptive sampling strategy, and intelligently balancing the sample utilization rate in the training process according to a loss function and a training stage; in addition, the reward function design is optimized through Manhattan distance reward and a repeated access punishment mechanism, and the robot is guided to efficiently explore the environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation and mechanical control, in particular to a mobile robot path planning method based on pulse deep reinforcement learning. BACKGROUND

[0002] Mobile robot path planning, as one of the core technologies in the field of robotics, mainly studies how to autonomously calculate a safe and efficient path from the starting position to the target position in a specific environment. This technology has been widely applied in various important scenarios such as autonomous driving, unmanned aerial vehicle inspection, and industrial logistics. With the continuous evolution of artificial intelligence and computing technology, path planning methods have undergone significant development from traditional algorithms to intelligent decision-making.

[0003] Early path planning methods mainly rely on graph search and mathematical optimization techniques such as A* algorithm and Dijkstra algorithm. These methods have good results in known static environments, but are difficult to deal with dynamic changes or unknown environments. To improve the response ability of the system in dynamic scenarios, researchers have proposed real-time planning algorithms such as dynamic window method (DWA) and rapidly-exploring random tree (RRT), enabling robots to adjust their travel paths in response to environmental changes.

[0004] In recent years, with the breakthroughs in deep learning and reinforcement learning technologies, artificial intelligence-based path planning methods have shown great potential. Through interaction with the environment, robots can autonomously learn navigation strategies in complex environments, achieving end-to-end planning from perception to decision-making. In particular, in multi-robot collaborative tasks, path planning not only needs to consider the motion constraints of individual agents, but also needs to solve the collision avoidance and efficiency optimization problems in group collaboration.

[0005] The invention patent with publication number CN202111386217.7 "A mobile robot path planning method" introduces a random decision mechanism based on map initialization and cost function setting: in each iteration, the system judges to execute the traditional particle swarm update step or perform particle mutation operation according to the random number. The mutation operation generates a mutation particle by selecting a poor particle, and merges it with the original particle swarm. After fitness evaluation, some poor performing particles are eliminated, thereby enhancing population diversity. The invention patent with publication number CN113159432A "A multi-agent path planning method based on deep reinforcement learning", which is a distributed path planning method, inputs the local observation information of a single agent into a neural network, uses a graph neural network to transfer information between agents, trains the neural network to approximate the strategy function, and outputs the movement strategy. The invention patent with publication number CN112325897B "A path planning method based on heuristic deep reinforcement learning", which uses the Actor-Critic reinforcement learning algorithm for training; adopts the classical heuristic algorithm A* as the heuristic function of reinforcement learning to construct heuristic information. The above patent methods all rely on random decision, which leads to unstable convergence process, repeated trial and error, slows down the speed, and limits the training efficiency to the interaction delay. Therefore, the present invention aims to provide a new scheme to fundamentally solve the above problems and achieve a significant improvement in the convergence speed of the training network. SUMMARY

[0006] The present invention provides a mobile robot autonomous navigation planning method based on brain-like pulse deep reinforcement learning, aiming to solve the slow convergence speed problem in traditional path planning. The method constructs a brain-like pulse neural network as a Q function approximator, uses the timing processing capability and biological rationality of SNN to improve the energy efficiency of path planning; at the same time, a double experience pool is designed to store ordinary experience and elite experience respectively, and a priority experience replay mechanism based on TD error is used for the two experience pools, a dynamic adaptive sampling strategy is used to adjust the sampling proportion of the two experience pools, and the sample utilization rate is intelligently balanced according to the loss function and the training stage during the training process; finally, the Manhattan distance reward and repeated access penalty mechanism are used to optimize the reward function design, guiding the robot to efficiently explore the environment and improve the training efficiency. Through simulation experiments, the algorithm of the present invention has achieved significant results in convergence speed and path quality, enabling the mobile robot to obtain better navigation performance in complex environments.

[0007] The technical scheme adopted by the present invention is: a mobile robot autonomous navigation planning method based on brain-like pulse deep reinforcement learning, comprising the following steps:

[0008] Step 1: Build a grid environment as a simulation platform in a python environment, which includes robot position, obstacles, passable areas, and the starting point and endpoint of the robot;

[0009] Step 2: Define the robot's discrete motion space: up (0), down (1), left (2), right (3), and realize state transition;

[0010] Step 3: Construct a training network and a target network based on a brain-like spiking neural network (SNN). The two networks have the same structure but different parameters. The parameters of the training network are θ, and the parameters of the target network are θ'.

[0011] Step 4: Encode the robot's current position as a two-dimensional matrix and input it into the SNN network. Use the Q-values ​​of the four actions as the output and filter out the actions with the largest Q-value at that moment.

[0012] Step 5: The robot randomly selects an action to explore with an exploration probability ε, and selects the optimal action according to the SNN network with a probability of 1-ε. The value of ε decays during the training process.

[0013] Step 6: Optimize the learning signal by calculating the reward value and performing a weighted sum of the main reward, Manhattan distance-guided reward, and repeat visit reward;

[0014] Step 7: Establish a dual-priority experience replay mechanism and set a reward threshold r. elite The experience pool is divided into a regular experience pool and a management experience pool, and then based on (s) t ,a t ,s t+1 In the algorithm, reward r enables intelligent classification of experience samples; where s t express t Time state, a t s represents the action taken at time t. t+1 The state after taking an action is the next state, and r is the immediate reward gained from the action.

[0015] Step 8: Adopt a dynamic adaptive sampling strategy to dynamically adjust the sampling ratio of the two experience pools according to the training stage and loss function, and use different sampling strategies in the auxiliary training stage and the main training stage.

[0016] Step 9: Update sample priorities using TD error and apply the overall loss function. Complete the network weight update;

[0017] Step 10: After training is complete, save the trained model, test the model, output the final path and visualize it.

[0018] Furthermore, step 1 specifically includes the following steps:

[0019] Step 1-1: Set up a simulation environment in Python to simulate a mobile robot in a real environment;

[0020] Steps 1-2: Set a 20×20 grid area as the working area in the simulation environment. Multiple black obstacles can be set within this area. Obstacles are represented by black pixels, and passable areas are represented by white pixels. Further, step 3 specifically includes the following steps:

[0021] Step 3-1: Construct the input layer. The input layer accepts a 20×20 grid environment state matrix as input, where obstacles are encoded as 1, free areas are encoded as 0, and the robot's current position is encoded as 2. The network environment state matrix is ​​converted into a four-dimensional tensor of shape [1,1,20,20] to meet the input requirements of the SNN network.

[0022] Step 3-2: Establish the spiking neural network layer. The network layer adopts a fully connected spiking neural network architecture, including a three-layer structure. Each layer consists of a fully connected layer and a LIF neuron layer.

[0023] First layer: Input dimension 400, output 256 spiking neurons, activated by snnntorch.Leaky;

[0024] The second layer has an input dimension of 256 and an output of 128 spiking neurons, maintaining the transmission of temporal information.

[0025] The third layer has an input dimension of 128 and an output of 64 spiking neurons, completing high-level feature extraction.

[0026] Step 3-3: Establish a membrane potential memory mechanism. Each spiking neuron layer maintains its own membrane potential state (mem1, mem2, mem3). During each forward propagation, the membrane potential accumulates with the time step and affects the neuron's spiking behavior. By initializing the membrane potential to None at the beginning of each training cycle, the system automatically re-initializes the membrane potential state, ensuring the independence and stability of the temporal calculations between different training samples.

[0027] Steps 3-4: Establish an output layer using a fully connected output layer. The input is a 64-dimensional pulse signal, and the output is the Q-value estimation of four actions, corresponding to the expected reward in the four movement directions of up, down, left, and right.

[0028] Furthermore, step 4 specifically includes the following steps:

[0029] Step 4-1: Encode the robot's current position information into a 20×20 two-dimensional matrix and input it into the spiking neural network, where the obstacle position value is 1, the passable area is 0, and the robot's current position is marked as 2. Extract the spatiotemporal features of the input matrix through the spiking neural network layer of the SNN, and use the temporal dynamic characteristics of the spiking neurons to capture the spatial structure of the environment and the robot's position relationship.

[0030] Step 4-2: Employ a deep Q-learning framework and use spiking neural network parameters θ to approximate the optimal action value function Q. * (s t ,a t The network continuously optimizes its parameters by minimizing the temporal difference error between the current Q-value estimate and the target Q-value, so that Q(s) t ,a t ,θ) gradually converges to Q * (s t ,a t As shown in the following formula:

[0031] Q(s t ,a t ,θ)→Q * (s t ,a t )

[0032] Step 4-3: The network output layer generates a 4-dimensional vector, which corresponds to the Q-value prediction of the four actions: up, down, left, and right. Each Q-value represents the expected reward after taking the corresponding action in the current environment, providing a quantitative basis for the robot's action decision.

[0033] Furthermore, step 5 specifically includes the following steps:

[0034] Step 5-1: Decaying Exploration Rate Design: The exploration probability ε dynamically decays during training, with an initial value set to 0.8. To ensure sufficient exploration of the environment in the early stages of training, it gradually decays according to the following rules:

[0035] Keep ε = 0.8 constant in the early stages of training to ensure sufficient initial exploration;

[0036] In subsequent training cycles, the exploration rate is gradually reduced according to the decay coefficient, as shown in the following formula:

[0037] ε = max(ε min ,ε×γ decay )

[0038] Where γ decay With an attenuation coefficient of 0.99, ε min The minimum value set in the experiment is 0.05.

[0039] Step 5-2: Action Selection Implementation: The robot randomly selects an action from the action space A = {'up', 'down', 'left', 'right'} with probability ε. With probability 1-ε, the robot uses an SNN network for forward propagation to calculate the Q-values ​​of the four actions, and selects the action corresponding to the largest Q-value.

[0040] Furthermore, step 6 specifically includes the following steps:

[0041] Step 6-1: Calculate the main reward, which includes the base step penalty, obstacle collision penalty, and goal achievement reward;

[0042] Base step penalty: The base reward for each move is r. o =-1, encourages the robot to reach the goal via the shortest path and avoids ineffective wandering behavior;

[0043] Obstacle collision penalty: When the robot attempts to move to an obstacle or tries to go out of bounds, a penalty of r is imposed. o A penalty of -3 is applied while maintaining the same position to ensure the safety of the path;

[0044] Reward for reaching the target location; when the robot successfully reaches the target location, it is rewarded with r. o A large reward of 100 clearly indicates the success of the task completion. The main reward formula is as follows:

[0045]

[0046] Step 6-2: Set Manhattan distance guidance reward: Calculate the Manhattan distance change between the robot and the target point in real time. A positive reward is given when the distance decreases, and a negative reward is given when the distance increases. The specific calculation formula is as follows:

[0047] r d =α·(d t-1 -d t )

[0048] Where: d t-1 d represents the Manhattan distance between the robot and the target point at time t-1. t Let d represent the Manhattan distance between the robot and the target point at time t, α be the reward coefficient, and d be the distance between the robot and the target point. t-1 and d t The calculation formula is as follows:

[0049] d t-1 =|r t-1 -r g |+|c t-1 -c g |

[0050] d t =|r t -r g |+|c t -c g |

[0051] Where: r t-1 and c t-1 Let r represent the row and column coordinates of the robot at time t-1, respectively. t and ct Let r represent the row and column coordinates of the robot at time t, respectively. g and c g The row and column coordinates representing the target location;

[0052] Step 6-3: Repeated Access Penalty Mechanism: Establish a location access record dictionary and impose an penalty (r) on repeated access to the same location. p = -3 penalty, otherwise r p =0, effectively preventing the robot from circling and wandering in local areas;

[0053] Step 6-4: Calculate the reward value, which is a weighted sum of the main reward, repeat visit reward, and Manhattan distance guidance reward:

[0054] r = r o +r d +r p .

[0055] Furthermore, step 7 specifically includes the following steps:

[0056] Step 7-1: Establish a dual experience pool architecture, including a normal experience pool and an elite experience pool;

[0057] Step 7-2: State s at time t t Next, the robot selects an action a based on the current strategy. t After the action is performed, the environment returns to a new state s. t+1 And the immediate reward value r, these will all serve as a reference for subsequent learning;

[0058] Step 7-3: Based on the empirical sample (s) t ,a t ,s t+1 The instant reward value r of (r) is categorized and stored:

[0059] When the reward value r≥r elite At that time, the sample is stored in the elite experience pool;

[0060] When the reward value r < r elite At that time, the sample is stored in the general experience pool;

[0061] Where: r elite The reward threshold is set to differentiate between normal experience and elite experience;

[0062] Step 7-4: Check the capacity of the two experience pools and remove the oldest experience; if the normal experience pool or elite experience pool reaches its capacity limit, remove the oldest experience.

[0063] Step 7-5: The experience samples from the two experience pools are fed into the network for training. After each training session, the sample priority is updated according to the new TD error to ensure that high-quality experience has a higher probability of being reused and to accelerate the learning process.

[0064] Furthermore, step 8 specifically includes the following steps:

[0065] Step 8-1: Establish a two-stage framework of auxiliary training stage and main training stage: the auxiliary training stage is used for the early stage of network training, focusing on basic skill learning, and the main training stage is used for the later stage of network policy optimization and convergence.

[0066] Step 8-2: The sampling ratio is dynamically calculated based on the training stage and the loss function;

[0067] Step 8-2-1: During the assisted training phase, the sampling ratio is dynamically calculated based on the loss function, as shown in the following formula:

[0068]

[0069] Where p0 is the proportion sampled from the normal experience pool; p1 is the proportion sampled from the elite experience pool, and the initial values ​​of p0 and p1 are both 0.5; ε is the exploration probability; l0 is the average loss of the normal experience pool; l1 is the average loss of the elite experience pool. The Huber loss is used to calculate the average loss of the experience pools, and the formula is as follows:

[0070]

[0071] In the formula, x is the TD error; when |x|<1: squared loss is used, the gradient is small, and the update is stable; when |x|≥1: linear loss is used to avoid gradient explosion.

[0072] During subsequent training, p0 will be updated based on the training loss from the two experience pools, and p1 will also be updated accordingly:

[0073] p1 = 1 - p0

[0074] Step 8-2-2: In the main training phase, focus on policy optimization and convergence, with a fixed sampling ratio: p0 = p1 = 0.5;

[0075] To ensure that both experience pools have sufficient sampling opportunities and to avoid sampling bias in extreme cases, a sampling boundary is set: p0∈[0.3,0.8].

[0076] Furthermore, step 9 specifically includes the following steps:

[0077] Step 9-1: Calculate the priority, priority sampling probability, and importance sampling weight of each sample in the two experience pools;

[0078] Step 9-1-1: Calculate sample priority. Each experience pool uses a SumTree structure for priority sampling. The sample priority is calculated based on the temporal difference error (TD error).

[0079] p i =|δ i |+∈

[0080] Where: δ i For TD error, ∈ is a very small constant used to prevent zero priority, δ i The calculation formula is as follows:

[0081]

[0082] Where: θ represents the training network parameters, ' represents the target network parameters, and Q(s) t ,a t ;θ) is obtained by training the network to understand the current state s t Forward propagation, select action a to be executed. t The corresponding Q value; r t The instant reward value is γ, the discount factor is γ, and the maximum value is γ. a Q(s t+1 ,a t ;θ') indicates the state s in the next state t+1 Next, select the action 'a' that maximizes the Q-value of the target network's output. t , The target network's Q-value;

[0083] Step 9-1-2: Calculate the sample priority sampling probability. The sample priority sampling probability P(i) is defined as follows:

[0084]

[0085] N represents the total capacity of the experience playback buffer, and the parameter σ controls the balance between uniform sampling and priority sampling, where σ∈{0,1}; specifically, setting σ=0 corresponds to pure uniform sampling, while σ=1 performs full priority sampling based on the timing difference error TD error;

[0086] Step 9-1-3: Calculate the importance sampling weights. In priority sampling, because high-error samples are used more frequently for updates and the data distribution is no longer the original empirical distribution, this can cause bias. The importance sampling weights are defined as follows:

[0087]

[0088] Where β is an adjustment parameter, the specific function of the formula is that when P(i) is large, w i Smaller values ​​reduce its impact on the loss function; when P(i) is small, wi If the value is relatively large, its weight should be increased appropriately.

[0089] Step 9-2: Based on the priority ranking of the data in the two experience pools, select the data with higher priority to train the spiking neural network. The specific steps are as follows:

[0090] Step 9-2-1: Sampling from two experience pools. Each time the network is updated, experience samples are taken from the two experience pools according to the sampling ratio calculated in step 8. Each sample contains the state s at time t. t Action a taken at time t t The reward value r of the action and the next state s after taking the action. t+1 ;

[0091] Step 9-2-2: Calculate the target Q value and select actions using the current network:

[0092]

[0093] Let A be the action space set, A = {'up', 'down', 'left', 'right'}

[0094] Calculate the target Q-value using the target network:

[0095]

[0096] Where γ is the discount factor; r t+1 This represents the expected reward at time t+1; θ represents the objective function value, i.e., the objective Q value, which is obtained by combining the reward value of the current action and the Q value of the optimal action corresponding to the next state; θ represents the parameters of the current neural network, and θ' represents the parameters of the target neural network;

[0097] Step 9-3: Update the parameters of the current network using the difference between the Q-value calculated by the current network and the Q-value of the target network;

[0098] Step 9-3-1: Use the Huber loss function to balance gradient stability and outlier robustness:

[0099]

[0100] In the formula, x is the TD error; when |x|<1: squared loss is used, the gradient is small, and the update is stable; when |x|≥1: linear loss is used to avoid gradient explosion.

[0101] Step 9-3-2: Calculate the importance sampling weight w in the empirical replay of step 9-1-3. i ;

[0102] Step 9-3-3: In the proposed method, the formula for calculating the overall loss function of the spiking neural network is:

[0103]

[0104] Where B: batch size, w i Weights are assigned to the importance of sample i.

[0105] Step 9-4: The formula for the SNN's impulse firing function is:

[0106] S[t]=Θ(U[ t ]-V th )

[0107] Where Θ is the step function, which is non-differentiable at the threshold, and U[t] is the membrane potential at time t. Due to the non-differentiability of the spiking neurons in a spiking neural network, a differentiable function with a similar shape is used to replace the actual gradient during backpropagation. The formula for the replacement gradient is as follows:

[0108]

[0109] Sl[ represents the gradient of the pulse output with respect to the membrane potential.] t ] represents the pulse output of layer l at time t, U l [t] represents the membrane potential of layer l at time t, V th Let σ' be the pulse firing threshold, and σ' be the fast sigmoid derivative, as shown in the following formula:

[0110]

[0111] Step 9-5: The formula for updating the weights of the SNN network layer is:

[0112]

[0113] Where η is the learning rate, W l Let L be the weight matrix of layer l. The overall loss function;

[0114] The actual formula for updating network weights is:

[0115] W l ←W l +ΔW l

[0116] Step 9-6: Network Update: The target network uses a soft update mechanism to maintain the stability of the learning objective and ensure a smooth training process. The update formula is as follows:

[0117] θ′←τ·θ+(1-τ)·θ′

[0118] Where τ is the soft update parameter, with a value range of 0.01 to 0.1.

[0119] Furthermore, step 10 specifically includes the following steps:

[0120] Step 10-1: Save and Load the Model: After training is complete, first save the weight parameters of the trained spiking neural network model as a .pth file, then load the saved model and perform performance testing.

[0121] Step 10-2: Use the trained SNN network to select the direction of movement, record the path coordinates in real time until the target is reached or the step limit is exceeded, and draw the path trajectory, obstacle distribution, start and end positions in the grid environment to intuitively present the path planning effect and obstacle avoidance capability of the model.

[0122] The beneficial effects of this invention are as follows:

[0123] 1. A spiking neural network was constructed as a Q-function approximator, and the temporal processing capability and biological rationality of the SNN were used to improve the efficiency of path planning.

[0124] 2. Design a dual experience pool to store ordinary experience and elite experience respectively, and use a priority experience replay mechanism based on TD error for the two experience pools. Adopt a dynamic adaptive sampling strategy to adjust the sampling ratio of the two experience pools, and intelligently balance the sample utilization rate according to the loss function and training phase during the training process.

[0125] 3. Optimize the reward function design by using Manhattan distance reward and repeated visit penalty mechanisms to guide the robot to explore the environment efficiently and accelerate convergence. Attached Figure Description

[0126] Figure 1 The method flowchart for the invention;

[0127] Figure 2 This is a diagram of the neuromorphic pulse deep reinforcement learning structure used in this invention;

[0128] Figure 3 The simulation map built for this invention;

[0129] Figure 4 This is a discrete action diagram for the present invention.

[0130] Figure 5 This is a diagram of the spiking neural network structure used in this invention;

[0131] Figure 6 This is a diagram illustrating the exploration rate decay mechanism used in this invention;

[0132] Figure 7 This is a structural diagram of the dual experience pool of the present invention;

[0133] Figure 8 This is a schematic diagram of the Suntree structure used in this invention;

[0134] Figure 9 The figure shows the simulation results of the present invention. Detailed Implementation

[0135] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0136] like Figure 1 , Figure 2 As shown, this invention is an autonomous navigation planning method for mobile robots based on neuromorphic pulse deep reinforcement learning, comprising the following steps:

[0137] Step 1: Set up a mesh environment as a simulation platform in Python. The mesh includes the robot's position, obstacles, passable areas, and the robot's start and end points. The specific steps are as follows:

[0138] Step 1-1: Set up a simulation environment in Python to simulate a mobile robot in a real environment;

[0139] Steps 1-2: Set a 20×20 grid area as the working area in the simulation environment. Multiple black obstacles can be set within this area. Obstacles are represented by black pixels, and passable areas are represented by white pixels; for example... Figure 3 As shown, the robot's starting point is set to the blue area in the upper left corner, and its ending point is set to the red area in the lower right corner.

[0140] Step 2: Define the robot's discrete motion space: up (0), down (1), left (2), right (3), and implement state transitions, such as... Figure 4 As shown, four robot actions are established in a grid environment: up, down, left, and right.

[0141] Step 3: Construct a training network and a target network based on a brain-like spiking neural network (SNN). Both networks have the same structure but different parameters. The training network has parameters θ, and the target network has parameters θ'. The specific steps are as follows:

[0142] Step 3-1: Construct the input layer. The input layer accepts a 20×20 grid environment state matrix as input, where obstacles are encoded as 1, free areas are encoded as 0, and the robot's current position is encoded as 2. The network environment state matrix is ​​converted into a four-dimensional tensor of shape [1,1,20,20] to meet the input requirements of the SNN network.

[0143] Step 3-2: Establish the spiking neural network layer. The network layer adopts a fully connected spiking neural network architecture, including a three-layer structure. Each layer consists of a fully connected layer and a LIF neuron layer.

[0144] First layer: Input dimension 400, output 256 spiking neurons, activated by snnntorch.Leaky;

[0145] The second layer has an input dimension of 256 and an output of 128 spiking neurons, maintaining the transmission of temporal information.

[0146] The third layer has an input dimension of 128 and an output of 64 spiking neurons, completing high-level feature extraction.

[0147] Step 3-3: Establish a membrane potential memory mechanism. Each spiking neuron layer maintains its own membrane potential state (mem1, mem2, mem3). During each forward propagation, the membrane potential accumulates with the time step and affects the neuron's spiking behavior. By initializing the membrane potential to None at the beginning of each training cycle, the system automatically re-initializes the membrane potential state, ensuring the independence and stability of the temporal calculations between different training samples.

[0148] Steps 3-4: Establish an output layer using a fully connected output layer. The input is a 64-dimensional pulse signal, and the output is the Q-value estimation of four actions, corresponding to the expected reward in the four movement directions of up, down, left, and right.

[0149] like Figure 5 As shown, the SNN network flattens the 20×20 grid map into a 400-dimensional input. After 256→128→64 dimensional compression and impulse feature extraction, it finally outputs the Q value corresponding to the four navigation actions of up, down, left and right. The robot can select the best movement direction based on the maximum Q value.

[0150] Step 4: Encode the robot's current position as a two-dimensional matrix and input it into the SNN network. Use the Q-values ​​predicted by the four actions as the output, and select the action with the largest Q-value at that moment. The specific steps are as follows:

[0151] Step 4-1: Encode the robot's current position information into a 20×20 two-dimensional matrix and input it into the spiking neural network, where the obstacle position value is 1, the passable area is 0, and the robot's current position is marked as 2. Extract the spatiotemporal features of the input matrix through the spiking neural network layer of the SNN, and use the temporal dynamic characteristics of the spiking neurons to capture the spatial structure of the environment and the robot's position relationship.

[0152] Step 4-2: Employ a deep Q-learning framework and use spiking neural network parameters θ to approximate the optimal action value function Q. * (S t ,at The network continuously optimizes its parameters by minimizing the temporal difference error between the current Q-value estimate and the target Q-value, so that Q(s) t ,a t ,θ) gradually converges to Q * (s t ,a t As shown in the following formula:

[0153] Q(s t ,a t ,θ)→Q * (s t ,a t )

[0154] Step 4-3: The network output layer generates a 4-dimensional vector, which corresponds to the Q-value prediction of the four actions: up, down, left, and right. Each Q-value represents the expected reward after taking the corresponding action in the current environment, providing a quantitative basis for the robot's action decision.

[0155] Step 5: The robot randomly selects an action to explore with an exploration probability ε, and selects the optimal action based on the SNN network with a probability of 1-ε. The value of ε decays during the training process. The specific steps are as follows:

[0156] Step 5-1: Exploration Probability Decay Design: The exploration probability ε dynamically decays during training, with an initial value of 0.8. To ensure sufficient exploration of the environment in the early stages of training, it gradually decays according to the following rules:

[0157] Keep ε = 0.8 constant in the early stages of training to ensure sufficient initial exploration;

[0158] In the later stages of training, the exploration rate is gradually reduced according to the decay coefficient, as shown in the following formula:

[0159] ε = max(ε min ,ε×γ decay )

[0160] Where γ decay =0.99, which is the attenuation coefficient; ε min =0.05;

[0161] Step 5-2: Action selection implementation: The robot randomly selects an action from the action space A = {'up', 'down', 'left', 'right'} with probability ε. With probability 1-ε, it uses the SNN network to forward propagate and calculate the Q value of the four actions, and selects the action corresponding to the largest Q value.

[0162] Step 6: Optimize the learning signal by calculating the reward value and performing a weighted sum of the main reward, Manhattan distance-guided reward, and repeat visit reward; the specific steps are as follows:

[0163] Step 6-1: Calculate the main reward, which includes the base step penalty, obstacle collision penalty, and goal achievement reward;

[0164] Base step penalty: The base reward for each move is r. o =-1, encourages the robot to reach the goal via the shortest path and avoids ineffective wandering behavior;

[0165] Obstacle collision penalty: When the robot attempts to move to an obstacle or tries to go out of bounds, a penalty of r is imposed. o A penalty of -3 is applied while maintaining the same position to ensure the safety of the path;

[0166] Reward for reaching the target location; when the robot successfully reaches the target location, it is rewarded with r. o A large reward of 100 clearly indicates the success of the task completion. The main reward formula is as follows:

[0167]

[0168] Step 6-2: Set Manhattan distance guidance reward: Calculate the Manhattan distance change between the robot and the target point in real time. A positive reward is given when the distance decreases, and a negative reward is given when the distance increases. The specific calculation formula is as follows:

[0169] r d =α·(d t-1 -d t )

[0170] Where: d t-1 d represents the Manhattan distance between the robot and the target point at time t-1. t Let d represent the Manhattan distance between the robot and the target point at time t, α be the reward coefficient, and d be the distance between the robot and the target point. t-1 and d t The calculation formula is as follows:

[0171] d t-1 =|r t-1 -r g |+|c t-1 -c g |

[0172] d t =|r t -r g |+|c t -c g |

[0173] Where: r t-1 and c t-1 Let r represent the row and column coordinates of the robot at time t-1, respectively. t and c tLet r represent the row and column coordinates of the robot at time t, respectively. g and c g The row and column coordinates representing the target location;

[0174] Step 6-3: Repeated Access Penalty Mechanism: Establish a location access record dictionary and impose an penalty (r) on repeated access to the same location. p = -3 penalty, otherwise r p =0, effectively preventing the robot from circling and wandering in local areas;

[0175] Step 6-4: Calculate the reward value, which is a weighted sum of the main reward, repeat visit reward, and Manhattan distance guidance reward:

[0176] r = r o +r d +r p .

[0177] Step 7: Establish a dual-priority experience replay mechanism and set a reward threshold r. elite The experience pool is divided into a regular experience pool and a management experience pool, and then based on (s) t ,a t ,s t+1 In the algorithm, reward r is used to achieve intelligent classification of experience samples; the specific steps are as follows:

[0178] Step 7-1: Establish a dual experience pool architecture, including a normal experience pool and an elite experience pool;

[0179] Step 7-2: State s at time t t Next, the robot selects an action a based on the current strategy. t After the action is performed, the environment returns to a new state s. t+1 And the immediate reward value r, these will all serve as a reference for subsequent learning;

[0180] Step 7-3: Categorize and store the experience samples according to their immediate reward value r:

[0181] When the reward value r≥r elite At that time, the sample is stored in the elite experience pool;

[0182] When the reward value r < r elite At that time, the sample is stored in the general experience pool;

[0183] Where: r elite The reward threshold is set to differentiate between normal experience and elite experience;

[0184] Step 7-4: Check the capacity of the two experience pools and remove the oldest experience; if the normal experience pool or elite experience pool reaches its capacity limit, remove the oldest experience.

[0185] Step 7-5: The experience samples from the two experience pools are fed into the network for training. After each training session, the sample priority is updated according to the new TD error to ensure that high-quality experience has a higher probability of being reused and to accelerate the learning process.

[0186] Step 8: Employ a dynamic adaptive sampling strategy, dynamically adjusting the sampling ratio of the two experience pools based on the training phase and loss function, using different sampling strategies in the auxiliary training phase and the main training phase; the specific steps are as follows:

[0187] Step 8-1: Establish a two-stage framework of auxiliary training stage and main training stage: the auxiliary training stage is used for the early stage of network training, focusing on basic skill learning, and the main training stage is used for the later stage of network policy optimization and convergence.

[0188] Step 8-2: The sampling ratio is dynamically calculated based on the training stage and the loss function;

[0189] Step 8-2-1: During the assisted training phase, the sampling ratio is dynamically calculated based on the loss function, as shown in the following formula:

[0190]

[0191] Where p0 is the proportion sampled from the normal experience pool; p1 is the proportion sampled from the elite experience pool, and the initial values ​​of p0 and p1 are both 0.5; ε is the exploration probability; l0 is the average loss of the normal experience pool; l1 is the average loss of the elite experience pool. The Huber loss is used to calculate the average loss of the experience pools, and the formula is as follows:

[0192]

[0193] In the formula, x is the TD error; when |x|<1: squared loss is used, the gradient is small, and the update is stable; when |x|≥1: linear loss is used to avoid gradient explosion.

[0194] During subsequent training, p0 will be updated based on the training loss from the two experience pools, and p1 will also be updated accordingly:

[0195] p1 = 1 - p0

[0196] Step 8-2-2: In the main training phase, focus on policy optimization and convergence, with a fixed sampling ratio: p0 = p1 = 0.5;

[0197] To ensure that both experience pools have sufficient sampling opportunities and to avoid sampling bias in extreme cases, a sampling boundary is set: p0∈[0.3,0.8].

[0198] Step 9: Update sample priorities using TD error and apply the overall loss function. Complete the network weight update; the specific steps are as follows:

[0199] Step 9-1: Calculate the priority, priority sampling probability, and importance sampling weight of each sample in the two experience pools;

[0200] Step 9-1-1: Calculate sample priority. Each experience pool uses a SumTree structure for priority sampling. The sample priority is calculated based on the temporal difference error (TD error).

[0201] p i =|δ i |+∈

[0202] Where: δ i For TD error, ∈ is a very small constant used to prevent zero priority, δ i The calculation formula is as follows:

[0203]

[0204] Where: θ represents the training network parameters, ' represents the target network parameters, and Q(s) t ,a t ;θ) is obtained by training the network to understand the current state s t Forward propagation, select action a to be executed. t The corresponding Q value; r t The instant reward value is γ, the discount factor is γ, and the maximum value is γ. a Q(s t+1 ,a t ;') indicates the next state s t+1 Next, select the action 'a' that maximizes the Q-value of the target network's output. t , The target network's Q-value;

[0205] Step 9-1-2: Calculate the sample priority sampling probability. The sample priority sampling probability P(i) of sample i is defined by the following equation:

[0206]

[0207] N represents the total capacity of the experience playback buffer, and the parameter σ controls the balance between uniform sampling and priority sampling, where σ∈{0,1}; specifically, setting σ=0 corresponds to pure uniform sampling, while σ=1 performs full priority sampling based on the timing difference error TD error;

[0208] Step 9-1-3: Calculate the importance sampling weights. In priority sampling, because high-error samples are used more frequently for updates and the data distribution is no longer the original empirical distribution, this can cause bias. The importance sampling weights are defined as follows:

[0209]

[0210] Where β is an adjustment parameter, the specific function of the formula is that when P(i) is large, w i Smaller values ​​reduce its impact on the loss function; when P(i) is small, w i If the value is relatively large, its weight should be increased appropriately.

[0211] Step 9-2: Based on the priority ranking of the data in the two experience pools, select the data with higher priority to train the spiking neural network. The specific steps are as follows:

[0212] Step 9-2-1: Sampling from two experience pools. Each time the network is updated, experience samples are taken from the two experience pools according to the sampling ratio calculated in step 8. Each sample contains the state s at time t. t Action a taken at time t t The reward value r of the action and the next state s after taking the action. t+1 ;

[0213] Step 9-2-2: Calculate the target Q value and select actions using the current network:

[0214]

[0215] Let A be the action space set, A = {'up', 'down', 'left', 'right'}

[0216] Calculate the target Q-value using the target network:

[0217]

[0218] Where γ is the discount factor; r t+1 This represents the expected reward at time t+1; θ represents the objective function value, i.e., the objective Q value, which is obtained by combining the reward value of the current action and the Q value of the optimal action corresponding to the next state; θ represents the parameters of the current neural network, and θ' represents the parameters of the target neural network;

[0219] Step 9-3: Update the parameters of the current network using the difference between the Q-value calculated by the current network and the Q-value of the target network;

[0220] Step 9-3-1: Use the Huber loss function to balance gradient stability and outlier robustness:

[0221]

[0222] In the formula, x is the TD error; when |x|<1: squared loss is used, the gradient is small, and the update is stable; when |x|≥1: linear loss is used to avoid gradient explosion.

[0223] Step 9-3-2: Calculate the importance sampling weight w in the empirical replay of step 9-1-3. i ;

[0224] Step 9-3-3: In the proposed method, the formula for calculating the overall loss function of the spiking neural network is:

[0225]

[0226] Where B: batch size, w i Weights are assigned to the importance of sample i.

[0227] Step 9-4: The formula for the SNN's impulse firing function is:

[0228] S[t]=Θ(U[t]-V th )

[0229] Where Θ is the step function, which is non-differentiable at the threshold, and U[t] is the membrane potential at time t. Due to the non-differentiability of the spiking neurons in a spiking neural network, a differentiable function with a similar shape is used to replace the actual gradient during backpropagation. The formula for the replacement gradient is as follows:

[0230]

[0231] Sl[ represents the gradient of the pulse output with respect to the membrane potential.] t ] represents the pulse output of layer l at time t, U l [t] represents the membrane potential of layer l at time t, V th Let σ' be the pulse firing threshold, and σ' be the fast sigmoid derivative, as shown in the following formula:

[0232]

[0233] Step 9-5: The formula for updating the weights of the SNN network layer is:

[0234]

[0235] Where η is the learning rate, W l Let l be the weight matrix of layer l. The overall loss function;

[0236] The actual formula for updating network weights is:

[0237] W l ←W l +ΔW l

[0238] Step 9-6: Network Update: The target network uses a soft update mechanism to maintain the stability of the learning objective and ensure a smooth training process. The update formula is as follows:

[0239] θ′←τ·θ+(1-τ)·θ′

[0240] Where τ is the soft update parameter, with a value range of 0.01 to 0.1.

[0241] Step 10: After training, save the trained model, test the model, output the final path, and visualize it. The specific steps are as follows:

[0242] Step 10-1: Save and Load the Model: After training is complete, first save the weight parameters of the trained spiking neural network model as a .pth file. Then load the saved model and perform performance testing.

[0243] Step 10-2: Use the trained SNN network to select the movement direction and record the path coordinates in real time until the target is reached or the step limit is exceeded. Draw the path trajectory, obstacle distribution, and start and end points in a grid environment to visually demonstrate the model's path planning effect and obstacle avoidance capability. Test the model results as follows: Figure 9 As shown.

[0244] The system of this invention was trained and tested in a simulation environment, and the training results are as follows: Figure 9 As shown, multiple metrics were selected for evaluation: the effectiveness of the planned path, the change in the robot's steps during training, and the change in the reward value during training. The changes in the robot's steps and reward value during training reflect the speed of the algorithm's convergence. The training batch size was 500. Figure 9 It can be seen that the training network converges around the 200th batch, and the path planning result obtained by the test model is the optimal path.

[0245] In summary, this invention is designed based on a spiking neural network and a Double DQN architecture. First, the robot's current position is encoded as a 20×20 grid matrix, serving as the input feature for the spiking neural network. The network extracts spatial navigation features of the environment through three fully connected layers and leakage integral-excited neurons, ultimately outputting the action value Q-values ​​in four directions: up, down, left, and right. This invention employs a dynamic dual-experience pool mechanism, classifying and storing experience samples into a normal pool and an elite pool based on reward values. The sampling ratio between the two pools is dynamically adjusted during the training phase, prioritizing the replay of high-quality experiences. During action decision-making, this invention combines an ε-greedy strategy with the Q-value maximization principle, selecting the action with the maximum Q-value in the current state as the robot's movement direction. Through the bio-inspired characteristics of the spiking neural network and the priority replay mechanism of the dual experience pool, this invention effectively improves the learning efficiency and navigation performance of autonomous navigation planning, achieving intelligent obstacle avoidance navigation in complex obstacle environments.

[0246] This invention can optimize the training process of the learning network, enabling the robot to make the most advantageous movement decisions based on real-time training results. At the same time, the design of the dual-priority experience pool and dynamic sampling probability makes the movements more efficient, greatly improving training efficiency.

[0247] The above description is a further detailed explanation of the present invention in conjunction with specific preferred embodiments. It should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all such deductions or substitutions should be considered to fall within the protection scope of the present invention.

Claims

1. A mobile robot autonomous navigation planning method based on brain-like pulse deep reinforcement learning, characterized in that, The method comprises the following steps: Step 1: build a grid environment as a simulation platform in a python environment, which includes the robot position, obstacles, passable areas, and the starting point and endpoint of the robot; Step 2: define the discrete action space of the robot: up (0), down (1), left (2), right (3), and realize state transition; Step 3: construct a training network and a target network based on a brain-like spiking neural network SNN, both of which have the same structure but different parameters, the training network parameters are θ, and the target network parameters are θ'; Step 4: encode the current position of the robot into a two-dimensional matrix input into the SNN network, and take the Q value of the four action predictions as the output, and select the action with the maximum Q value at this moment; Step 5: the robot randomly selects an action with an exploration probability ε, selects the optimal action according to the SNN network with a probability of 1-ε, and the value of ε decreases with the training process; Step 6: calculate the reward value, and weight and sum the main reward, Manhattan distance guide reward and repeated access reward to optimize the learning signal; Step 7: Establish a double-priority experience replay mechanism, set a reward threshold r elite , the experience pool is divided into ordinary experience pool and management experience pool, so as to realize intelligent classification of experience samples according to the reward r in (s t ,a t ,s t+1 ,r) where s t represents the state at time t, a t represents the action taken at time t, s t+1 is the next state after taking the action, and r is the immediate reward obtained from the action; Step 8: adopt a dynamic adaptive sampling strategy, dynamically adjust the sampling proportion of the two experience pools according to the training stage and the loss function, and adopt different sampling strategies in the auxiliary training stage and the main training stage; Step 9: Update sample priority by TD error, using overall loss function Complete network weight update, periodically soft update training network parameters to target network; Step 10: save the trained model after training, test the model, and output the final path and visualize the display.

2. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 1 specifically comprises the following steps: Step 1-1: build a simulation environment in a python environment to simulate a mobile robot in a real environment; Step 1-2: set a 20x20 grid area as a working area in the simulation environment, and set multiple black obstacles in the area, the obstacles in the grid are represented by black pixels, and the passable area is represented by white pixels.

3. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 3 specifically comprises the following steps: Step 3-1: build an input layer, which accepts a 20x20 grid environment state matrix as input, where obstacles are encoded as 1, free areas are 0, and the current position of the robot is encoded as 2, and the network environment state matrix is converted into a four-dimensional tensor with a shape of [1,1,20,20] to adapt to the input requirements of the SNN network; Step 3-2: establish a spiking neural network layer, which adopts a fully connected spiking neural network architecture, including three layers, each layer consisting of a fully connected layer and a LIF neuron layer: First layer: input dimension 400, output 256 spiking neurons, using snntorch.Leaky activation; Second layer: input dimension 256, output 128 spiking neurons, maintaining time sequence information transmission; Third layer: input dimension 128, output 64 spiking neurons, completing high-level feature extraction; Step 3-3: Establish a membrane potential memory mechanism, each pulse neuron layer maintains its own membrane potential state (mem1, mem2, mem3), during each forward propagation, the membrane potential accumulates with time steps and affects the pulse firing behavior of neurons, by initializing the membrane potential to None at the beginning of each training cycle, the system automatically reinitializes the membrane potential state, ensuring the independence and stability of the timing calculation between different training samples; Step 3-4: Establish an output layer, use a fully connected output layer, input is a 64-dimensional pulse signal, output Q value estimates of 4 actions, corresponding to the expected rewards of the up, down, left, and right four moving directions.

4. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 4 specifically includes the following steps: Step 4-1: Encode the current position information of the robot into a 20x20 two-dimensional matrix input pulse neural network, where the obstacle position value is 1, the passable area is 0, and the robot's current position is marked as 2, use the pulse neural network layer of SNN to extract the space-time features of the input matrix, and capture the spatial structure of the environment and the relationship between the robot's position using the timing dynamics of pulse neurons; Step 4-2: Using a deep Q-learning framework, network parameters θ are used to approximate the optimal action-value function Q * (s t ,a t ), the network continuously optimizes the network parameters by minimizing the temporal difference error between the current Q-value estimate and the target Q-value, so that Q(s t ,a t ,θ) gradually converges to Q * (s t ,a t ), as shown in the following formula: Q(s t ,a t ,θ)→Q * (s t ,a t ) Step 4-3: The network output layer generates a 4-dimensional vector, corresponding to the Q value prediction of the up, down, left, and right four actions respectively, each Q value represents the expected reward obtained after taking the corresponding action in the current environment state, providing a quantitative basis for the robot's action decision.

5. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 5 specifically includes the following steps: Step 5-1: Exploration probability decay design: the exploration probability ε dynamically decays during training, the initial value is set to 0.8, in order to ensure sufficient exploration of the environment in the early stage of training, and then gradually decay according to the following rules: Keep ε = 0.8 unchanged in the early stage of training to ensure sufficient initial exploration; In the later stage of training, gradually reduce the exploration rate according to the decay coefficient, the formula is as follows: ε = max(ε min , ε x γ decay ) where γ decay = 0.99 is the attenuation coefficient; ε min = 0.05; Step 5-2: Action selection implementation: the robot randomly selects an action from the action space A = {'up', 'down', 'left', 'right'} with probability ε, and uses the SNN network forward propagation to calculate the Q values of the four actions with probability 1-ε, and selects the action corresponding to the maximum Q value.

6. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 6 specifically includes the following steps: Step 6-1: Calculate the main reward, which includes the basic step penalty, obstacle collision penalty, and target arrival reward; Base step penalty: the base reward for each step moved is r o = -1, encouraging the robot to reach the goal in the shortest path, avoiding inefficient wandering behavior; Obstacle collision penalty: When the robot tries to move to an obstacle position or tries to cross the boundary, a penalty of r = -3 is given and the position is kept unchanged, ensuring the safety of the path. o = -3 is given and the position is kept unchanged, ensuring the safety of the path. Target arrival reward; r is given when the robot successfully arrives at the target position o A large reward of =100, which clearly identifies the success signal of task completion, the main reward formula is as follows: Step 6-2: Set the Manhattan distance guidance reward: calculate the Manhattan distance between the robot and the target point in real time, give a positive reward when the distance shortens, and a negative reward when the distance increases, the specific calculation formula is: r d = a - (d t-1 - d t ) wherein: d t-1 represents the Manhattan distance of the robot from the target point at time t-1, d t represents the Manhattan distance of the robot from the target point at time t, a is a reward coefficient, d t-1 and d t The calculation formula is as follows: d t-1 =|r t-1 -r g |+|c t-1 -c g | d t =|r t -r g |+|c t -c g | wherein: r t-1 and c t-1 denote the row and column coordinates of the robot at time t-1, respectively, r t and c t denote the row and column coordinates of the robot at time t, respectively, r g and c g denote the row and column coordinates of the target position; Step 6-3: Repeat visit penalty mechanism: Establish a location visit record dictionary, impose r p = -3 penalty for repeat visits to the same location, otherwise r p = 0, effectively preventing the robot from wandering in a local area; Step 6-4: Calculate the reward value, which is the weighted sum of the main reward, repeated access reward, and Manhattan distance guidance reward: r＝r o +r d +r p 。 7. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 7 specifically includes the following steps: Step 7-1: Establish a double experience pool architecture, including a normal experience pool and an elite experience pool; Step 7-2: At time t, state s t Next, the robot selects an action a according to the current policy t After executing the action, the environment returns a new state s t+1 and an immediate reward value r, which will be used as a reference for subsequent learning; Step 7-3: Store the immediate reward value r in the experience sample (s t , t , t+1 r) in a classified manner: When the reward value r ≥ r elite the sample is stored in the elite experience pool; When the reward value r < r elite the sample is stored in the normal experience pool; wherein: r elite is a set reward threshold for distinguishing between ordinary experience and elite experience; Step 7-4: Check the capacity of the two experience pools and remove the oldest experience; if the normal experience pool or the elite experience pool reaches the upper limit of the capacity, remove the oldest experience; Step 7-5: The experiences sampled from the two experience pools are fed into the network for training. After each training, the sample priority is updated according to the new TD error, ensuring that high-quality experiences have a higher reuse probability and accelerating the learning process.

8. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 8 specifically includes the following steps: Step 8-1: Establish a two-stage framework of auxiliary training stage and main training stage: the auxiliary training stage is used for early network training, focusing on basic skill learning, and the main training stage is used for later network policy optimization and convergence; Step 8-2: The sampling ratio is dynamically calculated according to the training stage and the loss function; Step 8-2-1: In the auxiliary training stage, the sampling ratio is dynamically calculated according to the loss function, as follows: Where p0 is the sampling ratio from the ordinary experience pool; p1 is the sampling ratio from the elite experience pool, p0, p1 training initial value is 0.5; ε is the exploration probability, l0 is the average loss of the ordinary experience pool, l1 is the average loss of the elite experience pool, the average loss of the experience pool is calculated using Huber loss, as follows: In the formula, x is the TD error; when |x| < 1: use square loss, small gradient, stable update; when |x| ≥ 1: use linear loss, avoid gradient explosion; In the subsequent training process, p0 and p1 will be updated according to the training loss of the two experience pools: p1 = 1 - p0 Step 8-2-2: In the main training stage, focus on policy optimization and convergence, the sampling ratio is a fixed value: p0 = p1 = 0.5; In order to ensure that both experience pools can obtain sufficient sampling opportunities and avoid sampling bias in extreme cases, set the sampling boundary: p0 ∈ [0.3, 0.8].

9. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 9 specifically includes the following steps: Step 9-1: Calculate the priority of each sample in the two experience pools, the priority sampling probability and the importance sampling weight; Step 9-1-1: Calculate the sample priority, use SumTree structure to realize priority sampling inside each experience pool, and calculate the sample priority based on the time difference error TD error: p i = |δ i |+∈ where: δ i is the TD error, ∈ is a very small constant to prevent zero priority, δ i is calculated as follows: Where: θ represents the training network parameters, θ' represents the target network parameters, and Q(s) represents the target network parameters. t ,a t ;θ) is obtained by training the network to understand the current state s t Forward propagation, select action a to be executed. t The corresponding Q value; r t The instant reward value is γ, the discount factor is γ, and the maximum value is γ. a Q(s t+1 ,a t ;θ') indicates the state s in the next state t+1 Next, select the action 'a' that maximizes the Q-value of the target network's output. t , The target network's Q-value; Step 9-1-2: Calculate the sample priority sampling probability, the priority sampling probability P(i) of sample i is defined as follows: N represents the total capacity of the experience replay buffer, and the parameter σ controls the balance between uniform sampling and priority sampling, where σ ∈ {0, 1}; Specifically, setting σ = 0 corresponds to pure uniform sampling, and σ = 1 corresponds to complete priority sampling based on the time difference error TD error; Step 9-1-3: Calculate the importance sampling weight, in priority sampling, because high error samples are used more frequently for updating and the data distribution is no longer the original experience distribution, which will cause bias, the importance sampling weight is defined as: where β is a regulation parameter, the specific role of the formula is that when P(i) is large, w i is small, reducing its influence in the loss function; when P(i) is small, w i is large, appropriately increasing its weight; Step 9-2: According to the priority ranking of the data in the two experience pools, select the data with high priority to train the spiking neural network, the specific steps are as follows: Step 9-2-1: Sampling from two experience pools: At each network update, experience samples are drawn from both experience pools according to the sampling proportions calculated in Step 8. Each sample contains the state s at time t t , the action a taken at time t t , the reward value r for the action, and the next state s after taking the action t+1 ; Step 9-2-2: Calculate the target Q value: select action using the current network: A is the action space set, A = {'up', 'down', 'left', 'right'} Use the target network to calculate the target Q value: where γ is a discount factor; r t+1 represents the expected reward at time t+1; represents the target function value, i.e., the target Q value, which is derived from the reward value of the current action and the Q value of the best action corresponding to the next state; θ is the training network parameter, and θ' is the target network parameter; Step 9-3: Update the parameters of the current network using the difference between the Q value calculated by the current network and the target network Q value; Step 9-3-1: Use the Huber loss function to balance gradient stability and outlier robustness: where x is the TD error; when |x| < 1: use square loss, small gradient, stable update; when |x| ≥ 1: use linear loss, avoid gradient explosion; Step 9-3-2: Calculate the importance sampling weight w in the experience replay of Step 9-1-3 i ; Step 9-3-3: In the proposed method, the overall loss function calculation formula of the spiking neural network is: where B: batch size, w i Importance sampling weight for sample i; Step 9-4: The spike firing function formula of SNN is: S[t] = Θ(U[t] - V th ) where Θ is the step function, which is not differentiable at the threshold, and U[t] is the membrane potential at time t. Due to the non-differentiability of the spiking neurons of the spiking neural network, a similar differentiable function is used to replace the true gradient during backpropagation. The substitute gradient formula is as follows: S is the gradient of the membrane potential for the pulse output l [t] is the pulse output of layer l at time t, U l [t] is the membrane potential value of layer l at time t, V th S is the gradient of the membrane potential for the pulse output Step 9-5: The SNN network layer weight update formula is: where η is the learning rate, W l is the weight matrix for layer / , is the overall loss function; The actual network weight update formula is: W l ←W l +ΔW l Step 9-6: Network update: the target network uses a soft update mechanism to maintain the stability of the learning goal and ensure a smooth training process. The update formula is as follows: θ' <- τ·θ + (1-τ)·θ' Where τ is the soft update parameter, with a value range of 0.01-0.

1.

10. The autonomous navigation planning method for mobile robots based on brain-inspired spiking deep reinforcement learning according to claim 1, characterized in that, The step 10 specifically comprises the following steps: Step 10-1: Save and load the model: after training, first save the trained spiking neural network model weight parameters as a.pth file, then load the saved model and perform performance testing, Step 10-2: Use the trained SNN network to select the moving direction, record the path coordinates in real time until reaching the target or exceeding the step limit, and draw the path trajectory, obstacle distribution, starting point and ending point position in the grid environment, to intuitively present the path planning effect and obstacle avoidance ability of the model.

Images