Mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning

Abstract

The invention relates to the technical field of mobile robot obstacle avoidance motion planning, solves the technical problems that the energy consumption is too high and the autonomous navigation capability and the obstacle avoidance performance are insufficient in the flexible obstacle avoidance motion planning process of a mobile robot, and particularly relates to a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning. The low energy consumption characteristic of the spiking neural network is combined with the flexible decision-making mechanism of reinforcement learning, so that the mobile robot can autonomously generate the optimal path in a complex environment, and meanwhile, the calculation cost and the energy waste are effectively reduced. According to the invention, the obstacle avoidance performance of the mobile robot in a dynamic environment can be effectively improved, so that the mobile robot can realize safe and efficient autonomous navigation in a complex and changeable environment, and the mobile robot can still maintain excellent performance while reducing energy consumption, thereby promoting wide application and practical progress of intelligent equipment in the field of intelligent manufacturing.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle avoidance motion planning for mobile robots, and particularly to an obstacle avoidance motion planning method for mobile robots based on pulsed hybrid reinforcement learning. Background Art

[0002] With the increasingly wide application of mobile robots in real life, the issues of their safety and efficiency in complex environments have received more and more attention. Especially in scenarios where they work together with dynamic and non-networked agents (such as pedestrians or other robots), how to effectively avoid collisions with other objects has become a key factor in ensuring the safe operation and efficient operation of robots. In crowded and dynamically changing environments, robots need to work in highly coordinated ways with humans and other robots.

[0003] In current robot obstacle avoidance technologies, robots need to perform a series of decision-making and execution processes such as environmental mapping, path planning, and motion control. Common methods include the A* algorithm, Dijkstra algorithm, artificial potential field method, dynamic window approach (DWA), etc. However, for the obstacle avoidance problem of mobile robots in dynamic environments, mobile robots face the problem of excessive energy consumption during the flexible obstacle avoidance motion planning process, and there are deficiencies in the autonomous navigation ability and obstacle avoidance performance in complex dynamic environments. As a result, mobile robots cannot achieve safe and efficient autonomous navigation in complex and changeable environments. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides an obstacle avoidance motion planning method for mobile robots based on pulsed hybrid reinforcement learning, which solves the technical problems of excessive energy consumption faced by mobile robots during the flexible obstacle avoidance motion planning process, as well as the deficiencies in the autonomous navigation ability and obstacle avoidance performance.

[0005] To solve the above technical problems, the present invention provides the following technical solution: An obstacle avoidance motion planning method for mobile robots based on pulsed hybrid reinforcement learning, the method comprising the following steps: S1. In any environmental interaction of the mobile robot, obtain the observation information including the observation value and the action space in each step scenario from the sensor; S2. Establish a hybrid pulsed reinforcement learning model that takes the observation information as input, enhances the mobile robot's perception of environmental information in the dynamic environment, and outputs the action instructions required for the mobile robot to avoid obstacles in the dynamic environment; S3. Establish a reward mechanism for guiding the path planning training of the mobile robot when the mobile robot executes the action instructions; S4. Optimize the parameters of the hybrid pulsed reinforcement learning model to obtain the optimal motion path for guiding the mobile robot to complete the obstacle avoidance task.

[0006] The present invention uses a population spiking neural network as a policy network, enabling a mobile robot to more accurately perceive changes in the surrounding environment and make rapid and reasonable decisions based on these changes. In terms of model design, the present invention adopts a population spiking coding method. Through a Gaussian distribution encoder, the observation information obtained by the mobile robot from sensors is converted into a spike train. After being processed by a multi-layer spiking neural network, these spike trains are decoded into continuous action instructions. This coding and decoding method can fully utilize the temporal processing ability of the spiking neural network, enabling the mobile robot to be more sensitive to environmental information perception in a dynamic environment. Through the above design, the mobile robot obstacle avoidance motion planning method based on hybrid spiking reinforcement learning proposed by the present invention can effectively improve the obstacle avoidance performance of the mobile robot in a dynamic environment, enabling it to achieve safe and efficient autonomous navigation in a complex and changeable environment.

[0007] Furthermore, the hybrid spiking reinforcement learning model includes: An encoding module with population coding, used to map observation information to the activity activation values of multi-channel neurons and generate a spike train with a fixed time step; A spiking neural network module with the spike train as the input, used to simulate the membrane potential change, synaptic current accumulation, and spike triggering mechanism of biological neurons and output encoded activities; And a decoding module that converts the encoded activities output by the spiking neural network module into a more intuitive action space expression.

[0008] Furthermore, the observed value: in the task of training to avoid obstacles, at each time step simultaneously includes the information of the mobile robot and the information of the obstacle; The action space: at each time step defines the manipulator action , respectively representing the velocity information in the

[0009] Furthermore, in the encoding module, it includes: Construct multiple receptive field neurons for the observed value of each input dimension . Assuming the input observed value is [[ID=3,2]]the response value of the th neuron is expressed as: ; Among them, is the receptive field center uniformly set in the input space; is the standard deviation, used to determine the activity intensity of the neuron; Subsequently, these response values is input into a voltage accumulator for time-series integration. When the membrane potential exceeds the firing threshold , the neuron fires a pulse at this time step. The iterative process is as follows: ; wherein, is the voltage value at time step ; is the pulse value fired at time step ; is a judgment function. If the input neuron voltage exceeds the threshold , it takes 1, otherwise it takes 0.

[0010] Furthermore, in the pulse neural network module, it includes: Introduce a pseudo-derivative mechanism, that is, artificially assign a non-zero gradient within the interval , that is: ; wherein, represents the neuron voltage; Determine whether the neuron will fire a pulse by calculating the change in the neuron membrane voltage. Its current update rule is as follows: ; wherein, is the decay factor of the current; is the weight of the linear layer; After introducing a learnable membrane time constant, the update rule of the membrane potential[[ID=<<MASK_0>>]] of the neuron at time step is as follows: ; wherein, , is the sigmoid activation function; represents the voltage value triggered at time; represents the external input; is the reset voltage; The update rule of any pulse neuron pulse signal is as follows: ; wherein, represents processing each element of the input vector. If it exceeds the threshold , it takes 1, otherwise it takes 0; After firing a pulse, reset the membrane potential of the neuron. The membrane potential after discharging is updated as follows: ; Among them, is the attenuation factor of the voltage.

[0011] Furthermore, in the spiking neural network module, it includes: For each action dimension , its corresponding spiking population will be linearly integrated and then non-linearly activated, that is: ; Among them, is the convolutional weight vector in this dimension; is the Tanh activation function.

[0012] Furthermore, in step S3, the specific process includes: If the mobile robot moves towards the target, a positive reward will be given in proportion to the decreasing distance , and the specific update rule is as follows: ; Among them, represents the Euclidean distance from the mobile robot to the target at time step; If it is found that the mobile robot is getting closer and closer to the obstacle, a progressive penalty will be given , and the specific formula is as follows: ; Among them, represents the minimum distance between the mobile robot and the obstacle; If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty will be given and the round will end, that is: ; Among them, , are the sizes of the mobile robot and the obstacle respectively; When the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large amount of reward will be given at one time and the task will be terminated, that is: ; Among them, represents the range of the target point; Introduce a time penalty , and based on the positive reward , penalty , penalty and the large amount of reward , obtain the total reward for guiding the path planning training of the mobile robot , and the expression is: ; wherein, represents the total reward under the reward mechanism.

[0013] Furthermore, in step S4, the specific process includes the following steps: S41. Establish a deep reinforcement learning framework for optimizing the parameters of the hybrid pulse reinforcement learning model based on the maximum entropy reinforcement learning algorithm. The deep reinforcement learning framework includes a policy network that generates a probability distribution to determine the actions of the mobile robot, and a value network that evaluates the value of the state-action pair; S42. Use minimizing the policy loss and the Q-value loss as the training objectives of the deep reinforcement learning framework to train the policy network and the value network; S43. Based on the training objective, use maximizing the expected return of the policy and minimizing the entropy of the policy as the objective for optimizing the policy loss, and the expression is: ; wherein, represents randomly sampling from the experience replay pool D; represents the entropy regularization coefficient, which is used to balance exploration and exploitation; represents the state and the action of the value, and are two different value estimates of the value network for the same state-action pair; S44. Calculate the prediction of the future reward of the hybrid pulse reinforcement learning model based on the reward mechanism , that is:[[ID=4)5]] ; wherein, represents the reward obtained from the current state and action; represents the discount factor, which determines the influence of future rewards; represents the logarithmic probability of the action under the current policy in the state<) , which is used to encourage the mobile robot to conduct more exploration; S45. According to the prediction of the future reward , use the double Q network to minimize the loss function to train the value network, that is: ; wherein, represents the value network loss; S46. Randomly initialize the parameters of the entire deep reinforcement learning framework and use the Adam optimizer to update the weights of each module.

[0014] Furthermore, the policy network includes an encoding module, a spiking neural network module, and a decoding module; The value network is composed of two MLP instances with a final output dimension of 1, and each MLP instance consists of three fully connected layers.

[0015] By means of the above technical solutions, the present invention provides a method for obstacle avoidance motion planning of a mobile robot based on spiking hybrid reinforcement learning, which has at least the following beneficial effects: 1. The present invention can effectively improve the obstacle avoidance performance of the mobile robot in a dynamic environment, enabling it to achieve safe and efficient autonomous navigation in a complex and changing environment. While reducing energy consumption, the mobile robot can still maintain excellent performance, thus promoting the wide application and practical process of intelligent equipment in the field of intelligent manufacturing.

[0016] 2. By combining the low energy consumption characteristics of the spiking neural network with the flexible decision-making mechanism of reinforcement learning, the mobile robot can autonomously generate the optimal path in a complex environment while effectively reducing the computational cost and energy waste. While reducing energy consumption, the mobile robot can still maintain excellent performance, thus promoting the wide application and practical process of intelligent equipment in the field of intelligent manufacturing.

[0017] 3. By cleverly combining the efficient energy utilization characteristics of the spiking neural network with the dynamic decision-making ability of reinforcement learning, the mobile robot can autonomously plan the optimal path in a complex environment while reducing unnecessary calculations, thereby significantly reducing energy consumption during operation. Description of the Drawings

[0018] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 is the flowchart of the method for obstacle avoidance motion planning of the mobile robot in the present invention; Figure 2 is the flowchart of generating action instructions by the hybrid spiking reinforcement learning model in the present invention; Figure 3 is the simulation schematic diagram of the obstacle avoidance motion planning of the mobile robot in the present invention. Detailed Embodiments

[0019] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Thus, the implementation process of how this application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0020] The research on robot obstacle avoidance technology not only has important theoretical significance but also has extensive practical application value. First of all, it is a key technology to achieve the autonomy and intelligence of robots, enabling robots to complete path planning and real-time dynamic adjustment without human intervention, thereby improving the flexibility and stability of their task execution. Secondly, the maturity of obstacle avoidance technology will greatly expand the application scenarios of robots. For example, in domestic service robots, they need to avoid furniture and people; in warehousing and logistics, they need to efficiently shuttle between shelves; in extreme environments such as agriculture, medical, construction, and even disaster relief, a highly reliable obstacle avoidance ability is more needed to ensure task completion and personnel safety. Therefore, in-depth research on robot obstacle avoidance technology is not only an embodiment of technological progress but also a key path to achieve a higher level of human-machine collaboration and social automation in the intelligent era.

[0021] As the third generation of artificial neural networks, spiking neural networks (SNNs) have received extensive attention for their information processing methods that are closer to biological neural systems. Different from traditional neural networks, SNNs transmit information through discrete spike signals, possessing temporal dynamic characteristics and sparse computing advantages, which show significant advantages in power consumption control and computing efficiency. Since spikes only occur when necessary, SNNs can greatly reduce redundant computations while maintaining performance, especially suitable for deployment in resource-constrained edge devices or low-power scenarios. At the same time, SNNs have a natural sensitivity to temporal information and can effectively process tasks with strong time correlations, such as sound recognition, event detection, and dynamic vision processing. In addition, their response to input changes is more agile, with a certain degree of robustness and generalization ability, providing a new direction for constructing more intelligent, efficient, and energy-saving neural network systems. With the development of neuromorphic hardware, SNNs also show great application potential in fields such as robot control, neuroscience modeling, and brain-inspired computing.

[0022] This embodiment combines reinforcement learning and spiking neural networks, demonstrating significant advantages. Reinforcement learning provides the system with the ability to dynamically adjust strategies, enabling it to autonomously learn optimal behaviors based on environmental feedback. Spiking neural networks, on the other hand, simulate the working mechanism of biological neurons in an efficient way of time and energy utilization, improving computational efficiency and robustness. This combination not only speeds up the learning process, enhances the adaptability and stability of the system, but also significantly reduces energy consumption and strengthens the resistance to noise and interference. At the same time, its learning mechanism is closer to biological intelligence, providing new ideas for achieving more advanced artificial intelligence, and has important research value and broad application prospects.

[0023] Based on the spiking neural network (SNN), this embodiment proposes a mobile robot obstacle avoidance motion planning method based on spiking hybrid reinforcement learning. By introducing a low-energy-consuming spiking neural network into the policy network, while retaining the traditional deep neural network structure in the value network. It enables the mobile robot to maintain excellent performance while reducing energy consumption, thus promoting the wide application and practical process of intelligent equipment in the field of intelligent manufacturing. As Figure 1 shown, this method includes the following steps: S1. In any environmental interaction of the mobile robot, obtain the observation information including the observation value and the action space in each step scenario from the sensor. Among them: Observation value: In the task of training to avoid obstacles, at each time step the observation information includes both the mobile robot information and the obstacle information. Specifically, the position of the mobile robot, the speed of the mobile robot, the target position, and the obstacle position. So the observation value is represented by dimensional data, representing the number of obstacles.

[0024] Action space: The action space is related to the joint speed control of the mobile robot. At each time step define the manipulator action which respectively represent the speed information in the

[0025] S2. Establish a hybrid spiking reinforcement learning model that takes the observation information as input, enhances the mobile robot's perception of environmental information in a dynamic environment, and outputs the action instructions required for the mobile robot to avoid obstacles in a dynamic environment. The hybrid spiking reinforcement learning model consists of an encoding module, a spiking neural network module, and a decoding module, as Figure 2 shown.

[0026] This embodiment designs a population coding module to map the observation information into the activity activation value of multi-channel neurons and generate a pulse sequence with a fixed time step. The observation value of constructs multiple receptive field neurons, whose response function follows Gaussian distribution. Suppose the input observation value is , Indicates the dimension Observed values The set of The response value of a neuron It can be expressed as: ; in, is the receptive field center uniformly set in the input space; is the standard deviation, which is used to determine the activity strength of the neuron. The choice of Gaussian distribution means that the response of the neuron to different input features is based on probability. Specifically, the response of the neuron to the observation value will follow the pattern of normal distribution.

[0027] Subsequently, these response values It is input into a voltage accumulator for time series integration. When the membrane potential exceeds the firing threshold When , the neuron emits a pulse at this time step. The iterative process is as follows: ; in, For the time step The voltage value at For the time step The pulse value released when Is a judgment function, if the input neuron voltage Exceeding the threshold , then it takes 1, otherwise it takes 0. In this embodiment, the threshold The value is 0.5. This mechanism ensures that neurons only generate discrete event signals under sufficient activation conditions, while maintaining high-fidelity representation capabilities for continuous inputs.

[0028] This example designs a spiking neural network module that takes a pulse sequence as input, including a spiking neural network based on the leakage-integration-discharge principle. The resulting pulse signal is used to output the resulting spike activity. The spiking neural network module simulates changes in biological neuron membrane potential, synaptic current accumulation, and spike triggering mechanisms, and outputs encoded activity.

[0029] Since the impulse function itself is not differentiable, in order to make the network trainable, a pseudo-derivative mechanism is introduced, that is, in the interval The internal gradient is artificially assigned non-zero, that is: ; Among them, represents the neuron voltage. While maintaining the pulse property, this mechanism allows end-to-end training using the conventional gradient descent method.

[0030] By calculating the change in the neuron membrane voltage to determine whether the neuron will fire a pulse, its current The update rule is as follows: ; Among them, is the decay factor of the current; is the weight of the linear layer; After introducing the learnable membrane time constant, the membrane potential of the neuron at time step is updated as follows: ; Among them, , ensuring . In the experiment, is the sigmoid activation function, that is ; represents The voltage value triggered at time ; represents the external input; is the reset voltage; The update rule of any spike neuron spike signal is as follows: ; Among them, represents processing each element of the input vector. If it exceeds the threshold , then take 1, otherwise take 0; After firing a pulse, reset the membrane potential of the neuron. The membrane potential after discharging is updated as follows: ; Among them,[[ID=7"]] is the decay factor of the voltage.

[0031] The spiking neural network module consists of 3 layers connected in sequence, namely two spiking neural layers and one output layer; among them, the role of the spiking neural layer is to convert the input data into a format suitable for processing by spiking neurons. The number of spiking neurons in the above spiking neural layers is 256, 256, and 20 respectively.

[0032] The final output of the spiking neural network module is a set of population neuron spike activities representing the action dimensions. To convert it into a continuous control signal, this embodiment designs a decoder based on one-dimensional grouped convolution to convert the encoded activities output by the spiking neural network module into a more intuitive action space representation. For each action dimension , its corresponding population of spikes will be linearly integrated and passed through a non-linear activation, i.e.: ; where is the convolution weight vector for this dimension, and T represents the transpose; it is converted into a continuous action space through one-dimensional convolution and an activation function. is the Tanh activation function, which ensures that the amplitude of the output action is limited within the interval [-1, 1], thus ensuring the controllability of the action. The role of the convolutional neural network is to extract effective feature information from the activities of neurons and map it to the action output of the mobile robot. [[ID=U14]]

[0033] S3. Establish a reward mechanism for guiding the path planning training of the mobile robot under the execution of action instructions. Further, this embodiment designs a detailed reward mechanism to ensure that the mobile robot can obtain sufficient reward values when it avoids obstacles and reaches the destination correctly without collision; while when the mobile robot collides, a certain penalty is given, thus effectively guiding the path planning training of the mobile robot. The overall reward mechanism formula is as follows: ; To encourage the mobile robot to complete the task as soon as possible, there is a small time penalty for each step , which is taken as 0.1 in this embodiment. This embodiment explains the remaining reward items and penalty items in detail as follows: If the mobile robot moves towards the target, a positive reward is given according to the proportion of the distance reduction , and the specific update rule is as follows: <U30> ; where represents the Euclidean distance from the mobile robot to the target at time step. This formula shows that the closer the mobile robot is to the target, the greater the reward value.

[0034] If it is found that the mobile robot is getting closer and closer to the obstacle, a progressive penalty is given , and the specific formula is as follows: ; where Represents the minimum distance between the mobile robot and the obstacle. This formula indicates that when it is found that the mobile robot is getting closer and closer to the obstacle, a certain penalty is given.

[0035] If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty is given And end the round, that is: ; Among them, 、 Are the size of the mobile robot and the obstacle respectively. When the distance between the mobile robot and the obstacle is less than the sum of the sizes of the mobile robot and the obstacle, it means that the two have collided, and a fixed penalty value is given.

[0036] When the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large one-time reward is given And terminate the task, that is: ; Among them, Represents the range of the target point, indicating that the mobile robot has reached the target point, the task is completed, and a large reward is given.

[0037] S4. Optimize the parameters of the hybrid pulse reinforcement learning model to obtain the optimal motion path for guiding the mobile robot to complete the obstacle avoidance task. In this embodiment, a deep reinforcement learning framework (SAC) is designed based on the maximum entropy reinforcement learning algorithm (Soft Actor-Critic) to optimize the parameters of the hybrid pulse reinforcement learning model, so as to obtain the optimal motion path, and use this optimal motion path to guide the mobile robot to complete the obstacle avoidance task.

[0038] The core framework of the deep reinforcement learning framework (SAC) includes a policy network and a value network. The purpose of the policy network is to train an optimal policy so that the mobile robot can select the optimal action under a given state. Among them, the policy network generates a probability distribution (Gaussian distribution) to determine the action of the mobile robot. The value network is used to evaluate the value of the state-action pair. The training objective of the deep reinforcement learning framework (SAC) is to minimize the policy loss and the Q-value loss to train the policy network and the value network. Specifically: The goal of optimizing the policy loss is to maximize the expected return of the policy and minimize the entropy of the policy (that is, maximize exploration). The specific formula is: ; Among them, Represents randomly sampling samples from the experience replay pool D; Represents the policy loss.

[0039] In this embodiment, the value network is trained by using a double Q-network to minimize the loss function. This method helps to reduce the overestimation bias, thereby improving the stability of the algorithm. That is: ; Among them, represents the loss of the value network; represents the prediction of future rewards, which is calculated through the target network and the policy network. and are two different value estimates of the value network for the same state-action pair. The value function is updated based on the Bellman equation, and the formula is as follows: ; Among them, represents the state and the action of value; represents the reward obtained from the current state and action; represents the discount factor, which determines the impact of future rewards; represents the expectation of future actions, and the actions are generated by the current policy ; represents the log probability of the action in the state under the current policy, which is used to encourage the mobile robot to explore more; represents the entropy regularization coefficient, which is used to balance exploration and exploitation. The calculation formula of the future reward prediction is as follows: ; The policy network in the deep reinforcement learning framework (SAC) includes the above-mentioned encoding module, spiking neural network module, and decoding module. The value network is composed of two MLP instances, and each MLP instance is composed of three fully connected layers. The input dimension is 64 + 2 = 66, and the final output dimension is 1.

[0040] Randomly initialize the parameters of the entire deep reinforcement learning framework (SAC), calculate the gradient of the actor loss with respect to the actor network parameters, and use the Adam optimizer to update the weights of the above encoding module, spiking neural network module, and decoding module.

[0041] In this embodiment, in order to verify the effectiveness of this method, a maze simulation environment is constructed. In this environment, the mobile robot needs to achieve autonomous navigation in a continuous action space, while avoiding dynamic obstacles and reaching the target position, as Figure 3As shown below. In this embodiment, different obstacle scenarios were tested, and the obtained results are shown in Table 1: Table 1 Results under Different Obstacles

[0042] This embodiment proposes a method for obstacle avoidance motion planning of a mobile robot based on pulsed hybrid reinforcement learning. This solution significantly reduces energy consumption during the operation of the system. By combining the low energy consumption characteristics of the pulsed neural network with the flexible decision-making mechanism of reinforcement learning, the robot can autonomously generate the optimal path in a complex environment, effectively reducing the computational cost and energy waste at the same time. And a membrane time constant that can introduce learning is introduced into the pulsed neural network, greatly enhancing the expressive ability of the network. This innovative method not only improves the endurance and working efficiency of the robot, but also enhances its adaptability and robustness in complex environments. In this way, this application aims to provide a brand-new solution for the autonomy and energy efficiency of mobile robots in practical applications, promote the wide application of robot technology in fields such as logistics, rescue, and exploration, and also provide new ideas and theoretical support for the design and development of intelligent robot systems.

[0043] Those of ordinary skill in the art can understand that all or part of the steps in implementing the above embodiment method can be completed by instructing relevant hardware through a program. Therefore, this application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0045] The above embodiments have introduced the present invention in detail. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A motion planning method for mobile robot obstacle avoidance based on pulsed hybrid reinforcement learning, characterized in that, The method includes the following steps: S1. In any environmental interaction of the mobile robot, obtain the observation information including the observation value and the action space in each step scenario from the sensor; S2. Establish a hybrid pulse reinforcement learning model that takes the observation information as input, enhances the mobile robot's perception of environmental information in a dynamic environment, and outputs the action instruction required for the mobile robot to avoid obstacles in the dynamic environment; the hybrid pulse reinforcement learning model includes: An encoding module with population coding, which is used to map the observation information to the activity activation values of multi-channel neurons and generate a pulse sequence with a fixed time step; A pulse neural network module that takes the pulse sequence as input, is used to simulate the membrane potential change, synaptic current accumulation and pulse triggering mechanism of biological neurons and outputs the coding activity; And a decoding module that converts the coding activity output by the pulse neural network module into a more intuitive action space expression; S3. Establish a reward mechanism for guiding the path planning training of the mobile robot when the mobile robot executes the action instruction; S4. Optimize the parameters of the hybrid pulse reinforcement learning model to obtain the optimal motion path for guiding the mobile robot to complete the obstacle avoidance task.

2. The mobile robot obstacle avoidance motion planning method according to claim 1, characterized in that In step S1: The observed values: In the task of training to avoid obstacles, at each time step simultaneously include the information of the mobile robot and the information of the obstacles; The action space: At each time step the manipulator actions are defined which respectively represent the velocity information in the 3. The obstacle avoidance motion planning method for a mobile robot according to claim 1, characterized in that In the encoding module, it includes: For each input dimension a plurality of receptive field neurons are constructed for the observed values, and let the input observed value be , then the response value of the th neuron is expressed as: ； Among them, is the receptive field center uniformly set in the input space; is the standard deviation, which is used to determine the activity intensity of the neuron; Subsequently, these response values are input into a voltage accumulator for temporal integration. When the membrane potential exceeds the firing threshold the neuron fires a pulse at this time step, and its iterative process is as follows: ； Among them, is the voltage value at time step ; is the pulse value emitted at time step ; is a judgment function. If the input neuron voltage exceeds the threshold , it takes 1, otherwise it takes 0.

4. The mobile robot obstacle avoidance motion planning method according to claim 3, characterized in that, In the pulse neural network module, it includes: Introduce a pseudo-derivative mechanism, that is, artificially assign a non-zero gradient within the interval as follows: ； Among them, represents the neuron voltage; Determine whether a neuron will fire by calculating the change in the membrane voltage of the neuron, and its current The update rule is as follows: ； Among them, is the decay factor of the current; is the weight of the linear layer; After introducing a learnable membrane time constant, the membrane potential of the neuron at time step is updated according to the following rule: ​ ； Among them, is the sigmoid activation function; represents the voltage value triggered at time represents the external input; is the reset voltage; The pulse signal of any pulse neuron has the following update rules: ； Among them, indicates that each element of the input vector is processed. If it exceeds the threshold , then take 1, otherwise take 0; After the pulse is delivered, the membrane potential of the neuron is reset, and the membrane potential after firing is updated as follows: ； Among them, is the attenuation factor of the voltage.

5. The mobile robot obstacle avoidance motion planning method according to claim 4, wherein In the pulse neural network module, it includes: For each action dimension , its corresponding pulse population will be linearly integrated and non-linearly activated, i.e.: ； Among them, is the convolution weight vector in this dimension; is the Tanh activation function.

6. The obstacle avoidance motion planning method for a mobile robot according to claim 1, characterized in that In step S3, the specific process includes: If the mobile robot moves towards the target, a positive reward is given in proportion to the decrease in distance , and the specific update rule is as follows: ； Among them, represents the Euclidean distance from the mobile robot to the target at the time step; If it is found that the mobile robot is getting closer and closer to an obstacle, a progressive penalty is given , and the specific formula is as follows: ； Among them, represents the minimum distance between the mobile robot and the obstacle; If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty is imposed and the round ends, that is: ； Among them, and are the size of the mobile robot and the obstacle respectively; If the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large one-time reward will be given. And terminate the task, that is: ； Among them, represents the range of the target point; Introduce time penalty , and based on positive rewards , penalties , penalties and large rewards , obtain the total reward for guiding the path planning training of the mobile robot , the expression is: ； Among them, represents the total reward under the reward mechanism.

7. The obstacle avoidance motion planning method for a mobile robot according to claim 1, characterized in that, In step S4, the specific process includes the following steps: S41. Based on the maximum entropy reinforcement learning algorithm, establish a deep reinforcement learning framework for optimizing the parameters of the hybrid pulse reinforcement learning model. The deep reinforcement learning framework includes a policy network that generates a probability distribution to determine the actions of the mobile robot, and a value network that is used to evaluate the value of the state-action pair; S42. Use minimizing the policy loss and the Q-value loss as the training objectives for training the policy network and the value network in the deep reinforcement learning framework; S43. Based on the training objective, use maximizing the expected return of the policy and minimizing the entropy of the policy as the objective for optimizing the policy loss, and the expression is: ； Among them, represents randomly sampling a sample from the experience replay pool D; represents the entropy regularization coefficient, which is used to balance exploration and exploitation; represents the state and the action of value, and are two different value estimates of the value network for the same state-action pair; S44. Calculate the prediction of the future reward by the hybrid pulse reinforcement learning model based on the reward mechanism , that is: ； Among them, represents the reward obtained from the current state and action; represents the discount factor, which determines the impact of future rewards; represents, under the current policy, the action in the state with the logarithmic probability, which is used to encourage the mobile robot to conduct more explorations; S45. According to the prediction of future rewards , use the double Q-network to minimize the loss function to train the value network, that is: ； Among them, represents the value network loss; S46. Randomly initialize the parameters of the entire deep reinforcement learning framework and use the Adam optimizer to update the weights of each module.

8. The mobile robot obstacle avoidance motion planning method according to claim 7, wherein The policy network includes an encoding module, a pulse neural network module and a decoding module; The value network is composed of two MLP instances with a final output dimension of 1, and each MLP instance is composed of three fully connected layers.

Images