Obstacle avoidance motion planning method for mobile robots based on pulse hybrid reinforcement learning

Abstract

The present invention relates to the technical field of obstacle avoidance motion planning for mobile robots. It addresses the technical issues of excessive energy consumption, insufficient autonomous navigation capabilities, and insufficient obstacle avoidance performance faced by mobile robots during flexible obstacle avoidance motion planning. In particular, it relates to a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning. By combining the low energy consumption characteristics of pulse neural networks with the flexible decision-making mechanism of reinforcement learning, the mobile robot can autonomously generate an optimal path in a complex environment while effectively reducing computing costs and energy waste. The present invention can effectively improve the obstacle avoidance performance of a mobile robot in a dynamic environment, enabling it to achieve safe and efficient autonomous navigation in complex and changing environments. It enables the mobile robot to maintain excellent performance while reducing energy consumption, thereby promoting the widespread application and practical application of intelligent equipment in the field of intelligent manufacturing.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle avoidance motion planning for mobile robots, and in particular to a method for obstacle avoidance motion planning for mobile robots based on pulse hybrid reinforcement learning. Background Art

[0002] As mobile robots become increasingly common in real-world applications, their safety and efficiency in complex environments are attracting increasing attention. Especially in scenarios where they operate alongside dynamic, non-networked agents (such as pedestrians or other robots), effectively avoiding collisions with these objects has become a key factor in ensuring safe and efficient robot operation. In densely populated and dynamically changing environments, robots must work in close coordination with humans and other robots.

[0003] Current robot obstacle avoidance technology requires robots to perform a series of decision-making and execution processes, including environment mapping, path planning, and motion control. Commonly used methods include the A* algorithm, Dijkstra's algorithm, artificial potential field method, and dynamic window method (DWA). However, mobile robots face high energy consumption during flexible obstacle avoidance motion planning, and their autonomous navigation and obstacle avoidance capabilities in complex dynamic environments are insufficient. This makes it difficult for mobile robots to achieve safe and efficient autonomous navigation in complex and changing environments. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning, which solves the technical problems faced by mobile robots in the process of flexible obstacle avoidance motion planning, such as excessive energy consumption, insufficient autonomous navigation ability and obstacle avoidance performance.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning, the method comprising the following steps:

[0006] S1. During any interaction with the environment, the mobile robot obtains observation information of each scene, including observation values ​​and action space, from the sensor.

[0007] S2. Establish a hybrid pulse reinforcement learning model that uses observation information as input to enhance the mobile robot's perception of environmental information in a dynamic environment and outputs the action instructions required for the mobile robot to avoid obstacles in the dynamic environment;

[0008] S3. Establishing a reward mechanism for guiding the path planning training of the mobile robot when the mobile robot executes action instructions;

[0009] S4. Optimize the parameters of the hybrid pulse reinforcement learning model to obtain the optimal motion path for guiding the mobile robot to complete the obstacle avoidance task.

[0010] The present invention utilizes a group of pulse neural networks as a strategy network, so that the mobile robot can perceive changes in the surrounding environment more accurately and make quick and reasonable decisions based on these changes. In terms of model design, the present invention adopts a group pulse coding method. Through a Gaussian distributed encoder, the observation information obtained by the mobile robot from the sensor is converted into a pulse sequence. After these pulse sequences are processed by a multi-layer pulse neural network, they are decoded into continuous action instructions. This encoding and decoding method can make full use of the timing processing capabilities of the pulse neural network, making the mobile robot more sensitive to environmental information in a dynamic environment. Through the above design, the mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning proposed by the present invention can effectively improve the obstacle avoidance performance of the mobile robot in a dynamic environment, so that it can achieve safe and efficient autonomous navigation in a complex and changeable environment.

[0011] Furthermore, the hybrid pulse reinforcement learning model includes:

[0012] A population encoding module that maps observation information into activity activation values ​​of multi-channel neurons and generates a pulse train with a fixed time step;

[0013] The spiking neural network module takes a pulse train as input, simulates the changes in biological neuron membrane potential, synaptic current accumulation, and pulse triggering mechanism, and outputs encoded activity;

[0014] and, a decoding module that converts the encoded activity output by the spiking neural network module into a more intuitive action space representation.

[0015] Furthermore, the observation value: in the task of training obstacle avoidance, each time step The mobile robot information and obstacle information are also included;

[0016] The action space: at each time step Define the robot arm action , respectively Speed ​​information in direction.

[0017] Furthermore, the encoding module includes:

[0018] For each input dimension The observation value of constructs multiple receptive field neurons, and the input observation value is , then The response value of a neuron Expressed as:

[0019] ;

[0020] in, is the receptive field center uniformly set in the input space; is the standard deviation, which is used to determine the activity intensity of neurons;

[0021] Subsequently, these response values It is input into a voltage accumulator for time series integration. When the membrane potential exceeds the firing threshold When , the neuron emits a pulse at this time step, and the iterative process is as follows:

[0022] ;

[0023] in, For the time step The voltage value at For the time step The pulse value released when Is a judgment function, if the input neuron voltage Exceeding the threshold , then it takes 1, otherwise it takes 0.

[0024] Furthermore, the pulse neural network module includes:

[0025] Introduce the pseudo-derivative mechanism, that is, in the interval The internal gradient is artificially assigned non-zero, that is:

[0026] ;

[0027] in, represents the neuron voltage;

[0028] By calculating the change in neuronal membrane voltage, the current The update rules are as follows:

[0029] ;

[0030] in, is the attenuation factor of the current; is the weight of the linear layer;

[0031] After introducing the learnable membrane time constant, the neuron membrane potential The update rules are as follows:

[0032] ;

[0033] in, , is the sigmoid activation function; express The voltage value triggered at the moment; Indicates external input; is the reset voltage;

[0034] Any pulse neuron pulse signal The update rules are as follows:

[0035] ;

[0036] in, Indicates processing each element of the input vector, if it exceeds the threshold , then take 1, otherwise take 0;

[0037] After the pulse is released, the membrane potential of the neuron is reset. Updates as follows:

[0038] ;

[0039] in, is the voltage attenuation factor.

[0040] Furthermore, the pulse neural network module includes:

[0041] For each action dimension , and its corresponding pulse group will be linearly integrated and nonlinearly activated, i.e.:

[0042] ;

[0043] in, is the convolution weight vector on this dimension; is the Tanh activation function.

[0044] Furthermore, in step S3, the specific process includes:

[0045] If the mobile robot moves towards the target, a positive reward is given in proportion to the distance reduction. , the specific update rules are as follows:

[0046] ;

[0047] in, Indicates Time step, the Euclidean distance from the mobile robot to the target;

[0048] If the mobile robot is found to be getting closer and closer to an obstacle, a gradual penalty will be given. , the specific formula is as follows:

[0049] ;

[0050] in, Indicates the minimum distance between the mobile robot and the obstacle;

[0051] If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty will be given. And end the round, that is:

[0052] ;

[0053] in, 、 are the sizes of the mobile robot and the obstacle respectively;

[0054] If the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large one-time reward will be given. And terminate the task, that is:

[0055] ;

[0056] in, Indicates the range of the target point;

[0057] Introducing time penalties , and based on positive rewards ,punish ,punish And large rewards , obtain the total reward for guiding the mobile robot path planning training , the expression is:

[0058] ;

[0059] in, Represents the total reward under the reward mechanism.

[0060] Furthermore, in step S4, the specific process includes the following steps:

[0061] S41. Establishing a deep reinforcement learning framework for optimizing parameters of a hybrid pulse reinforcement learning model based on a maximum entropy reinforcement learning algorithm, wherein the deep reinforcement learning framework includes a policy network for generating a probability distribution for determining actions of the mobile robot and a value network for evaluating the value of state-action pairs;

[0062] S42. Minimizing the policy loss and Q-value loss to train the policy network and value network as the training goal of the deep reinforcement learning framework;

[0063] S43. Based on the training objective, the goal of optimizing the strategy loss is to maximize the expected return of the strategy and minimize the entropy of the strategy. The expression is:

[0064] ;

[0065] in, Indicates that samples are randomly drawn from the experience replay pool D; represents the entropy regularization coefficient, which is used to balance exploration and utilization; Indicates status and actions of value, and are two different value estimates of the same state-action pair by the value network;

[0066] S44. Calculating the prediction of future rewards using a hybrid pulse reinforcement learning model based on a reward mechanism ,Right now:

[0067] ;

[0068] in, Indicates the reward obtained for the current state and action; represents the discount factor, which determines the impact of future rewards; Indicates that under the current strategy, the action In state The logarithmic probability under is used to encourage the mobile robot to explore more;

[0069] S45. Based on predictions of future rewards , use the double Q network to minimize the loss function to train the value network, that is:

[0070] ;

[0071] in, represents the value network loss;

[0072] S46. Randomly initialize the parameters of the entire deep reinforcement learning framework and use the Adam optimizer to update the weights of each module.

[0073] Furthermore, the strategy network includes an encoding module, a pulse neural network module and a decoding module;

[0074] The value network consists of two MLP instances with a final output dimension of 1, and each MLP instance consists of three fully connected layers.

[0075] By means of the above technical solution, the present invention provides a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning, which has at least the following beneficial effects:

[0076] 1. The present invention can effectively improve the obstacle avoidance performance of mobile robots in dynamic environments, enabling them to achieve safe and efficient autonomous navigation in complex and changing environments. It can also enable mobile robots to maintain excellent performance while reducing energy consumption, thereby promoting the widespread application and practical application of intelligent equipment in the field of intelligent manufacturing.

[0077] 2. This invention combines the low energy consumption of spiking neural networks with the flexible decision-making mechanisms of reinforcement learning, enabling mobile robots to autonomously generate optimal paths in complex environments while effectively reducing computational costs and energy waste. This reduces energy consumption while maintaining excellent performance, thereby promoting the widespread application and practical application of intelligent equipment in the field of intelligent manufacturing.

[0078] 3. By cleverly combining the efficient energy utilization characteristics of pulse neural networks with the dynamic decision-making capabilities of reinforcement learning, the present invention enables mobile robots to autonomously plan optimal paths in complex environments while reducing unnecessary calculations, thereby significantly reducing energy consumption during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0080] Figure 1 Flowchart of the obstacle avoidance motion planning method for a mobile robot in the present invention;

[0081] Figure 2 A flow chart for generating action instructions for the hybrid pulse reinforcement learning model of the present invention;

[0082] Figure 3 Schematic diagram of the simulation of the obstacle avoidance motion planning of the mobile robot in the present invention. DETAILED DESCRIPTION

[0083] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0084] Researching robot obstacle avoidance technology not only holds important theoretical significance but also has broad practical application value. First, it is a key technology for achieving robot autonomy and intelligence, enabling robots to complete path planning and real-time dynamic adjustments without human intervention, thereby improving the flexibility and stability of their task execution. Second, the maturity of obstacle avoidance technology will greatly expand the application scenarios of robots. For example, home service robots need to avoid furniture and people, and in warehousing and logistics, they need to efficiently shuttle between shelves. In extreme environments such as agriculture, healthcare, construction, and even disaster relief, highly reliable obstacle avoidance capabilities are even more necessary to ensure task completion and personnel safety. Therefore, in-depth research on robot obstacle avoidance technology is not only a manifestation of technological progress but also a key path to achieving higher levels of human-machine collaboration and social automation in the intelligent era.

[0085] Spiking neural networks (SNNs), as third-generation artificial neural networks, have attracted widespread attention for their information processing methods that more closely resemble biological neural systems. Unlike traditional neural networks, SNNs transmit information through discrete pulse signals, which exhibit temporal dynamics and sparse computational advantages, resulting in significant advantages in power consumption and computational efficiency. Because pulses occur only when necessary, SNNs can significantly reduce redundant computation while maintaining performance, making them particularly suitable for deployment in resource-constrained edge devices or low-power scenarios. Furthermore, SNNs possess a natural sensitivity to temporal information, making them effective for tasks with strong temporal correlation, such as sound recognition, event detection, and dynamic visual processing. Furthermore, they respond more quickly to input changes and possess a certain degree of robustness and generalization, offering new opportunities for building more intelligent, efficient, and energy-efficient neural network systems. With the development of neuromorphic hardware, SNNs have also shown great potential in areas such as robotic control, neuroscience modeling, and brain-inspired computing.

[0086] This embodiment combines reinforcement learning with spiking neural networks, showing significant advantages. Reinforcement learning provides the system with the ability to dynamically adjust its strategy, enabling it to autonomously learn optimal behavior based on environmental feedback, while spiking neural networks simulate the working mechanism of biological neurons in an efficient way of time and energy utilization, improving computational efficiency and robustness. This combination not only speeds up learning and improves the adaptability and stability of the system, but also significantly reduces energy consumption and enhances resistance to noise and interference. At the same time, its learning mechanism, which is closer to biological intelligence, provides new ideas for achieving more advanced artificial intelligence, and has important research value and broad application prospects.

[0087] Based on the spiking neural network (SNN), this embodiment proposes a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning. By introducing a low-energy spiking neural network into the policy network, while retaining the traditional deep neural network structure in the value network, the mobile robot can maintain excellent performance while reducing energy consumption, thereby promoting the widespread application and practical application of intelligent equipment in the field of intelligent manufacturing. Figure 1 As shown, the method includes the following steps:

[0088] S1. During any interaction with the environment, the mobile robot obtains observation information from sensors for each step of the scene, including observation values ​​and action space.

[0089] Observation: In the obstacle avoidance task, at each time step The observation information includes both the mobile robot information and the obstacle information. Specifically, it includes the mobile robot position, mobile robot speed, target position, and obstacle position. So the observation value is used Dimensional data representation, Indicates the number of obstacles.

[0090] Action space: The action space is related to the joint velocity control of the mobile robot. At each time step Define the robot arm action , respectively Speed ​​information in direction.

[0091] S2. Establish a hybrid pulse reinforcement learning model that uses observation information as input to enhance the mobile robot's perception of environmental information in a dynamic environment and outputs the action instructions required for the mobile robot to avoid obstacles in a dynamic environment. The hybrid pulse reinforcement learning model consists of an encoding module, a pulse neural network module, and a decoding module, such as Figure 2 shown.

[0092] This embodiment designs a population coding module to map the observation information into the activity activation value of multi-channel neurons and generate a pulse sequence with a fixed time step. The observation value of constructs multiple receptive field neurons, whose response function follows Gaussian distribution. Suppose the input observation value is , Indicates the dimension Observed values The set of The response value of a neuron It can be expressed as:

[0093] ;

[0094] in, is the receptive field center uniformly set in the input space; is the standard deviation, which is used to determine the activity strength of the neuron. The choice of Gaussian distribution means that the response of the neuron to different input features is based on probability. Specifically, the response of the neuron to the observation value will follow the pattern of normal distribution.

[0095] Subsequently, these response values It is input into a voltage accumulator for time series integration. When the membrane potential exceeds the firing threshold When , the neuron emits a pulse at this time step. The iterative process is as follows:

[0096] ;

[0097] in, For the time step The voltage value at For the time step The pulse value released when Is a judgment function, if the input neuron voltage Exceeding the threshold , then it takes 1, otherwise it takes 0. In this embodiment, the threshold The value is 0.5. This mechanism ensures that neurons only generate discrete event signals under sufficient activation conditions, while maintaining high-fidelity representation capabilities for continuous inputs.

[0098] This example designs a spiking neural network module that takes a pulse sequence as input, including a spiking neural network based on the leakage-integration-discharge principle. The resulting pulse signal is used to output the resulting spike activity. The spiking neural network module simulates changes in biological neuron membrane potential, synaptic current accumulation, and spike triggering mechanisms, and outputs encoded activity.

[0099] Since the impulse function itself is not differentiable, in order to make the network trainable, a pseudo-derivative mechanism is introduced, that is, in the interval The internal gradient is artificially assigned non-zero, that is:

[0100] ;

[0101] in, Represents the neuron voltage. This mechanism allows end-to-end training using conventional gradient descent methods while maintaining the spike nature.

[0102] By calculating the change in neuronal membrane voltage, the current The update rules are as follows:

[0103] ;

[0104] in, is the attenuation factor of the current; is the weight of the linear layer;

[0105] After introducing the learnable membrane time constant, the neuron membrane potential The update rules are as follows:

[0106] ;

[0107] in, ,make sure In the experiment, is the sigmoid activation function, that is ; express The voltage value triggered at the moment; Indicates external input; is the reset voltage;

[0108] Any pulse neuron pulse signal The update rules are as follows:

[0109] ;

[0110] in, Indicates processing each element of the input vector, if it exceeds the threshold , then take 1, otherwise take 0;

[0111] After the pulse is released, the membrane potential of the neuron is reset. Updates as follows:

[0112] ;

[0113] in, is the voltage attenuation factor.

[0114] The spiking neural network module consists of three layers connected in sequence: two spiking neural layers and an output layer. The spiking neural layer converts input data into a format suitable for spiking neural network processing. The number of spiking neurons in the spiking neural layer is 256, 256, and 20, respectively.

[0115] The final output of the spiking neural network module is a group of neuron spike activities representing the action dimension. To convert it into a continuous control signal, this embodiment designs a decoder based on one-dimensional group convolution to convert the encoded activity output by the spiking neural network module into a more intuitive action space expression. For each action dimension , and its corresponding pulse group will be linearly integrated and nonlinearly activated, i.e.:

[0116] ;

[0117] in, is the convolution weight vector on this dimension, and T represents transpose; it is converted into a continuous action space through one-dimensional convolution and activation function. The Tanh activation function ensures that the amplitude of the output action is limited to the interval [-1, 1], thereby ensuring the controllability of the action. The role of the convolutional neural network is to extract effective feature information from the activity of neurons and map it to the action output of the mobile robot.

[0118] S3. Establish a reward mechanism to guide the mobile robot's path planning training when the mobile robot executes action instructions. Furthermore, this embodiment designs a detailed reward mechanism that, based on the action instructions executed by the mobile robot when avoiding obstacles, ensures that the mobile robot receives sufficient reward value when it arrives at its destination without collision. When the mobile robot collides, a certain penalty is imposed, thereby effectively guiding the mobile robot's path planning training. The overall reward mechanism formula is as follows:

[0119] ;

[0120] To encourage the mobile robot to complete the task as quickly as possible, a small time penalty is imposed on each step. , in this embodiment, it is 0.1. This embodiment provides a detailed explanation of the remaining reward items and penalty items, as follows:

[0121] If the mobile robot moves towards the target, a positive reward is given in proportion to the distance reduction. , the specific update rules are as follows:

[0122] ;

[0123] in, Indicates The Euclidean distance between the mobile robot and the target is the time step. This formula shows that the closer the mobile robot is to the target, the greater the reward value.

[0124] If the mobile robot is found to be getting closer and closer to an obstacle, a gradual penalty will be given. , the specific formula is as follows:

[0125] ;

[0126] in, This formula indicates that when the mobile robot is found to be getting closer to an obstacle, a certain penalty will be given.

[0127] If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty will be given. And end the round, that is:

[0128] ;

[0129] in, 、 are the sizes of the mobile robot and the obstacle, respectively. When the distance between the mobile robot and the obstacle is less than the sum of the sizes of the mobile robot and the obstacle, it means that a collision has occurred between the two and a fixed penalty value is given.

[0130] If the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large one-time reward will be given. And terminate the task, that is:

[0131] ;

[0132] in, Indicates the range of the target point, indicating that the mobile robot has reached the target point, the mission is completed, and a large reward is given.

[0133] S4. Optimize the parameters of the hybrid pulse reinforcement learning model to obtain an optimal motion path for guiding the mobile robot to complete the obstacle avoidance task. This embodiment designs a deep reinforcement learning framework (SAC) based on the maximum entropy reinforcement learning algorithm (Soft Actor-Critic) to optimize the parameters of the hybrid pulse reinforcement learning model to obtain an optimal motion path. This optimal motion path is used to guide the mobile robot to complete the obstacle avoidance task.

[0134] The core framework of the Deep Reinforcement Learning (SAC) framework consists of a policy network and a value network. The purpose of the policy network is to train an optimal policy that enables the mobile robot to select the optimal action in a given state. The policy network generates a probability distribution (Gaussian distribution) to determine the mobile robot's actions. The value network is used to evaluate the value of state-action pairs. The training goal of the Deep Reinforcement Learning (SAC) framework is to minimize the policy loss and Q-value loss to train the policy network and value network. Specifically:

[0135] The goal of optimizing the policy loss is to maximize the expected return of the policy and minimize the entropy of the policy (i.e., maximize exploration). The specific formula is:

[0136] ;

[0137] in, Indicates that samples are randomly drawn from the experience replay pool D; represents the strategy loss.

[0138] This embodiment trains the value network by minimizing the loss function using a double Q network. This method helps reduce overestimation bias and thus improves the stability of the algorithm. That is:

[0139] ;

[0140] in, represents the value network loss; Represents the prediction of future rewards, calculated by the target network and the policy network. and are two different value estimates of the same state-action pair by the value network. The value function is updated based on the Bellman equation, as follows:

[0141] ;

[0142] in, Indicates status and actions of value; Indicates the reward obtained for the current state and action; represents the discount factor, which determines the impact of future rewards; Represents the expectation of future actions, which are determined by the current strategy generate; Indicates that under the current strategy, the action In state The logarithmic probability under is used to encourage the mobile robot to explore more; Represents the entropy regularization coefficient, which is used to balance exploration and utilization. Future reward prediction The calculation formula is as follows:

[0143] ;

[0144] The policy network in the Deep Reinforcement Learning framework (SAC) includes the aforementioned encoder, spiking neural network, and decoder modules. The value network, on the other hand, consists of two MLP instances, each of which consists of three fully connected layers, with an input dimension of 64 + 2 = 66 and a final output dimension of 1.

[0145] Randomly initialize the parameters of the entire deep reinforcement learning framework (SAC), calculate the gradient of the actor loss with respect to the actor network parameters, and use the Adam optimizer to update the weights of the above encoding module, spiking neural network module, and decoding module.

[0146] In order to verify the effectiveness of this method, this example constructs a maze simulation environment. In this environment, the mobile robot needs to achieve autonomous navigation in the continuous action space while avoiding dynamic obstacles and reaching the target position. Figure 3 This embodiment tests different obstacle scenarios, and the results are shown in Table 1:

[0147] Table 1 Results under different obstacles

[0148]

[0149] This embodiment proposes a mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning, which greatly reduces energy consumption during system operation. By combining the low energy consumption characteristics of pulse neural networks with the flexible decision-making mechanism of reinforcement learning, the robot can autonomously generate the optimal path in a complex environment, while effectively reducing computing costs and energy waste. And the membrane time constant that can be introduced into the pulse neural network is introduced, which greatly enhances the expression ability of the network. This innovative method not only improves the robot's endurance and work efficiency, but also enhances its adaptability and robustness in complex environments. In this way, this application aims to provide a new solution for the autonomy and energy efficiency of mobile robots in practical applications, promote the widespread application of robotics technology in logistics, rescue, exploration and other fields, and also provide new ideas and theoretical support for the design and development of intelligent robot systems.

[0150] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the same or similar parts between the embodiments. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiments.

[0152] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A mobile robot obstacle avoidance motion planning method based on pulse hybrid reinforcement learning, characterized in that: The method comprises the following steps: S1. During any interaction with the environment, the mobile robot obtains observation information of each scene, including observation values ​​and action space, from the sensor. S2. Establishing a hybrid pulse reinforcement learning model that uses observation information as input to enhance the mobile robot's perception of environmental information in a dynamic environment and outputs action instructions required for the mobile robot to avoid obstacles in the dynamic environment; the hybrid pulse reinforcement learning model includes: A population encoding module that maps observation information into activity activation values ​​of multi-channel neurons and generates a pulse train with a fixed time step; The spiking neural network module takes a pulse train as input, simulates the changes in biological neuron membrane potential, synaptic current accumulation, and pulse triggering mechanism, and outputs encoded activity; and, a decoding module that converts the encoded activity output by the spiking neural network module into a more intuitive representation of the action space; S3. Establish a reward mechanism for guiding the path planning training of the mobile robot when the mobile robot executes action instructions. The specific process includes: If the mobile robot moves towards the target, a positive reward is given in proportion to the distance reduction. , the specific update rules are as follows: ； in, Indicates Time step, the Euclidean distance from the mobile robot to the target; If the mobile robot is found to be getting closer and closer to an obstacle, a gradual penalty will be given. , the specific formula is as follows: ； in, Indicates the minimum distance between the mobile robot and the obstacle; If the distance between the mobile robot and the obstacle is less than the set obstacle radius, that is, the mobile robot collides with the obstacle, a penalty will be given. And ends with: ； in, 、 are the sizes of the mobile robot and the obstacle respectively; If the distance between the mobile robot and the target is less than the target range, that is, the mobile robot successfully reaches the target point, a large one-time reward will be given. And terminate the task, that is: ； in, Indicates the range of the target point; Introducing time penalties , and based on positive rewards ,punish ,punish And large rewards , obtain the total reward for guiding the mobile robot path planning training , the expression is: ； in, Represents the total reward under the reward mechanism; S4. Optimize the parameters of the hybrid pulse reinforcement learning model to obtain the optimal motion path for guiding the mobile robot to complete the obstacle avoidance task. The specific process includes the following steps: S41. Establishing a deep reinforcement learning framework for optimizing parameters of a hybrid pulse reinforcement learning model based on a maximum entropy reinforcement learning algorithm, wherein the deep reinforcement learning framework includes a policy network for generating a probability distribution for determining actions of the mobile robot and a value network for evaluating the value of state-action pairs; S42. Minimizing policy loss and Q-value loss to train the policy network and value network as the training goal of the deep reinforcement learning framework; S43. Based on the training objective, the goal of optimizing the strategy loss is to maximize the expected return of the strategy and minimize the entropy of the strategy. The expression is: ； in, Indicates that samples are randomly drawn from the experience replay pool D; represents the entropy regularization coefficient, which is used to balance exploration and utilization; Indicates status and actions of value, and are two different value estimates of the same state-action pair by the value network; S44. Calculating the prediction of future rewards using a hybrid pulse reinforcement learning model based on a reward mechanism ,Right now: ； in, Indicates the reward obtained for the current state and action; represents the discount factor, which determines the impact of future rewards; Indicates that under the current strategy, the action In state The logarithmic probability under is used to encourage the mobile robot to explore more; S45. Based on predictions of future rewards , use the double Q network to minimize the loss function to train the value network, that is: ； in, represents the value network loss; S46. Randomly initialize the parameters of the entire deep reinforcement learning framework and use the Adam optimizer to update the weights of each module.

2. The mobile robot obstacle avoidance motion planning method according to claim 1, characterized in that: In step S1: The observation value: In the task of training obstacle avoidance, each time step The mobile robot information and obstacle information are also included; The action space: at each time step Define the robot arm action , respectively Speed ​​information in direction.

3. The mobile robot obstacle avoidance motion planning method according to claim 1, characterized in that: The encoding module includes: For each input dimension The observation value of constructs multiple receptive field neurons, and the input observation value is , then The response value of a neuron Expressed as: ； in, is the receptive field center uniformly set in the input space; is the standard deviation, which is used to determine the activity intensity of neurons; Subsequently, these response values It is input into a voltage accumulator for time series integration. When the membrane potential exceeds the firing threshold When , the neuron emits a pulse at this time step, and the iterative process is as follows: ； in, For the time step The voltage value at For the time step The pulse value released when Is a judgment function, if the input neuron voltage Exceeding the threshold , then it takes 1, otherwise it takes 0.

4. The mobile robot obstacle avoidance motion planning method according to claim 3, characterized in that: The pulse neural network module includes: Introduce the pseudo-derivative mechanism, that is, in the interval The internal gradient is artificially assigned non-zero, that is: ； in, represents the neuron voltage; By calculating the change in neuronal membrane voltage, the current The update rules are as follows: ； in, is the attenuation factor of the current; is the weight of the linear layer; After introducing the learnable membrane time constant, the neuron membrane potential The update rules are as follows: ； in, , is the sigmoid activation function; express The voltage value triggered at the moment; Indicates external input; is the reset voltage; Any pulse neuron pulse signal The update rules are as follows: ； in, Indicates processing each element of the input vector, if it exceeds the threshold , then take 1, otherwise take 0; After the pulse is released, the membrane potential of the neuron is reset. Updates as follows: ； in, is the voltage attenuation factor.

5. The mobile robot obstacle avoidance motion planning method according to claim 4, characterized in that: The pulse neural network module includes: For each action dimension , and its corresponding pulse group will be linearly integrated and nonlinearly activated, i.e.: ； in, is the convolution weight vector on this dimension; is the Tanh activation function.

6. The mobile robot obstacle avoidance motion planning method according to claim 1, characterized in that: The strategy network includes an encoding module, a pulse neural network module and a decoding module; The value network consists of two MLP instances with a final output dimension of 1, and each MLP instance consists of three fully connected layers.

Images