Obstacle avoidance method and device based on brain-like pulse reinforcement learning

Abstract

The invention provides an obstacle avoidance method and device based on brain-like pulse reinforcement learning, and relates to the technical field of automation and intelligent sensing, and the method comprises the steps: carrying out the preprocessing of the linear speed, angular speed, target distance, direction and radar data of a robot, inputting the data into a pulse neural network, and obtaining the motion of the robot; wherein reinforcement learning DDPG is used to train the spiking neural network, a commentator network is used to optimize parameters, and a trained spiking neural network model is obtained if a loss value is smaller than a certain threshold value. In the pulse neural network, target data and self data are processed by one full connection layer, radar data are processed by the other full connection layer, fusion features output by the two full connection layers are weighted, and then robot actions are obtained through the full connection layers. According to the invention, the graph-free obstacle avoidance navigation of the robot in a static complex scene is realized, and the navigation performance is improved.

Description

Technical Field

[0001] The present invention belongs to the field of automation and intelligent sensing technology, and in particular relates to an obstacle avoidance method and device based on brain-like pulse reinforcement learning. Background Art

[0002] As the complexity of mobile robots exploring unknown environments increases, being able to navigate autonomously becomes increasingly important. Traditionally, navigation relies on global knowledge in the form of a map of the environment. However, for many applications that require effective navigation, building informative maps is prohibitively expensive due to timeliness and energy consumption. Deep reinforcement learning (DRL) methods, such as deep deterministic policy gradients (DDPG), make it possible to learn optimal control policies for map-less navigation, but their optimality comes at the cost of high energy consumption. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides an obstacle avoidance method and device based on brain-like pulse reinforcement learning, which can effectively improve the success rate of autonomous navigation in complex environments without maps, thereby enhancing the adaptability and safe driving capabilities of intelligent vehicles.

[0004] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0005] In one aspect, the present invention provides an obstacle avoidance method based on brain-like pulse reinforcement learning, comprising:

[0006] Collecting the robot's linear velocity, angular velocity, target distance, direction and radar data, and performing preprocessing, wherein the preprocessing includes taking the linear velocity, angular velocity, target distance and direction as target data and normalizing them, and simplifying the radar data;

[0007] Input all pre-processed data into a trained spiking neural network model for decision making to obtain the robot's actions; wherein the spiking neural network model is trained based on reinforcement learning DDPG, and the reinforcement learning DDPG is used to learn action strategies in different environments;

[0008] The pulse neural network model includes a first sub-network and a second sub-network. The first sub-network receives pre-processed target data and extracts target features, and the second sub-network receives pre-processed radar data and extracts environmental features.

[0009] Furthermore, the pulse neural network model also includes a fully connected layer and an attention layer; the target features extracted by the first sub-network and the environmental features extracted by the second sub-network are fused and weighted by the attention layer, and then the action decision results of the robot are output through the fully connected layer; wherein, the first and second sub-networks have the same structure, both of which are two fully connected layers connected in series.

[0010] Furthermore, the spiking neural network model is trained according to the following steps:

[0011] The preprocessed data is input into the spiking neural network model to obtain the robot's movements;

[0012] The reward value of the reinforcement learning DDPG is calculated based on the robot's state and action, input into the critic network, and the output result is negative to get the loss value;

[0013] Performing reverse iterative update on the spiking neural network model using the loss value;

[0014] When the updated loss value is less than the target threshold, the trained spiking neural network model is obtained.

[0015] Further, the reverse iterative updating of the spiking neural network model using the loss value includes: determining a replacement gradient according to the loss value and a gradient function;

[0016] The spiking neural network model is updated in reverse iteratively using the substitute gradient.

[0017] Furthermore, the simplified processing of the radar data includes: merging the 180-dimensional radar data collected by the robot, merging every 10 dimensions, and after obtaining 18-dimensional data, taking the median of the data in each new dimension, dividing the radar minimum scanning distance by the median of each new dimension, and mapping the radar data of each new dimension to between 0 and 1.

[0018] Furthermore, the target data is randomly generated using a script, and the radar data is acquired using a simulation environment.

[0019] Furthermore, the simulation environment is Ros + Gazebo.

[0020] On the other hand, the present invention provides an obstacle avoidance device based on brain-like pulse reinforcement learning, comprising:

[0021] The acquisition module collects the robot's linear velocity, angular velocity, target distance, direction and radar data and performs preprocessing, wherein the preprocessing includes taking the linear velocity, angular velocity, target distance and direction as target data and normalizing them, and simplifying the radar data;

[0022] A control module is used to input all pre-processed data into a trained spiking neural network model for decision-making to obtain the robot's actions; wherein the spiking neural network model is trained based on reinforcement learning DDPG, and the reinforcement learning DDPG is used to learn action strategies in different environments;

[0023] The pulse neural network model includes a first sub-network and a second sub-network. The first sub-network receives pre-processed target data and extracts target features, and the second sub-network receives pre-processed radar data and extracts environmental features.

[0024] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned obstacle avoidance method based on brain-like pulse reinforcement learning when executing the computer program.

[0025] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned obstacle avoidance method based on brain-like pulse reinforcement learning.

[0026] The obstacle avoidance method and device based on brain-like pulse reinforcement learning provided by the present invention input the robot radar data and target data into the improved pulse neural network for calculation, wherein the first and second fully connected layers in the pulse neural network model extract target features and environmental features respectively, which effectively improves the success rate of autonomous navigation when facing complex environments without maps, thereby enhancing the adaptability and safe driving capabilities of intelligent vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the obstacle avoidance method based on brain-like pulse reinforcement learning of the present invention;

[0028] Figure 2 It is a structural schematic diagram of the pulse neural network model of the present invention;

[0029] Figure 3 It is a schematic diagram of the structure of the fully connected layer of the present invention;

[0030] Figure 4 Schematic diagram of the working principle of the pulse neural network model of the present invention;

[0031] Figure 5 A schematic diagram of the process of training a spiking neural network model according to the present invention;

[0032] Figure 6 It is a schematic diagram of the structure of the obstacle avoidance device based on brain-like pulse reinforcement learning of the present invention;

[0033] Figure 7 It is a schematic structural diagram of the electronic device of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, the present invention provides an obstacle avoidance method based on brain-like pulse reinforcement learning, comprising the following steps:

[0036] Step 110, collecting the robot's linear velocity, angular velocity, target distance, direction and radar data, and performing preprocessing;

[0037] Step 120, input the preprocessed data into the trained spiking neural network model for decision making to obtain the robot's actions.

[0038] Wherein, the spiking neural network model is trained based on reinforcement learning DDPG, and the reinforcement learning DDPG is used to learn action strategies in different environments;

[0039] Specifically, after specifying the target through the Ros interface or command line, the program obtains the target data through Ros. The target data indicates the distance and direction of the target. The radar data is obtained by the acquisition module, indicating whether there are obstacles in a certain direction. Through reinforcement learning, the best control strategy is obtained to avoid obstacles when moving towards the target.

[0040] The collected radar data are simplified to obtain simplified radar data.

[0041] The obstacle avoidance method based on brain-like pulse reinforcement learning provided by the present invention is calculated by inputting robot radar data and target data into a modified pulse neural network. The two sub-networks in the improved pulse neural network model are used to extract target features and environmental features. It also includes an attention layer, which is used to learn the weights when fusion features are used, so that the model can learn the decision-making methods of obstacle avoidance and advancement to achieve better navigation success rate and effect. The power consumption of the spiking neural network SNN (Spiking Neural Network, SNN) is lower than that of ANN (Artificial Neural Network, ANN), so it is suitable for deployment in robot systems. When facing complex environments without maps, it effectively improves the success rate of autonomous navigation, thereby enhancing the adaptability and safe driving capabilities of intelligent vehicles.

[0042] In some embodiments, the target data is randomly generated using a script, and the radar data is acquired using a simulation environment, which is Ros + Gazebo.

[0043] Specifically, the robot used in the real scene is Turtlebot3-burger.

[0044] like Figure 2 As shown in the figure, the improved pulse neural network model includes at least three fully connected layers and one attention layer. The pre-processed data is input into the network and first divided into target and self-state data (i.e., the robot's linear speed, angular speed, target distance, direction) and radar data, and then sent to two sub-networks. Each sub-network includes at least two fully connected layers. After extracting features, they are weighted based on the learned weights based on the attention layer, and finally the robot's action, i.e., the left and right wheel speeds, is obtained through the fully connected layer.

[0045] Figure 3 is a schematic diagram of the structure of the fully connected layer provided by the present invention, such as Figure 3 As shown in the figure, since the input data (the spike signal of the nth layer) is one-dimensional, a one-dimensional fully connected layer (1DFC) is used here. The one-dimensional fully connected layer is followed by a leaky integrate-and-fire (LIF) spike neuron.

[0046] The activation function of each fully connected layer is replaced by LIF spiking neurons for data nonlinear operations. The output of the fully connected layer is sent to the LIF spiking neurons and converted into a pulse sequence. This process not only effectively extracts global timing features, but also significantly reduces energy consumption.

[0047] The current equation of the LIF spiking neuron is expressed as:

[0048] ,

[0049] In the formula, is the current through the LIF pulse neuron at time t, is the current through the resistor R, is the current passing through capacitor C.

[0050] According to the above formula, we can deduce:

[0051] ,

[0052] In the formula, is the voltage across the LIF pulse neuron at time t, is the resting potential, is the membrane potential at t = 0, is the membrane time constant, , is the resistance value of resistor R, is the capacitance value of capacitor C.

[0053] Figure 4 Schematic diagram of the working principle of the pulse neural network model provided by the present invention. Figure 4 As shown, the process of obtaining the robot action is as follows:

[0054] (1) Input the target and its own state data into the first sub-network to extract target features, and input the radar data into the second sub-network to extract environmental features;

[0055] (2) Combine the extracted target features and environmental features and input them into the attention layer;

[0056] (3) The merged features are weighted through the attention layer, and the weighted features are input into the fully connected layer to obtain the pulse data;

[0057] (4) Decode the pulse data. First, calculate the average number of pulses emitted. According to the maximum speed of the robot, calculate the speed of the robot's left and right wheels, that is:

[0058] ,

[0059] ,

[0060] in, Indicates the speed of the robot's right wheel, Indicates the robot's left wheel speed, is the maximum speed of the robot, is the minimum speed of the robot, and Spikecount represents an array of the average number of pulses emitted.

[0061] like Figure 5 As shown, in some embodiments, the brain-like pulse reinforcement learning obstacle avoidance method provided by the present invention also includes:

[0062] Initialize the network parameters (corresponding to the neural network parameters) of the pulse neural network model;

[0063] The preprocessed fusion data is input into the pulse neural network model to obtain the robot action value, where the action value is the action that the model considers to have the greatest value in the current state;

[0064] The current state and the current output action value are input into the critic network model, and the output value is negative as the loss value;

[0065] The spiking neural network model is updated iteratively in reverse order using the loss value to update the loss value;

[0066] When the updated loss value is less than the target threshold, the trained spiking neural network model is obtained.

[0067] Specifically,

[0068] The current state and the output action value of the spiking neural network model are input into the critic network model. The critic network model evaluates and obtains the Q value. The loss value of the spiking neural network is as follows:

[0069] ,

[0070] In the formula, is the loss value, and Q is the output value of the critic network model.

[0071] In some embodiments, the improved spiking neural network model is reversely iterated and updated using the loss value, including:

[0072] Determine the alternative gradient based on the loss value and the gradient function;

[0073] The improved spiking neural network model is updated iteratively in reverse using alternative gradients.

[0074] Specifically, the expression of the gradient function is as follows:

[0075] ,

[0076] In the formula, is the gradient function, is the learning rate, is the loss value;

[0077] Substitute the loss value into the gradient function to obtain the substitute gradient, and use the substitute gradient to perform reverse iterative update on the spiking neural network model.

[0078] The obstacle avoidance method based on brain-like pulse reinforcement learning provided by the present invention determines the substitute gradient through the loss value and the gradient function, and uses the substitute gradient to perform reverse iterative update on the pulse neural network model, thereby further improving the navigation performance of the pulse neural network model.

[0079] In some embodiments, before the pre-processed fusion data is input into the improved spiking neural network, the method further includes:

[0080] The target data is normalized and the radar data is simplified.

[0081] Specifically, in the target data, two dimensions are used to represent the target direction. If the target is in front, the first dimension is 0, and the second dimension is the target direction angle. Otherwise, the first dimension is the absolute value of the target direction angle, and the second dimension is 0. For the target distance, if it is less than the minimum target distance, it is considered that the car has reached the target. The original target distance is divided by the minimum target distance, and the original target distance is mapped to between 0 and 1.

[0082] The data itself includes angular velocity and linear velocity. The angular velocity and linear velocity are normalized by dividing them by their maximum values, thereby mapping them to between 0 and 1.

[0083] For radar data, the radar data collected by the 180-dimensional robot is merged, and the data is merged every 10 dimensions. After obtaining 18-dimensional data, the median of the data in each dimension is taken. The minimum scanning distance of the radar is divided by the median data of each dimension, and the radar data of each dimension is mapped to between 0 and 1.

[0084] After all the data are mapped to between 0 and 1, pulse encoding is performed. For each time step, a matrix is ​​generated using random numbers, and the matrix is ​​compared with the mapped data. At the same position in the two-dimensional matrix, if the mapped data is smaller than the random number matrix, the output matrix at that position is 0, otherwise it is 1. The random number data of the matrix is ​​converted into a 01 matrix according to the comparison result as the input of the pulse neural network model.

[0085] The obstacle avoidance method based on brain-like pulse reinforcement learning provided by the present invention ensures the consistency of data by normalizing and pulse encoding the fused data, thereby providing a more stable foundation for improving the pulse neural network. These preprocessing steps help eliminate noise and outliers in the data, prevent abnormal data points from adversely affecting model training, and also speed up the learning process.

[0086] The obstacle avoidance method based on brain-like pulse reinforcement learning provided by the present invention improves the navigation performance of the model in complex scenarios by segmenting different data to extract features and weight attention separately.

[0087] The robot obstacle avoidance device based on brain-like pulse reinforcement learning provided by the present invention is described below. The robot obstacle avoidance device based on brain-like pulse reinforcement learning described below and the obstacle avoidance method based on brain-like pulse reinforcement learning described above can be referenced to each other.

[0088] Figure 6 is a schematic diagram of the structure of the obstacle avoidance device based on brain-like pulse reinforcement learning provided by the present invention, such as Figure 6 As shown, the present invention provides an obstacle avoidance device based on brain-like pulse reinforcement learning, comprising:

[0089] The acquisition module 610 is used to acquire the robot's linear velocity, angular velocity, target distance, direction and radar data.

[0090] The calculation module 620 is used to input the pre-processed fusion data into the trained spiking neural network model for calculation to obtain the robot action;

[0091] Among them, the pulse neural network model has two sub-networks, which are trained based on reinforcement learning DDPG. The reinforcement learning DDPG is used to learn action strategies in different environments, and the two sub-networks are used to extract target features and environmental features.

[0092] In some embodiments, the spiking neural network model includes at least five fully connected layers and one attention layer;

[0093] The signal input into the improved pulse neural network model is first sliced, the target data is input into two fully connected layers, the radar data is input into two fully connected layers, the extracted features are merged and input into the attention layer, and finally the result is obtained through another fully connected layer;

[0094] Among them, the attention layer weights the input fusion features.

[0095] In some embodiments, the target data is randomly generated using a script, and the radar data is acquired using a simulation environment, which is Ros + Gazebo.

[0096] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the aforementioned obstacle avoidance method based on brain-like pulse reinforcement learning;

[0097] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0098] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned obstacle avoidance method based on brain-like pulse reinforcement learning.

[0099] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0100] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An obstacle avoidance method based on brain-like pulse reinforcement learning, characterized in that: include: Collecting the robot's linear velocity, angular velocity, target distance, direction and radar data, and performing preprocessing, wherein the preprocessing includes taking the linear velocity, angular velocity, target distance and direction as target data and normalizing them, and simplifying the radar data; Input all pre-processed data into a trained spiking neural network model for decision making to obtain the robot's actions; wherein the spiking neural network model is trained based on reinforcement learning DDPG, and the reinforcement learning DDPG is used to learn action strategies in different environments; The pulse neural network model includes a first sub-network and a second sub-network. The first sub-network receives pre-processed target data and extracts target features, and the second sub-network receives pre-processed radar data and extracts environmental features.

2. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 1, characterized in that: The pulse neural network model also includes a fully connected layer and an attention layer; the target features extracted by the first sub-network and the environmental features extracted by the second sub-network are fused and weighted by the attention layer, and then the action decision results of the robot are output through the fully connected layer; wherein, the first and second sub-networks have the same structure, both of which are two fully connected layers connected in series.

3. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 1, characterized in that: The spiking neural network model is trained according to the following steps: The preprocessed data is input into the spiking neural network model to obtain the robot's movements; The reward value of the reinforcement learning DDPG is calculated based on the robot's state and action, input into the critic network, and the output result is negative to get the loss value; Performing reverse iterative update on the spiking neural network model using the loss value; When the updated loss value is less than the target threshold, the trained spiking neural network model is obtained.

4. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 3 is characterized in that: The reverse iterative updating of the spiking neural network model using the loss value includes: Determine a replacement gradient according to the loss value and the gradient function; The spiking neural network model is updated in reverse iteratively using the substitute gradient.

5. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 3 is characterized in that: The simplified processing of radar data includes: merging the 180-dimensional radar data collected by the robot, merging every 10 dimensions, and after obtaining 18-dimensional data, taking the median of the data in each new dimension, dividing the radar minimum scanning distance by the median of each new dimension, and mapping the radar data of each new dimension to between 0 and 1.

6. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 1, characterized in that: The target data is randomly generated using a script, and the radar data is acquired using a simulation environment.

7. The obstacle avoidance method based on brain-like pulse reinforcement learning according to claim 6, characterized in that: The simulation environment is Ros + Gazebo.

8. An obstacle avoidance device based on brain-like pulse reinforcement learning, characterized in that: include: The acquisition module collects the robot's linear velocity, angular velocity, target distance, direction and radar data and performs preprocessing, wherein the preprocessing includes taking the linear velocity, angular velocity, target distance and direction as target data and normalizing them, and simplifying the radar data; A control module is used to input all pre-processed data into a trained spiking neural network model for decision-making to obtain the robot's actions; wherein the spiking neural network model is trained based on reinforcement learning DDPG, and the reinforcement learning DDPG is used to learn action strategies in different environments; The pulse neural network model includes a first sub-network and a second sub-network. The first sub-network receives pre-processed target data and extracts target features, and the second sub-network receives pre-processed radar data and extracts environmental features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the obstacle avoidance method based on brain-like pulse reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the obstacle avoidance method based on brain-like pulse reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images