A Robot Control Method, Electronic Device and Storage Medium Based on SAC and Neural Circuit Strategy

Abstract

A robot control method, electronic device and storage medium based on SAC and neural circuit strategy belong to the technical field of robot behavior control. To improve the processing speed and intelligent characteristics of the robot, the present invention combines robot control with the SAC algorithm in reinforcement learning, getting rid of the limitations of traditional control algorithms on the model, improving the learning speed of the robot and the utilization efficiency of empirical samples, and solving the problems that the robot is prone to fall into local optimum and unable to reasonably plan actions. By constructing a four-layer neural circuit strategy network as the control output generation network, the robot control can be efficiently and accurately completed. Compared with other current technologies, this method has lower requirements for hardware computing power, can have higher computing efficiency, and only needs a small number of neurons to achieve better results. The working principle of this method is closer to the functional principle of nerve cells, has more advanced theoretical support, and has more development potential in the field of artificial intelligence.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot behavior control, and particularly relates to a robot control method, an electronic device, and a storage medium based on SAC and neural circuit strategies. Background Art

[0002] The latest advances in the field of robotics are being applied in various industries. A key aspect of robot design is the development of control systems that can effectively handle complex and dynamic environments. Reinforcement learning has emerged as a powerful tool that enables robots to learn how to perform specific tasks by obtaining feedback from the environment.

[0003] Despite the progress made in reinforcement learning algorithms, it remains challenging to develop control systems that can robustly adapt to environmental changes and effectively execute complex tasks. Although these reinforcement learning methods have better performance than traditional networks, they often lack generalization ability and may fall into suboptimal decisions, overfitting to the training data. Summary of the Invention

[0004] The problem to be solved by the present invention is to improve the processing speed and intelligent characteristics of the robot, and a robot control method, an electronic device, and a storage medium based on SAC and neural circuit strategies are proposed.

[0005] To achieve the above object, the present invention is realized by the following technical solutions:

[0006] A robot control method based on SAC and neural circuit strategies, comprising the following steps:

[0007] S1. Construct a simulation environment, design the initial position, speed, and inclination angle of the robot, and collect the motion state data of the robot;

[0008] S2. Construct a linear layer, input the motion state data of the robot collected in step S1 into the linear layer for linear transformation to obtain a motion state feature sequence of the robot;

[0009] S3. Construct an SAC actor network based on a neural circuit policy network, input the motion state feature sequence of the robot obtained in step S2 into the SAC actor network based on the neural circuit policy network, the perception layer of the neural circuit policy receives the motion state feature sequence of the robot generated in step S2, and uses inhibitory signals or excitatory signals with positive and negative polarities to transmit to the transfer layer, the inhibitory signals or excitatory signals are transmitted to the control layer through the transfer layer, and then are cyclically transmitted and output to the drive layer, the drive layer updates the states of all neurons, and each motion neuron calculates the output membrane potential through a non-linear activation function, and the bionic neuron with the highest output membrane potential is the robot control strategy;

[0010] S4. Construct the SAC critic network, input the current robot state and the robot control strategy obtained in step S3 into the SAC critic network. The SAC critic network calculates the output term of the SAC critic network based on the Q-value function to evaluate the effectiveness of the current robot state and the robot control strategy;

[0011] S5. Construct a replay buffer, store the current robot state and the robot control strategy obtained in step S3 into the replay buffer, store the output term of the SAC critic network obtained in step S4 into the replay buffer, and then update the weights of the SAC actor network and the SAC critic network based on the neural circuit policy network according to the information data stored in the replay buffer to complete the robot control based on SAC and the neural circuit policy.

[0012] Further, the simulation environment in step S1 is implemented based on PyBullet Walker2D and PyBulletAnt.

[0013] Further, the neural circuit policy network in step S3 is constructed by a four-layer hierarchical network topology, including a perception layer, a transfer layer, a control layer, and a drive layer. The neurons in the perception layer are perception neurons N s , the neurons in the transfer layer are intermediate neurons N i , the neurons in the control layer are command neurons N c , and the neurons in the drive layer are motor neurons N m ; the connection from the perception neurons to the intermediate neurons is dominated by feedforward, the connection between the intermediate neurons and the command neurons is highly coincident, and the connection from the command neurons to the motor neurons is feedforward;

[0014] The neural circuit policy network receives the motion state feature sequence of the robot through the perception neurons, then transmits the motion state feature sequence of the robot to the intermediate neurons and the command neurons to generate an output decision, and then transmits the generated output decision to the motor neurons to output the robot control strategy.

[0015] Further, the specific implementation method of step S3 includes the following steps:

[0016] S3.1. The perception layer receives the motion state feature sequence of the robot: Convert the motion state feature sequence of the robot obtained in step S2 into a pulse signal and input it into the perception neurons of the perception layer of the neural circuit policy network. The perception layer transmits inhibitory signals or excitatory signals to the transfer layer through synapses with different polarities, and updates the state of the intermediate neurons according to the weights of the synapses, while continuing to receive the motion state feature sequence of the robot and transmitting it to the transfer layer;

[0017] S3.2. Relay at the intermediate layer: The interneuron receives an excitatory signal or an inhibitory signal. The excitatory signal increases the membrane potential of the neuron, and the inhibitory signal decreases the membrane potential of the neuron. During the signal transmission process, when the membrane potential of the presynaptic neuron is higher than the transmission threshold at the positive-polarity synapse, the signal intensity will be enhanced, and when the membrane potential of the presynaptic neuron is higher than the transmission threshold at the negative-polarity synapse, the signal intensity will be decreased, so as to simulate the biological neural system model;

[0018] S3.3. Control layer loop: The command neuron receives the excitatory signal or inhibitory signal sent by the interneuron in step S3.2, and transmits the excitatory signal or inhibitory signal to the motor neuron through the synapse. The command neuron also simultaneously receives the output signal generated in the previous time interval of its own control layer, and the two jointly act on the membrane potential of the motor neuron;

[0019] After the motor neuron in the S3.4. drive layer receives the signal from the control layer, the neuron corresponding to the highest value of its membrane potential is encoded as the robot's motion mode, and the output corresponding to this neuron is selected as the robot control strategy.

[0020] Furthermore, the neural state representation equation for the postsynaptic neuron i receiving the input current from the presynaptic neuron j in step S3 is:

[0021]

[0022] where, x i is the current state of neuron i, that is, the membrane potential, is the time constant of neuron i with leakage conductance , and τ i is different for different bionic neurons, thus ensuring the asynchrony of the membrane potential change. w ij is the synaptic weight from neuron j to neuron i, is the membrane capacitance, σ i (x j ) is the neuron activation function, which is positively correlated with the signal intensity, is the resting potential, E ij is the reversal synaptic potential, which defines the polarity of the synapse;

[0023] The overall coupling sensitivity of the bionic neuron is expressed as:

[0024]

[0025] determines the reaction speed of the bionic neuron during the decision-making process.

[0026] Furthermore, the specific implementation method of step S4 includes the following steps:

[0027] S4.1. Build the SAC critic network, and set the π entropy of the robot control policy at time step t to H(π(·|s t )) and add the π entropy of the robot control policy at all time steps t except the first time step to the Q-value function Q π (s,a). The calculation expression is:

[0028]

[0029] where s represents the state, a represents the action; α is the temperature parameter that determines the weight of the entropy term; γ is the discount factor, and γ ∈ (0,1);

[0030] S4.2. Connect the Q-value function obtained in step S4.1 with the value function V π (s). The calculation expression is:

[0031]

[0032] S4.3. Based on the Q-value function in step S4.1, establish the entropy reward objective function y i . The calculation expression is:

[0033]

[0034] where r is the reward generated by the environment, and the subsequent actions are sampled from the current robot control policy;

[0035] S4.4. Use the reparameterization method to learn the robot control policy, and complete the sampling of the policy by calculating a deterministic function that depends on the state, policy variables, and uncorrelated noise ξ. The calculation expression of the deterministic function is:

[0036]

[0037] S4.5. Based on the deterministic function constructed in step S4, transform the output term of SAC into maximizing the expectation of the noise. The calculation expression is:

[0038]

[0039] By evaluating the value of the current state and activity of the robot control policy, provide feedback to the actor network. When taking the current action from the current state, the critic network inputs the current state and action and generates a Q-value to predict the effectiveness of the current robot state and the robot control policy.

[0040] Furthermore, the information in the replay buffer in step S5 is generated by the soft value function, which includes the output term of the SAC critic network and the entropy term that encourages exploration.

[0041] An electronic device, including a memory and a processor, where the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the robot control method based on the SAC and neural circuit strategy.

[0042] A computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the robot control method based on the SAC and neural circuit strategy.

[0043] Advantages of the present invention:

[0044] For the robot control method based on the SAC and neural circuit strategy of the present invention, based on the SAC (SoftArtist-Critic) algorithm, it combines the advantages of the actor-critic method and soft Q-learning with the advantages of stochastic policy optimization and DDPG learning method. SAC is a model-free and policy-free technique that uses deep neural networks to obtain policies, Q functions, and policy entropy rewards. In SAC, the trade-off between the expected return and policy entropy is maximized by simultaneously training the policy and Q function. Policy entropy encourages exploration and helps prevent the policy from collapsing into a single sub-optimal action. The neural circuit strategy is adopted as the actor network to implement the output of robot control decisions in the actor network.

[0045] For the robot control method based on the SAC and neural circuit strategy of the present invention, it combines robot control with the SAC algorithm in reinforcement learning, gets rid of the limitations of traditional control algorithms on models, improves the learning speed of the robot and the utilization efficiency of empirical samples, and solves the problems that the robot is prone to falling into local optima and unable to reasonably plan actions.

[0046] For the robot control method based on the SAC and neural circuit strategy of the present invention, by constructing a four-layer neural circuit strategy network, it can complete robot control efficiently and accurately. Compared with other technologies in the current period, this method has lower requirements for hardware computing power, can have higher computing efficiency, and only needs a small number of neurons to achieve better results.

[0047] For the robot control method based on the SAC and neural circuit strategy of the present invention, the introduction of offline reinforcement learning in the present invention allows the agent to obtain a better policy from the already collected and determined dataset through the reinforcement learning algorithm without interacting with the environment. From a safety perspective, it avoids the possible occurrence of danger and huge losses caused by the interaction between the agent that has not been well learned and the environment.

[0048] A robot control method based on SAC and neural circuit strategy according to the present invention is a specific implementation of the third-generation neural network in robot control. Compared with other technologies in the current period, the working principle of this method is closer to the functional principle of nerve cells, has more advanced theoretical support, and has more development potential in the field of artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of a robot control method based on SAC and neural circuit strategy according to the present invention;

[0050] Figure 2 is a schematic diagram of the joint movement of the PyBullet Walker2D robot according to the present invention;

[0051] Figure 3 is a schematic flow diagram of a robot control method based on SAC and neural circuit strategy according to the present invention;

[0052] Figure 4 is the basic model of the neural circuit strategy network of a robot control method based on SAC and neural circuit strategy according to the present invention;

[0053] Figure 5 is the block diagram of the SAC algorithm of a robot control method based on SAC and neural circuit strategy according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only a part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention usually described and shown in the drawings here can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0055] Therefore, the detailed description of the specific embodiments of the present invention provided in the drawings below is not intended to limit the scope of the present invention claimed, but only represents the selected specific embodiments of the present invention. All other specific embodiments obtained by those skilled in the art based on the specific embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0056] To further understand the content, features and effects of the present invention, the following specific embodiments are exemplified and accompanied by the attached Figure 1 - attached Figure 5 The details are as follows: Specific Embodiment 1:

[0058] A robot control method based on SAC and neural circuit strategy, comprising the following steps:

[0059] S1. Build a simulation environment, design the initial position, speed and inclination angle of the robot, and collect the motion state data of the robot;

[0060] Furthermore, the simulation environment in step S1 is implemented based on PyBullet Walker2D and PyBulletAnt;

[0061] Furthermore, PyBullet Walker2d involves a two-dimensional walking robot that faces the challenge of moving forward as fast as possible. Since the robot has a higher probability of falling and more degrees of freedom than other works, this challenge is considered more challenging. The 6 degrees of freedom of Walker2D consist of five links - the torso, two links on each leg (upper and lower links), and two feet, as Figure 2 shown. The hip joint θ5 connecting the two legs and the torso has two degrees of freedom, allowing the two legs to rotate simultaneously in the sagittal plane and the frontal plane. The right leg can rotate at an inclined angle in the frontal plane and at an angle in the frontal plane, while the left leg can rotate at an inclined angle in the frontal plane and at an angle in the frontal plane. The links of the upper and lower legs θ4θ7θ6 are connected by knee joints, each joint having 1 degree of freedom, allowing rotation in the sagittal plane. The calf links are connected to the feet through ankle joints, each ankle joint having 2 degrees of freedom, allowing the feet to rotate simultaneously in the sagittal plane and the frontal plane. The right leg ankle joint θ1 can rotate the foot at a certain angle in the frontal plane and at a certain angle in the frontal plane, while the left leg ankle joint can rotate the foot at a certain angle in the frontal plane and at a certain angle in the frontal plane. θ9θ 10 θ2 Each rotating joint is driven by a single DC motor without friction.

[0062] Furthermore, PyBulletAnt involves a three-dimensional quadruped robot that must walk forward as fast as possible in the environment. In Ant, both the observation domain and the action domain are high-dimensional. The reward obtained by the agent is a combination of several factors, including survival reward, the forward speed of the robot, and penalties for joint jamming and certain actions taken by the robot.

[0063] S2. Build a linear layer, input the motion state data of the robot collected in step S1 into the linear layer for linear transformation, and obtain the motion state feature sequence of the robot;

[0064] S3. Construct the SAC actor network based on the neural circuit policy network, and input the motion state feature sequence of the robot obtained in step S2 into the SAC actor network based on the neural circuit policy network. The perception layer of the neural circuit policy receives the motion state feature sequence of the robot generated in step S2, and uses synapses with positive and negative polarities to transmit inhibitory signals or excitatory signals to the transfer layer. The inhibitory signals or excitatory signals are transmitted to the control layer through the transfer layer, and then are cyclically transmitted and output to the drive layer. The drive layer updates the states of all neurons, and each motor neuron calculates the output membrane potential through a non-linear activation function. The bionic neuron with the highest output membrane potential is the robot control strategy;

[0065] Further, the neural circuit policy network described in step S3 is constructed by a four-layer hierarchical network topology, including a perception layer, a transfer layer, a control layer, and a drive layer. The neurons in the perception layer are perception neurons N s , the neurons in the transfer layer are intermediate neurons N i , the neurons in the control layer are command neurons N c , and the neurons in the drive layer are motor neurons N m ; The connection from the perception neuron to the intermediate neuron is a feedforward-dominated connection, the connection between the intermediate neuron and the command neuron is a highly coincident connection, and the connection from the command neuron to the motor neuron is a feedforward connection;

[0066] The neural circuit policy network receives the motion state feature sequence of the robot through the perception neuron, then transmits the motion state feature sequence of the robot to the intermediate neuron and the command neuron to generate an output decision, and then transmits the generated output decision to the motor neuron to output the robot control strategy;

[0067] Further, the specific implementation method of step S3 includes the following steps:

[0068] S3.1. The perception layer receives the motion state feature sequence of the robot: Convert the motion state feature sequence of the robot obtained in step S2 into a pulse signal and input it into the perception neurons of the perception layer of the neural circuit policy network. The perception layer transmits inhibitory signals or excitatory signals to the transfer layer through synapses with different polarities, and updates the state of the intermediate neuron according to the weight of the synapse, and at the same time continues to receive the motion state feature sequence of the robot and transmit it to the transfer layer;

[0069] S3.2. Transfer layer transfer: The intermediate neuron receives the excitatory signal or inhibitory signal. The excitatory signal will increase the neuron membrane potential, and the inhibitory signal will decrease the neuron membrane potential. During the signal transmission process, when the membrane potential of the source neuron on the positive-polarity synapse is higher than the transmission threshold, the signal intensity will be enhanced, and when the membrane potential of the source neuron on the negative-polarity synapse is higher than the transmission threshold, the signal intensity will be reduced, so as to simulate the biological nervous system model;

[0070] S3.3. Control layer loop: The command neuron receives the excitation signal or inhibition signal sent by the interneuron in step S3.2, and transmits the excitation signal or inhibition signal to the motor neuron through the synapse. The command neuron also simultaneously receives the output signal generated in the previous time interval of its own control layer, and the two jointly act on the membrane potential of the motor neuron;

[0071] After the motor neuron in the drive layer receives the signal from the control layer, the neuron corresponding to the maximum value of its membrane potential is encoded as the robot's motion mode, and the output corresponding to this neuron is selected as the robot control strategy;

[0072] Furthermore, the neural state representation equation for the postsynaptic neuron i receiving the input current from the presynaptic neuron j in step S3 is:

[0073]

[0074] where x i is the current state of neuron i, that is, the membrane potential, is the time constant of neuron i with leakage conductance , and τ i is different on different bionic neurons, thus ensuring the asynchrony of the membrane potential change. w ij is the synaptic weight from neuron j to neuron i, i is the membrane capacitance, σ j (x ) is the neuron activation function, which is positively correlated with the signal strength, is the resting potential, E ij is the reversal synaptic potential, which defines the polarity of the synapse;

[0075] The expression of the overall coupling sensitivity

[0076]

[0077]

[0078] Furthermore, in the neural circuit policy network, each bionic neuron uses the membrane potential to represent the neuron state, and dynamically updates the membrane potential with a differential equation. The neuron state is jointly determined by the current membrane potential and the action of the input synapses from the upper-layer neurons to the current neuron;

[0079] Furthermore, the interneuron establishes synaptic connections with both the sensory neuron and the command neuron; the command neuron establishes synaptic connections with both the interneuron and the motor neuron, and establishes self-feedback synaptic connections with other command neurons to form a cyclic connection structure.

[0080] Furthermore, the neural circuit policy network creates bionic neurons and establishes synapses that can transmit inhibitory and excitatory signals between the bionic neurons. The synapses change the state of the target bionic neuron through asynchronous bipolar signal transmission between the synapses, and the state update of the bionic neuron corresponds to the processing of the image feature vector pulse signal.

[0081] Specifically, the sensory neurons receive the feature vector sequence, convert the feature vector sequence into a pulse signal, stimulate the interneurons, and send inhibitory or excitatory signals to the interneurons. The interneurons are used to escape the obtained feature vector sequence and send inhibitory or excitatory signals to the command neurons; the command neurons save and make decisions on the timing information, establish a self-loop in the control layer, and at the same time stimulate the motor neurons, sending inhibitory or excitatory signals to the motor neurons and other command neurons; the motor neurons in the drive layer output the actions applied to each joint according to the level of their own pulse signals.

[0082] In this embodiment, in the neural circuit policy, each bionic neuron uses the membrane potential to represent the neuron state, and the membrane potential is dynamically updated by a differential equation. The neuron state is jointly determined by the current membrane potential and the action of the input synapse from the upper neuron to the current neuron, and the synapses connecting between the bionic neurons have different weights and two polarities. Among them, the synapse with the positive polarity will increase the membrane potential of the target neuron, while the synapse with the negative polarity will decrease the membrane potential of the target neuron. Therefore, different synapses have different effects on the membrane potential of the target neuron.

[0083] In this embodiment, a typical network of the neural circuit policy, the tap-withdrawal circuit, is used. Refer to Figure 4 . The tap-withdrawal neural circuit consists of two groups of motor neurons: the forward motor neurons (FWD) and the backward motor neurons (REV), three types of interneurons: AVD, PVC, and DVA, two types of command neurons AVA and AVB, and four types of sensory neurons: PVD, PLM, AVM, and ALM (touch sensors). The interneurons in this circuit have excitatory and inhibitory synaptic connections, and these synaptic connections are cyclic. The TW neural circuit consists of 11 neurons and 28 synapses.

[0084] S4. Construct the SAC critic network, input the current robot state and the robot control strategy obtained in step S3 into the SAC critic network. The SAC critic network calculates the output term of the SAC critic network based on the Q-value function to evaluate the effectiveness of the current robot state and the robot control strategy;

[0085] Furthermore, the SAC algorithm is a reinforcement learning technique that combines the advantages of the actor-critic method and soft Q-learning with the advantages of stochastic policy optimization and the DDPG learning method; the trade-off between the expected return and the policy entropy in the SAC algorithm is achieved by maximizing through the simultaneous training of the policy and the Q function, and the policy entropy encourages exploration and helps prevent the policy from collapsing into a single suboptimal action;

[0086] Furthermore, the specific implementation method of step S4 includes the following steps:

[0087] S4.1. Construct an SAC critic network, and set the π entropy of the robot control policy at time step t to be H(π(·|s t ))). Except for the first time step, the π entropy of the robot control policy at all time steps t is added to the Q-value function Q π (s,a), and the calculation expression is:

[0088]

[0089] where s represents the state, a represents the action; α is the temperature parameter that determines the weight of the entropy term; γ is the discount factor, and γ ∈ (0,1);

[0090] S4.2. Connect the Q-value function obtained in step S4.1 with the value function V π (s), and the calculation expression is:

[0091]

[0092] S4.3. Based on the Q-value function in step S4.1, establish an entropy reward objective function y i , and the calculation expression is:

[0093]

[0094] where r is the reward generated by the environment, and the subsequent actions are sampled from the current robot control policy;

[0095] S4.4. Adopt the reparameterization method to learn the robot control policy, and complete the sampling of the policy by calculating a deterministic function that depends on the state, policy variables, and uncorrelated noise ξ. The calculation expression of the deterministic function is:

[0096]

[0097] S4.5. Based on the deterministic function constructed in step S4, convert the output term of SAC into maximizing the expectation of the noise, and the calculation expression is:

[0098]

[0099] By evaluating the current state and the value of the actions of the robot control strategy, feedback is provided to the actor network. When taking the current action from the current state, the critic network inputs the current state and action and generates a Q-value to predict the effectiveness of the current robot state and the robot control strategy;

[0100] Furthermore, the SAC algorithm is a prominent model-free reinforcement learning technique that excels in continuous control tasks. The algorithm block diagram is as Figure 5 shown. To learn the optimal strategy for the task at hand, it combines the actor-critic architecture, stochastic policies, and value function optimization. The key components of the neural circuit policy and the SAC algorithm are:

[0101] Actor network: It inputs the current state of the system and outputs the necessary actions. And we have modified the actor network through the neural circuit policy. In the case of controlling the Walker2D robot, the actor network will obtain relevant information such as the current joint angles and joint velocities and output the torques to be applied to each joint.

[0102] Critic network: The evaluation network provides feedback to the actor network by evaluating the value of the current state and actions. When taking the current action from the current state, the critic network inputs the current state and action and generates a q-value indicating the predicted cumulative reward.

[0103] Replay buffer: The actor and critic technology networks use the information stored in the replay buffer for updating. A set of transitions is collected from the replay buffer and used to update the network in each training process.

[0104] Flexible value function: The SAC algorithm uses a flexible value function that includes an entropy term in addition to the expected cumulative reward. This encourages the policy to be more exploratory and less deterministic as it avoids the problem of getting stuck in suboptimal policies.

[0105] Temperature parameter: The weight of the entropy term in the objective function is controlled by the temperature parameter. It is automatically updated throughout the training process to ensure that the policy remains sufficiently random to explore the nature of the state space.

[0106] The SAC algorithm can be used to learn a policy that maps the current state of the robot to a series of actions that will cause the robot to move in the desired direction when driving the Walker2D robot in PyBullet. The algorithm uses a neural network to select actions throughout the training process and then iteratively adjusts the parameters of the neural network based on a loss function. The generated policy can then be used to operate the robot in the simulation.

[0107] S5. Construct a replay buffer, store the current robot state and the robot control policy obtained in step S3 into the replay buffer, store the SAC critic network output item obtained in step S4 into the replay buffer, and then update the weights of the SAC actor network and the SAC critic network based on the neural circuit policy network based on the information data stored in the replay buffer, completing the robot control based on SAC and the neural circuit policy.

[0108] Furthermore, the information in the replay buffer in step S5 is generated by a soft value function, which includes the output item of the SAC critic network and an entropy item that encourages exploration.

[0109] The robot control method based on SAC and the neural circuit policy described in this embodiment consists of the above steps to form an independent robot control method based on the neural circuit policy and the SAC algorithm. Compared with other technologies, this aspect proposes a brand-new method that combines the spiking neural network with the reinforcement learning framework, which can achieve a relatively low overall computational load while ensuring the rapid response ability. Specific Embodiment 2:

[0111] An electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the robot control method based on SAC and the neural circuit policy described in Specific Embodiment 1.

[0112] The computer device of the present invention can be a device including a processor and a memory, such as a single-chip microcomputer including a central processing unit. And the processor is used to implement the steps of the above-mentioned robot control method based on SAC and the neural circuit policy when executing the computer program stored in the memory.

[0113] The so-called processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Specific Embodiment 3:

[0116] A computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the robot control method based on SAC and neural circuit strategy described in Specific Embodiment 1.

[0117] The computer-readable storage medium of the present invention may be any form of storage medium readable by the processor of the computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned robot control method based on SAC and neural circuit strategy can be implemented.

[0118] The computer program includes computer program code, which may be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice within the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0119] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0120] Although the present application has been described above with reference to specific embodiments, various improvements can be made to it and its components can be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application can be combined with each other in any way. The reason for not exhaustively describing the situations of these combinations in this specification is only to save space and resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A robot control method based on SAC and neural circuit strategy, characterized in that, It includes the following steps: S1. Build a simulation environment, design the initial position, speed and inclination angle of the robot, and collect the motion state data of the robot; S2. Build a linear layer, input the motion state data of the robot collected in step S1 into the linear layer for linear transformation to obtain the motion state feature sequence of the robot; S3. Build a SAC actor network based on the neural circuit policy network, input the motion state feature sequence of the robot obtained in step S2 into the SAC actor network based on the neural circuit policy network. The perception layer of the neural circuit policy receives the motion state feature sequence of the robot obtained in step S2, and uses inhibitory signals or excitatory signals with positive and negative polarities to transmit to the transfer layer. The inhibitory signals or excitatory signals are transmitted to the control layer through the transfer layer, and then are cyclically transmitted and output to the drive layer. The drive layer updates the states of all neurons, and each motion neuron calculates the output membrane potential through a non-linear activation function. The bionic neuron with the highest output membrane potential is the robot control strategy; S4. Build a SAC critic network, input the obtained current robot state and the robot control strategy into the SAC critic network. The SAC critic network calculates the output term of the SAC critic network based on the Q-value function to evaluate the effectiveness of the current robot state and the robot control strategy; S5. Build a replay buffer, store the obtained current robot state and the robot control strategy in the replay buffer, store the output term of the SAC critic network obtained in step S4 in the replay buffer, and then update the weights of the SAC actor network and the SAC critic network based on the neural circuit policy network based on the information data stored in the replay buffer to complete the robot control based on SAC and the neural circuit policy.

2. The robot control method based on the SAC and neural circuit strategy according to claim 1, wherein, The simulation environment in step S1 is implemented based on PyBullet Walker2D and PyBullet Ant.

3. The robot control method based on the SAC and neural circuit strategy according to claim 2, wherein, The neural circuit policy network described in step S3 is constructed by a four-layer hierarchical network topology, including a perception layer, a transit layer, a control layer, and a drive layer. The neurons in the perception layer are perception neurons N s , the neurons in the transit layer are intermediate neurons N i , the neurons in the control layer are command neurons N c , and the neurons in the drive layer are motor neurons N m ; The connection from the perception neurons to the intermediate neurons is dominated by feedforward, the intermediate neurons and the command neurons have a highly overlapping connection, and the connection from the command neurons to the motor neurons is feedforward; The neural circuit policy network receives the motion state feature sequence of the robot through the perception neurons, then transmits the motion state feature sequence of the robot to the intermediate neurons and command neurons to generate an output decision, and then transmits the generated output decision to the motion neurons to output the robot control strategy.

4. A robot control method based on the SAC and neural circuit strategy according to claim 3, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. The perception layer receives the motion state feature sequence of the robot: Convert the motion state feature sequence of the robot obtained in step S2 into a pulse signal and input it into the perception neurons of the perception layer of the neural circuit policy network. The perception layer transmits inhibitory signals or excitatory signals to the transfer layer through synapses with different polarities, and updates the state of the intermediate neurons according to the weights of the synapses, and at the same time continues to receive the motion state feature sequence of the robot and transmit it to the transfer layer; S3.

2. Transfer layer transfer: The intermediate neurons receive excitatory signals or inhibitory signals. The excitatory signals will increase the neuron membrane potential, and the inhibitory signals will decrease the neuron membrane potential. During the signal transmission process, when the membrane potential of the source neuron on the positive-polarity synapse is higher than the transmission threshold, the intensity of the signal will be enhanced, and when the membrane potential of the source neuron on the negative-polarity synapse is higher than the transmission threshold, the intensity of the signal will be reduced, so as to simulate the biological nervous system model; S3.

3. Control layer loop: The command neuron receives the excitation signal or inhibition signal sent by the interneuron in step S3.2, and transmits the excitation signal or inhibition signal to the motor neuron through the synapse. The command neuron also simultaneously receives the output signal generated in the previous time interval of its own control layer, and the two act together on the membrane potential of the motor neuron; S3.

4. After the motor neuron in the drive layer receives the signal from the control layer, the neuron corresponding to the highest value of its membrane potential is encoded as the robot's movement mode, and the output corresponding to this neuron is selected as the robot control strategy.

5. The robot control method based on the SAC and neural circuit strategy according to claim 4, wherein, The neural state representation equation for the postsynaptic neuron i receiving the input current from the presynaptic neuron j in step S3 is: where, x i is the current state of neuron i, i.e., the membrane potential, is the time constant of neuron i with leakage conductance τ varies among different biomimetic neurons, thus ensuring the asynchrony of membrane potential changes, w i varies, ij is the synaptic weight from neuron j to neuron i, is the membrane capacitance, σ i (x j ) is the neuron activation function, which is positively correlated with the signal intensity, is the resting potential, E ij is the reversal synaptic potential, which defines the polarity of the synapse; Overall coupling sensitivity of bionic neurons The expression is as follows: The response speed of the bionic neuron in the decision-making process is determined.

6. A robot control method based on the SAC and neural circuit strategy according to claim 5, characterized in that, The information in the replay buffer of step S5 is generated by the soft value function, which includes the output term of the SAC critic network and the entropy term that encourages exploration.

7. An electronic device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of a robot control method according to any one of claims 1-6, which is based on SAC and neural circuit strategies.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a robot control method according to any one of claims 1-6, which is based on SAC and neural circuit strategies.

Images