A smooth control method for a rat robot based on transfer reinforcement learning

Abstract

The present invention discloses a rat robot smooth control method based on transfer reinforcement learning, which constructs an environmental model about rats by collecting certain rat robot control data, and then trains a reinforcement learning control algorithm capable of achieving smooth control in a simulation environment based on the model and fluency constraints, and then migrates the environmental model and the control algorithm to a real environment so that the algorithm can quickly adapt to achieve automatic control of rats to complete complex maze tasks. The present invention achieves a more accurate prediction of the future posture of rats, and can measure the uncertainty of rat behavior at the same time. Through the construction of a simulation platform, the reinforcement learning algorithm can be trained in the simulation platform, and there is no need to train the reinforcement learning model with a large amount of data, which can help the algorithm to quickly adapt in a real environment. The present invention proposes a fluency measurement index, and combined with the reinforcement learning control algorithm, it can effectively achieve smooth and accurate control of rats.

Description

Technical Field

[0001] The invention belongs to the technical field of rat robots, and in particular relates to a rat robot fluency control method based on transfer reinforcement learning. Technical Background

[0002] Animal robots are a new type of robot. They are hybrid intelligent entities that integrate artificial intelligence and biological intelligence. They not only retain the advantages of intelligent organisms that have evolved naturally, but also have flexible and coordinated movements. They are also stronger than independent animals or machines. Animal robots have a wide range of applications and can be used in military reconnaissance, disaster search and rescue, medical diagnosis and other scenarios. As a frontier field of scientific research, its motion control has attracted the attention of many scholars.

[0003] Animal robots are the product of the cross-integration of animal physiology, computer science, control science and other disciplines. The principle is to use brain-computer interface technology to install electrodes on the animal brain, and generate electrical signals through the electrode backpack to stimulate the animal to complete the specified action. Compared with traditional robots, animal robots are directly based on the animal body, without the need to consider the use of alloys or special materials. They retain the good functions of the animal itself through natural evolution, and can autonomously realize movement and obstacle avoidance functions without the need for special control algorithms to detect and judge the environment in complex environments. The endurance of animal robots has been greatly improved, solving the long-standing problem of power supply in the current robotics field.

[0004] Since the control of rats is uncertain, it is difficult for rats to complete the specified tasks smoothly because of the inefficiency and imprecise control when manually adjusting the stimulation parameters. Reinforcement learning can automatically interact with the environment to obtain an optimal control strategy, but reinforcement learning also has the problem that it takes a long time to learn to obtain the optimal control strategy. Therefore, how to use less data combined with reinforcement learning control algorithms to achieve smooth control of rat behavior is a very big challenge. Summary of the invention

[0005] The purpose of the present invention is to provide a rat robot smooth control method based on transfer reinforcement learning to solve the problem that smooth behavior control is difficult to achieve artificially in the technology and requires a large number of biological experiments.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0007] A rat robot smooth control method based on transfer reinforcement learning comprises: constructing an environment model based on an integrated LSTM (ELSTM), then building a simulated maze platform and simulated rats, then training a simulated environment for an RTRPO control algorithm, and finally online adaptation of the ELSTM and RTRPO, wherein the simulated maze platform is constructed based on an integrated environment model, restrictions on fluency indicators are added to the environment model, and the RTRPO algorithm is used to complete the simulation training task; the ELSTM model and the RTRPO model are migrated to a real environment, the ELSTM environment model is modified by interacting with the real environment, and then virtual data is generated by the ELSTM environment model to help the RTRPO model quickly complete online adaptation.

[0008] Each LSTM network in the ELSTM environmental model is randomly initialized to obtain different initial values, and then different key posture point sequence data are used for training. The time series information is represented by a recurrent neural network. Each LSTM network uses the key posture points of a real rat and the current stimulation instruction information as input, and the mean and variance of the output results are used as the mean and variance of the model. The variance is used to represent the uncertainty of the simulation environment, and a normal distribution is established with the mean and variance. The key posture points of the rat at the next moment are predicted by sampling the normal distribution, and the ELSTM environmental model is trained using the L2 distance as the loss function. The key posture points are based on a third-perspective overhead video. The key points of the real rat robot in the video are detected and extracted by a difference-based background modeling method, and 8 two-dimensional key points are set, including the head, the base of the tail, the left front paw, the left hind paw, the right front paw, the right hind paw, the midpoint of the neck, and the midpoint of the spine.

[0009] The simulated maze platform uses MuJoCo software to build a simulated maze environment. The simulated environment is used to simulate the reinforcement learning control algorithm in the real environment to establish an interaction mechanism between rats and electrical stimulation. Reinforcement learning is learned by continuous interaction with the virtual environment, and finally the optimal control strategy is obtained; the virtual environment includes five main parts: action (a), current state (s), next moment state (s′), reward function (r), and state transfer function (p(s′|a,s)); the actions include moving forward, stopping, turning left, and turning right. The current state is the coordinates of the current rat's key posture point and the end point of the maze. The next moment state is the coordinates of the rat's key posture point and the end point of the maze at the next moment. The reward function is whether the rat has completed the maze task. When the center of gravity of the rat is away from the target position x g When the Euclidean distance of is less than the specified threshold, the reward value is 1, otherwise it is -1. Then the reward function can be written as:

[0010] R=+1if((x m -x g )<σ));

[0011] R=-1, otherwise.

[0012] x m , x g is the center of gravity and end point position of the rat, and the trained ELSTM model is used as the state transfer function to represent the change in the position of the simulated rat after stimulation.

[0013] The RTRPO control algorithm includes a strategy function and a value function. The action space of the value function and the strategy function are both 4, and the state space is 18. Compared with the traditional TRPO control algorithm, the RTRPO control algorithm uses LSTM to replace the fully connected layer, and uses historical information to help the algorithm better select actions. Based on the above-built simulation environment, an RTRPO control algorithm that can control the simulated rat robot to smoothly complete the maze task is trained under the restriction of the fluency index. In order to perform fluency control, the cumulative reward function is maximized. becomes the function that maximizes the cumulative reward under constraints, Next, maximize Maximization Among them, from c(s t ,a t ) is an abnormal motion metric, δ represents the threshold value of fluency restriction, and the strategy function of the RTRPO control algorithm gives real-time instructions to control the behavior of the rat according to the key posture points of the simulated rat, so that the simulated rat robot can smoothly complete the maze task. The reinforcement learning algorithm obtains an optimal strategy function through continuous interactive learning in the simulation environment, that is, the next control instruction is determined according to the coordinates of the current 8 key points of the rat to form a closed-loop control. The simulation training process is:

[0014] (1) Initialize the parameters φ of the state function V and the parameters θ of the policy function π.

[0015] (2) Set the hyperparameters KL divergence threshold δ, backtracking coefficient α, and maximum backtracking step number K.

[0016] (3) By executing the policy function π k , to collect the trajectory D k ={τ i},

[0017] (4) Calculate the return R between steps t and T t .

[0018] (5) Calculate the advantage function A t =R t -V t .

[0019] (6)

[0020] (7) Using conjugate gradient calculation in is the inverse matrix of the Hessian matrix

[0021] (8) Using linear search calculation where j∈{0,1,2…K} is a and The smallest value, among which

[0022] (9) Update parameter φ through TD-error:

[0023]

[0024] (10) Execute (2) again and start the loop until the policy function π converges.

[0025] The fluency measurement index mainly includes a posture difference index and a speed frequency domain index. The posture difference index refers to a measurement method between two posture points in adjacent time series established by using the fine posture key point characterization data of the rat robot's motion behavior, which is used to measure the difference between the two postures. Specifically, the skeleton sequence is constructed using the 8 key points of the rat robot, and the rat individual is abstracted into a two-dimensional fine skeleton representation described by the posture key points. The Gromov-Wasserstein (GW) distance, which is suitable for non-rigid deformation and good stability, is used to analyze the correspondence between any two key points in the fine posture, and establish a difference matrix of all key point posture data of the rat robot, thereby obtaining a comprehensive difference measurement between the two sets of skeletons. If the posture difference is large, then the GW distance will be relatively large. By calculating the posture difference between two adjacent frames, the moment when obvious posture switching occurs in the sequence posture can be calculated, and the fragments with large differences from natural motion can be found in time, and then the smooth switching and sequence control between its motion states can be achieved by adjusting the control strategy. For posture variables with T postures Each posture has V key points, and the two-dimensional coordinates of each key point are (x, y). So for any two postures Calculate the quasi-GW distance between two sets of posture sequences under a fixed transmission scheme:

[0026] c(p,q)=‖M p -M q ‖

[0027] Where M represents the pairwise distance matrix, where M i,j =‖p i -p j‖.

[0028] The speed frequency domain index refers to the use of the frequency domain characteristics of the movement speed as one of the bases of the smoothness to measure the degree of body shaking. Taking the average coordinates of N key points on the horizontal plane of the rat robot as the center of mass, its motion trajectory is converted into a time series composed of two-dimensional center of mass coordinates, which contains statistical data such as position, speed, distance, and movement time, which can be used to measure its motion state. The finite difference method is used to approximate the speed v of the current frame. The main manifestation of abnormality is that the frequency of body shaking is higher than that of normal movement. This shaking movement is mainly concentrated in a fixed frequency band in the frequency domain, and a large peak will appear in this frequency band, while the unilateral spectrum of normal movement has a flat curve in this frequency band and does not appear a large peak. Therefore, the measurement of spastic abnormal movement is measured by calculating the proportion of the power spectrum size of this frequency band in the total power spectrum. The higher the proportion score, the larger the area of ​​the wave, indicating that the degree of spastic shaking p(v) of the rat robot is higher.

[0029] The rat robot's smooth movement metric combines two types of abnormal movement indicators, namely:

[0030] y=c(p,q)+p(v)

[0031] The overall movement smoothness assessment y is placed in the range [0,1], where 0 represents the ideal movement quality and 1 represents the maximum deviation from the ideal movement. Higher scores indicate a greater degree of abnormality, while lower scores indicate greater smoothness.

[0032] The ELSTM and RTRPO online adaptation uses the above-trained ELSTM environment model and RTRPO control algorithm to migrate to the real environment, and collects data by continuously interacting with the real environment to update the ELSTM model and RTRPO control algorithm. At the same time, the updated ELSTM model is used to generate virtual data to help the RTRPO control algorithm complete the online adaptation process faster, so that the real rat robot can smoothly complete the maze task. The online adaptation process is:

[0033] (1) Use the parameters in the simulation environment to initialize the state function V parameter φ, the policy function π parameter θ, the KL divergence limit δ, the backtracking coefficient α, and the maximum backtracking step size K.

[0034] (2) Collect some trajectories D by executing the policy function π k ={τ i}.

[0035] (3) Calculate the return R between steps t and T t .

[0036] (4) Calculate the advantage function A t =R t -Vt .

[0037] (5)

[0038] (6) Using conjugate gradient calculation

[0039] (7) Using linear search calculation where j∈{0,1,2…K} is a and The smallest value,

[0040] (8) Update parameter φ through TD-error:

[0041]

[0042] (9)min∑ k ||f(s,a;θ k )-(s′-s)||.

[0043] (10) Generate virtual trajectories using ELSTM and RTRPO.

[0044] (11) Use ELSTM virtual data to train RTRPO, the same as steps (2)-(7).

[0045] After completing the adaptation, in the real control task, the RTRPO control algorithm achieves smooth and precise control of the rat based on the coordinates of the rat's real-time key posture points.

[0046] The control instructions are generated by electrodes implanted in the rat's brain after receiving control signals generated by an electrical stimulation backpack on the rat's back.

[0047] Specifically, a real-time attitude estimation algorithm uses the third-view data of the maze provided by the top camera as the input of the attitude estimation algorithm. The top camera is composed of: The top camera is an ordinary network camera of model ANCHD1080P, with a resolution of 640×480, a lens focal length of 6mm, and a photosensitive chip size of 1 / 3 feet.

[0048] The present invention controls the behavior of rats through electrical stimulation. A large number of studies have found that specific tasks such as search and rescue in confined scenes can be accomplished by electrically stimulating designated areas of the rat's cerebral cortex.

[0049] The present invention has the following effective effects:

[0050] (1) Using ELSTM, combined with historical information, a more accurate prediction of the rat’s future posture can be achieved, while also being able to measure the uncertainty of the rat’s behavior.

[0051] (2) The present invention builds a simulation platform to enable the reinforcement learning algorithm to be trained in the simulation platform. A large amount of data is not required to train the reinforcement learning model. At the same time, the integrated LSTM can help the algorithm to quickly adapt in the real environment.

[0052] (3) The present invention proposes a fluency measurement index and combines it with a reinforcement learning control algorithm to effectively achieve smooth and accurate control of rats. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of a rat smooth control method based on transfer reinforcement learning.

[0054] Figure 2 Schematic diagram of the integrated model structure.

[0055] Figure 3 is a schematic diagram of the scene, where Figure 3 (A) is an experimental platform for a rat smooth control method based on transfer reinforcement learning, and Figure 3 (B) is a rat robot wearing an electronic backpack.

[0056] Figure 4 Schematic diagram of the interaction process between reinforcement learning and the environment.

[0057] Figure 5 This is the flow chart of the RTRPO algorithm.

[0058] Figure 6 There are 8 key points for the rat posture. DETAILED DESCRIPTION

[0059] The present invention will be further described below in conjunction with the accompanying drawings and implementation examples.

[0060] Example 1

[0061] like Figure 1As shown, the process of the rat smooth control method based on transfer reinforcement learning consists of 4 parts in total. In the first part, 4000 frames of images of rat posture changes after stimulation are first collected to form the training set of the ELSTM model. The key posture points and stimulation instructions at the current moment are used as input, the key posture points at the next moment are used as predicted labels, and the L2 distance is used as the loss function to train the ELSTM model to obtain an environmental model that can normally predict posture changes. The key posture points are based on the third-perspective overhead video. The key points of the real rat robot in the video are detected and extracted by the difference-based background modeling method. The 8 two-dimensional key points include the head, the root of the tail, the left front paw, the left hind paw, the right front paw, the right hind paw, the midpoint of the neck and the midpoint of the spine; the second part is to build a simulated maze platform based on the ELSTM environmental model, and use MuJoCo software to construct a simulated maze and a simulated rat. For the simulated rat, an approximate rat is constructed by inputting the key posture points of the rat. After the rat receives the control instruction, the key points of the rat at the next moment are predicted based on the ELSTM model, and a new rat is reconstructed in the maze. The third part of the simulation environment training RTRPO control algorithm includes a strategy function and a value function. The action space of the value function and the strategy function is 4, and the state space is 18. Based on the simulation environment built above, under the condition of fluency index restriction, a RTRPO control algorithm that can control the simulated rat robot to smoothly complete the maze task is trained. In order to perform fluency control, the cumulative reward function is maximized. becomes the function that maximizes the cumulative reward under constraints, Next, maximize Maximization Where c(s t ,a t ) represents smoothness control, which is used to measure abnormal movements during rat locomotion.

[0062] The strategy function of the RTRPO control algorithm gives real-time instructions to control the behavior of the rat according to the key posture points of the simulated rat, so that the simulated rat robot can smoothly complete the maze task. The reinforcement learning algorithm obtains an optimal strategy function by continuous interactive learning in the simulation environment, that is, the next control instruction is determined according to the coordinates of the current 8 key points of the rat to form a closed-loop control. During the update process, the strategy function is updated by the strategy gradient method, and the value function is updated by the TD error. The fourth part is the online adaptation of the ELSTM and RTRPO, which uses the above-trained ELSTM environment model and RTRPO control algorithm to migrate to the real environment, and continuously interacts with the real environment to collect data to update the ELSTM model and RTRPO control algorithm, wherein the ELSTM is updated using the L2 distance loss function, the RTRPO strategy function is updated by the strategy gradient method, and the value function is updated by the TD error. At the same time, the updated ELSTM model is used to generate virtual data to help the RTRPO control algorithm complete the online adaptation process faster, so that the real rat robot can smoothly complete the maze task, and the virtual data is the state transition pair of (s, a, s′).

[0063] like Figure 2 As shown in the figure, the ELSTM model is composed of 3 completely identical neural networks. Each neural network includes 3 layers. The input is the coordinates and actions of the current key posture points of the rat, and the output is the distribution of posture points after stimulation. The distribution used by the ELSTM model is Gaussian distribution, the mean is the mean of the output of each model, and the variance is the variance of the output of each model.

[0064] As shown in Figure 3, Figure 3 (A) is an experimental platform for a rat smooth control method based on transfer reinforcement learning, including a top camera at the top and a maze platform at the bottom. The top camera is used to obtain panoramic information of the entire scene, so that the posture estimation algorithm can obtain the coordinate position of the key points of the rat from the panoramic information. Figure 3 (B) is a rat robot wearing an electronic backpack. The electronic backpack on the rat's head will send electrical stimulation to the designated area of ​​the rat's cerebral cortex according to the received instructions to control the rat to complete the designated behavior.

[0065] like Figure 4 As shown in the figure, it is the interaction process between the reinforcement learning algorithm and the simulation environment in the present invention. The RTRPO algorithm maintains a strategy function π(s t ,a t ), indicating that in state s t , the optimal action to be taken is a t The update of the policy function is achieved by maximizing the reward in continuous interaction with the environment under the condition of fluency constraints.

[0066] RTRPO is a same-track strategy, so the same strategy needs to be used until the end of the round to update the strategy function and store all states, actions, and rewards in the round into buffer D. k RTRPO limits the update step size of the policy gradient through the conjugate gradient and linear search algorithm. The update formula is:

[0067]

[0068] where j∈{0,1,2…K} is a and The smallest value, For the value function, it is directly calculated by t The specific algorithm flow chart is as follows: Figure 5 shown.

[0069] The optimizer used is the ADAM optimizer.

[0070] The reward is whether the rat has completed the maze task. When at least one of the key points of the rat is away from the target position x g When the Euclidean distance between is less than the specified threshold, the reward value is 1, otherwise it is -1. Then the reward function can be written as:

[0071] R=+1,if((x m -x g )<σ));

[0072] R=-1, otherwise.

[0073] like Figure 6 As shown, it is a schematic diagram of the key posture points of the rat in the present invention, where points 1-8 represent the right hind limb, left hind limb, right forelimb, left forelimb, the connection point between the body and the base of the tail, the top of the head, the lower edge of the neck, and the posterior spine point of the rat, respectively, and point 9 represents the center of gravity of the rat.

Claims

1. A rat robot smooth control method based on transfer reinforcement learning, characterized in that: This is achieved through the following scheme: first, the environment model based on ELSTM is constructed, then the simulation maze platform and the simulation rat are built, followed by the RTRPO control algorithm simulation environment training, and finally the ELSTM and RTRPO are adapted online. The simulation maze platform is constructed based on the integrated environment model, and the restriction of the fluency measurement index is added to the environment model. The RTRPO control algorithm is used to complete the simulation training task; the ELSTM environment model and the RTRPO control algorithm are migrated to the real environment, and the ELSTM environment model is modified by interacting with the real environment. Then, virtual data is generated through the ELSTM environment model to help the RTRPO control algorithm quickly complete the online adaptation to obtain smooth control for the current rat robot; The input of the ELSTM environmental model is the key posture points of the real rat and the current stimulation instruction information, and the output is the key posture points of the real rat at the next moment after being stimulated. The ELSTM environmental model is trained using the L2 distance as the loss function. The key posture points are detected and extracted based on the third-person perspective overhead video, and the key points of the real rat robot in the video are detected and extracted by a difference-based background modeling method. Eight two-dimensional key points are set, including the head, the base of the tail, the left front paw, the left hind paw, the right front paw, the right hind paw, the midpoint of the neck, and the midpoint of the spine; The simulated maze platform uses MuJoCo software to build a simulated maze environment, and uses a trained ELSTM environment model to represent the changes in the position of the simulated rat after being stimulated; Based on the constructed simulated maze platform, an RTRPO control algorithm was trained under the restriction of fluency index to control the simulated rat robot to complete the maze task smoothly. The strategy function of the RTRPO control algorithm gave real-time instructions to control the rat behavior according to the key posture points of the simulated rat, so that the simulated rat robot could complete the maze task smoothly. The instructions were turn left, turn right, go forward, and stop.

2. A rat robot smooth control method based on transfer reinforcement learning as claimed in claim 1, characterized in that: Aiming at the problems of stiff action switching and unnatural movement behavior in controlling the rat robot, a smooth control index was introduced in the simulation environment training process to obtain a strategy that can achieve smooth control of the rat. Based on the abnormal motion characteristics of the rat robot, the difference in posture between adjacent frames and the frequency domain characteristics of motion speed were extracted as key indicators to measure the smoothness of its motion trajectory.

3. A rat robot smooth control method based on transfer reinforcement learning as claimed in claim 1, characterized in that: After migrating the trained ELSTM environment model and RTRPO control algorithm to the real environment for online adaptation, the real rat robot can complete the maze task smoothly.

Images