River-imitated robot deep reinforcement learning control method

Abstract

The invention relates to the field of robot control, aims to overcome the defects in the background technology, and provides a masked palm robot-imitating deep reinforcement learning control method so as to shorten the training time, improve the learning efficiency and improve the learning effect. According to the technical scheme, the method comprises the following steps of 1, constructing a D3QN model; 2, setting a space state and a reward function; step 3, acquiring a bionic swimming track of the rear foot joint of the masked civets; 4, the bionic swimming robot is controlled to move according to the bionic swimming track; and step 5, training the D3QN model. According to the invention, appropriate actions can be quickly learned in a real underwater environment, the environment is better adapted, the training efficiency and effect are also improved, and experimental results show that the robot can reach a stable swimming state in synchronous and staggered bionic swimming modes.

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to a deep reinforcement learning control method for a beaver-like robot. Background Art

[0002] The paddling of the underwater webbed biped robot will cause the water flow field to change continuously, resulting in unbalanced hydrodynamic forces that constantly change the robot's posture. Since the water environment and the robot swimming are nonlinear and strongly coupled systems, it often requires complex mathematical operations to establish a hydrodynamic model of the controlled object, which makes modeling difficult and is not conducive to the development of underwater robots.

[0003] Artificial intelligence methods provide a solution. Model-free reinforcement learning algorithms can control and train the robot's motion through environmental feedback without building a hydrodynamic model of the controlled object, thereby achieving the robot's adaptive effect. This is similar to how a human baby learns to walk, learning through trial and error. However, since the algorithm itself has a certain degree of randomness, it is dangerous to the robot body and experimenters, and has problems such as long training time, low learning efficiency, and poor learning effect. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the above-mentioned background technology and provide a deep reinforcement learning control method for a beaver-like robot to shorten the training time, improve the learning efficiency and enhance the learning effect.

[0005] The technical solution of the present invention is:

[0006] The deep reinforcement learning control method of the beaver-like robot includes the following steps:

[0007] Step 1: Build the D3QN model;

[0008] Step 2: Set the space state and reward function;

[0009] Step 3: Obtain the bionic swimming trajectory of the beaver's hind foot joint;

[0010] Step 4: Control the movement of the bionic swimming robot according to the bionic swimming trajectory;

[0011] Step 5: Train the D3QN model.

[0012] The neural network of the D3QN model includes a value network and a target network; the value network and the target network both include an input layer, a hidden layer, and an output layer.

[0013] In the step 2,

[0014] State Space :

[0015]

[0016] Where: Motor angle , which represents the real-time angle of the robot's six joint motors, which are the six motors of the two hind legs; the real-time pitch angle , indicating the real-time pitch angle change of the robot during movement; the training round sign is jumped out , indicating that the robot jumps out of the current training set; reaching the expected goal mark , indicating whether the robot can stay within the desired pitch angle through 5 steps;

[0017] Reward function for synchronized swimming :

[0018]

[0019] Staggered swimming pattern :

[0020]

[0021] in: Represents the synchronous swimming reward coefficient; The reward value representing the difference between the current pitch angle and the desired pitch angle; The reward value indicating whether the expected condition is met; Represents the reward value compared with the average historical reward value; Represents the reward value compared with the historical pitch angle; Indicates the reward value for comparing the speed of pulling the leg forward with the speed of pulling the leg backward during synchronous movement, to encourage the robot to pull the leg backward faster than the speed of pulling the leg forward during synchronous swimming; Represents the staggered swimming reward coefficient.

[0022] In the step 2,

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] in: The reward weight coefficient representing the reward value of the difference between the current pitch angle and the expected pitch angle; The reward weight coefficient indicating whether the reward value meets the expected condition; The reward weight coefficient representing the reward value compared with the mean of historical reward values; The reward weight coefficient representing the reward value compared with the historical pitch angle; The reward weight coefficient representing the reward value of comparing the forward leg retraction speed with the backward stroke speed during synchronous movement; Indicates the current pitch angle; Indicates the pitch angle of the expected target; Represents the historical average pitch angle; Indicates that the expected condition is met, the value is 0 or 1; Indicates the backward stroke time; Indicates forward stroke time; Indicates the current reward value; Represents the historical reward value.

[0031] The step 3 comprises:

[0032] 3.1) Observe the video of beavers swimming in nature and obtain the discrete trajectory points of the ankle joint during swimming ;

[0033] 3.2) Based on the discrete trajectory points, use the B-spline function to fit the ankle joint trajectory of the beaver when swimming;

[0034]

[0035] 3.3) Establish the inverse kinematics analysis of the hind leg joints of the bionic swimming robot;

[0036] 3.3.1) Establish the geometric relationship diagram of the hindfoot motion joints of the bionic robot: hip joint angle , knee angle , ankle joint angle , Footpath length , Thigh length , Thigh length , Thigh length ;

[0037] 3.3.2) Discrete trajectory points of the ankle joint of the bionic swimming robot in the hip joint coordinate system , establish inverse kinematics analysis.

[0038] The inverse kinematics analysis of step 3.3.2) includes:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] .

[0045] The step 4 comprises:

[0046] On the basis of obtaining the bionic swimming trajectory of the beaver's hind foot joint, the discrete swimming action sequence number is output, which is output as the bionic swimming trajectory by the action mapping function to control the movement of the bionic swimming robot;

[0047] The action mapping function is:

[0048]

[0049]

[0050]

[0051]

[0052] in: represents the action space; The action representing the best performing trajectory; represents the target action obtained; represents the initial angles of the six motors of the two hind legs during synchronous swimming; Represents the initial angles of the six motors of the two hind legs during staggered swimming.

[0053] The beneficial effects of the present invention are:

[0054] The present invention utilizes a deep reinforcement learning method and the swimming trajectory of beavers in nature to effectively solve the nonlinear and strong coupling problems of an underwater webbed bipedal swimming robot when swimming in water. The robot has a certain posture adjustment capability in both swimming modes. In the synchronous mode, the floating range of the pitch angle can be stabilized within the interval of [-5,5]. In the staggered swimming mode, the floating range of the pitch angle can be stabilized within the interval of [-3,3]. The robot has a certain adaptive capability and the training process has a certain safety. The robot has the advantages of short training time, high efficiency and good learning effect, and provides an effective solution for the intelligent movement of underwater robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a framework diagram of the present invention.

[0056] Figure 2 It is a synchronous swimming experiment diagram of the present invention.

[0057] Figure 3 It is a staggered swimming experimental diagram of the present invention.

[0058] Figure 4a It is a geometric relationship diagram of the hind foot motion joints of the bionic swimming robot of the present invention.

[0059] Figure 4b It is a geometric relationship diagram of the hind foot motion joints of the bionic swimming robot of the present invention.

[0060] Figure 5 It is a schematic diagram of swimming training actions at 10 action angles of the bionic swimming robot of the present invention.

[0061] Figure 6 It is a graph of the loss function during training of the present invention.

[0062] Figure 7 It is a comparison diagram of the pitch angle effect of the swimming of the present invention and the general swimming situation. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] The deep reinforcement learning control method of the beaver-like robot includes the following steps:

[0065] Step 1: Build the D3QN model

[0066] The D3QN model (Dueling Double Deep Q-learning) is an improvement based on the DQN model. It introduces Double DQN and Dueling DQN technologies to improve model stability and performance.

[0067] The D3QN model mainly consists of two neural networks. The D3QN model fits the action-value function through the neural network and selects actions based on the action-value function output by the neural network.

[0068] like Figure 1As shown in the figure, the two neural networks of the D3QN model are the value network and the target network. The value network is responsible for real-time training and updating and outputting the maximum Q value. The target network is updated with a delay of 80 steps and outputs the target Q value, which is used to calculate the loss function of the neural network and train the network.

[0069] The target network and the value network have the same structure, both using Dueling networks, including a three-layer network structure, with 10 neurons in the input layer, 128 neurons in the hidden layer, and an output layer through the action function And the action value function calculate , There are 10 neurons. There is 1 neuron, There are 10 neurons, and the activation function uses the Relu linear activation function.

[0070] Output action function It is a 10-dimensional action vector, including the real-time angles of the seven joint motors of the bionic swimming robot, and the output action value function It is also a 10-dimensional action vector, containing the Q value information of 10 action spaces. It is expressed by formula (1):

[0071] (1)

[0072] The target network and the value network use the same structure to facilitate parameter assignment. At the beginning of training, the parameters of the two networks are consistent. Every once in a while, the value network assigns the parameters to the target network. As the algorithm runs, the Q values ​​calculated by the two networks differ. At this time, the error between the maximum Q value and the target Q value is calculated, and the stochastic gradient descent method is used to minimize the error and train the value network parameters.

[0073] In the initial state, the agent will be in the state space Input the value network, and output the value of each action after calculation through the value network value, and then use The policy randomly chooses an action Execution and interaction with the environment are rewarded . Strategy refers to An action is randomly selected with probability The probability of selecting the action corresponding to the maximum Q value calculated by the current value network.

[0074] Step 2: Set up space state and reward function

[0075] 2.1) Set space status

[0076] State Space Including: Motor angle , Real-time pitch angle , jump out of the training round sign , achieve the expected goal sign .

[0077] State Space It is expressed by formula (2):

[0078] (2)

[0079] Where: Motor angle , represents the real-time angle of the robot's six joint motors, which are the six motors of the two hind legs (hip joint motor, knee joint motor, ankle joint motor); real-time pitch angle , indicating the real-time pitch angle change of the robot during movement; the training round sign is jumped out , indicating that the robot jumps out of the current training set; reaching the expected goal mark , indicating whether the robot can stay within the desired pitch angle through 5 steps.

[0080] 2.2) Setting the reward function

[0081] Synchronous swimming mode:

[0082] According to the training objectives of the underwater bionic legged robot, the reward function of the robot in the synchronous swimming mode is set to five parts, whose values ​​are the products of the synchronous swimming weight matrix and the corresponding reward value matrix.

[0083] Reward Function It is expressed by formula (3):

[0084] (3)

[0085] in: Represents the synchronous swimming reward coefficient; The reward value representing the difference between the current pitch angle and the desired pitch angle; The reward value indicating whether the expected condition is met; Represents the reward value compared with the average historical reward value; Represents the reward value compared with the historical pitch angle; Represents the reward value for comparing the speed of pulling the leg forward with the speed of paddling backward during synchronous motion, to encourage the robot to paddling backward faster than paddling forward during synchronous swimming.

[0086] Staggered swimming mode:

[0087] According to the training objectives of the underwater bionic legged robot, the reward function of the robot in the interlaced swimming mode is set to four parts, whose values ​​are the products of the alternating swimming weight matrix and the corresponding reward value matrix.

[0088] Reward Function The number is expressed by formula (4):

[0089] (4)

[0090] in: represents the staggered swimming reward coefficient; The reward value representing the difference between the current pitch angle and the desired pitch angle; The reward value indicating whether the expected condition is met; Represents the reward value compared with the average historical reward value; Represents the reward value compared with the historical pitch angle.

[0091] in addition:

[0092] (5)

[0093] (6)

[0094] (7)

[0095] (8)

[0096] (9)

[0097] (10)

[0098] (11)

[0099] in: The reward weight coefficient representing the reward value of the difference between the current pitch angle and the expected pitch angle; The reward weight coefficient indicating whether the reward value meets the expected condition; The reward weight coefficient representing the reward value compared with the mean of historical reward values; The reward weight coefficient representing the reward value compared with the historical pitch angle; The reward weight coefficient representing the reward value of comparing the forward leg retraction speed with the backward stroke speed during synchronous movement; Indicates the current pitch angle; Indicates the pitch angle of the expected target; Represents the historical average pitch angle; Indicates that the expected condition is met, the value is 0 or 1; Indicates the backward stroke time; Indicates forward stroke time; Indicates the current reward value; Represents the historical reward value.

[0100] Step 3: Obtain the bionic swimming trajectory of the beaver's hind foot joints

[0101] Based on the beaver swimming trajectory in nature, the learning of the bionic swimming robot is controlled. The fitting steps are as follows:

[0102] 3.1) Observe the video of beavers swimming in nature, take the beaver hip joint as the origin, establish the hip joint coordinate system, denoted as {O1}, and obtain the discrete trajectory points of the ankle joint during swimming .

[0103] 3.2) Based on the discrete trajectory points, use the B-spline function to fit the ankle joint trajectory of the beaver when swimming.

[0104] B-spline function is a function developed on Bezier curve, which overcomes the inconvenience caused by the overall controllability of the curve and improves the two shortcomings of the curve that cannot be partially modified and the curve splicing is relatively complicated. This curve has many differences from Bezier curve and polynomial curve. First, B-spline function can be processed in sections to fit a smooth trajectory curve; secondly, cubic B-spline function fitting can achieve the effect of adjusting a certain section of the function separately without affecting the characteristics of other parts of the curve.

[0105] The total equation of the B-spline function is:

[0106] (12)

[0107] The basis functions of the B-spline function are:

[0108] (13)

[0109] Formulas (12) and (13) are existing formulas.

[0110] The trajectory simulation is performed using a fourth-order cubic B-spline function, and its component formula is:

[0111] (14)

[0112] (15)

[0113] (16)

[0114] 3.3) Establish inverse kinematics analysis of the hind leg joints of a bionic swimming robot

[0115] 3.3.1) Establish the geometric relationship diagram of the hind leg motion joints of the bionic robot

[0116] Based on the kinematic coordinate system and motion analysis of the bionic swimming robot, the geometric solution of inverse kinematics is used to establish a bionic swimming robot.

[0117] Geometric diagram of the robot's hindfoot kinematic joints.

[0118] Based on the study of biological beavers, beavers mainly rely on their hind limbs to swim in the water, so the bionic swimming robot divides the hind feet into hip joints, knee joints, ankle joints, and passive bending joints (webbed feet). When the hind feet move forward, the webbed feet bend, and when the hind feet paddle backward, the webbed feet straighten.

[0119] The present invention establishes a coordinate system at each joint of the bionic swimming robot, and uses the DH parameter method to formulate the kinematic equations of the hind limbs of the bionic swimming robot. A global coordinate system is established at the center of gravity of the bionic swimming robot, which is marked as {O0}. A hip joint coordinate system is established at the hip joint of the bionic swimming robot, which is marked as {O1}. A knee joint coordinate system is established at the knee joint of the bionic swimming robot, which is marked as {O2}. An ankle joint coordinate system is established at the ankle joint of the bionic swimming robot, which is marked as {O3}.

[0120] like Figure 4a , Figure 4b As shown, in the bionic swimming robot: hip joint angle , knee angle , ankle joint angle , Footpath length , Thigh length , Thigh length , Thigh length , the rear foot extends during the backstroke phase so the instep length Increase, the hind foot contracts and bends during the forward leg-drawing phase, so the instep length Reduce.

[0121] The DH parameters of the hind foot kinematics of the bionic swimming robot are shown in Table 1:

[0122] Table 1

[0123] Joints <![CDATA[α i-1 ]]> <![CDATA[a i-1 ]]> <![CDATA[d i ]]> <![CDATA[θ i ]]> 1 0 <![CDATA[l1]]> 0 <![CDATA[θ hip <!-- 6 -->]]> 2 0 <![CDATA[l2]]> 0 <![CDATA[θ knee ]]> 3 0 <![CDATA[l3]]> 0 <![CDATA[θ ankle ]]>

[0124] 3.3.2) Discrete trajectory points of the ankle joint of the bionic swimming robot in the hip joint coordinate system , establish the inverse kinematics analysis, expressed by equations (17)-(22):

[0125] (17)

[0126] (18)

[0127] (19)

[0128] (20)

[0129] (twenty one)

[0130] (twenty two)

[0131] in, is a constant.

[0132] Step 4: Control the movement of the bionic swimming robot according to the bionic swimming trajectory

[0133] On the basis of obtaining the bionic swimming trajectory of the beaver's hind foot joint, the discretized swimming action sequence number is output, which is output as the bionic swimming trajectory by the action mapping function to control the movement of the bionic swimming robot.

[0134] The action mapping function is expressed by equations (23)-(26):

[0135] (twenty three)

[0136] (twenty four)

[0137] (25)

[0138] (26)

[0139] in: represents the action space; The action representing the best performing trajectory; represents the target motion obtained (the movement angles of the six motors of the two hind legs); represents the initial angles of the six motors of the two hind legs during synchronous swimming; represents the initial angles of the six motors of the two hind legs during staggered swimming; Formula (23) represents the swimming action sequence of the motor, such as Figure 5 As shown in Figure 2, when the D3QN model outputs the swimming action number, the robot calculates the motion trajectory according to the swimming action number and formula (24) to adapt to environmental changes in real time and improve the robot's biological swimming characteristics and training efficiency; the factors selected in formulas (25) and (26) are related to the robot's body settings.

[0140] Step 5: Train the D3QN model

[0141] At the beginning of training, the bionic swimming robot is in the initial state. The pitch sensor obtains the current pitch angle of the bionic swimming robot and inputs the pitch angle into the D3QN model. After the D3QN model is calculated, the target network outputs the target Q value and the swimming action sequence number at the next moment, calculates the reward value under the current action, and maps it to the bionic swimming robot through the action mapping function. The two hind legs of the bionic swimming robot move and swimming begins. In the subsequent process, the pitch sensor obtains the pitch angle of the bionic swimming robot at the current moment, and transmits the state of the next moment to the bionic swimming robot to calculate the reward value under the current action.

[0142] In order to simplify the training, the training rounds of the bionic swimming robot are set to 100 times, and each round has 100 steps. Figure 6 As shown in the figure, after about 1 hour of training, the D3QN model gradually converges, and the bionic swimming robot learns swimming movements suitable for the current environment, which greatly improves the training efficiency of the bionic swimming robot.

[0143] The following is a detailed description of the process of training the D3QN model:

[0144] In the initialization algorithm stage, set the size D of the experience replay library Replay-buffer, the parameters w of the current network, the parameters w- of the target network, the size N of the data sampling mini-batch, and the update cycle Nr of the target network.

[0145] During the algorithm training phase, the D3QN model is based on the current state use Strategy takes action; the action output by the inverse kinematics solver D3QN model is based on the action amplitude Solve the robot's swimming elliptical trajectory, change the coordinates, solve the robot's hip joint angle, knee joint angle, ankle joint angle when swimming, and input them into the robot. The robot controls the motors of the three joints to move to the corresponding angles to achieve interaction between the robot and the environment; finally, return the state after the action, the reward value, and the round end flag d, forming a data tuple Save to the experience replay database Replay-buffer.

[0146] When the amount of sampled data in the Replay-buffer reaches the requirement for training the D3QN model, training begins. First, a mini-batch of data tuples is sampled from the experience replay database Replay-buffer. The value network is used in the D3QN model to estimate the next state. The action of constructing the loss function, calculating the loss function, performing back propagation, and updating the parameters of the current network. Finally, when the number of training times reaches the target network number requirement, the target network is updated.

[0147] The present invention realizes the stable pitch posture swimming control of the bionic swimming robot based on the combination of the synchronous and staggered swimming gait curves of beavers and the reinforcement learning algorithm.

[0148] The attitude stabilization control of the bionic swimming robot is divided into two parts: first, the motion decision layer of the bionic swimming robot uses the D3QN model to determine the current input state of the bionic swimming robot. , output action number ;

[0149] The second is the motion control layer of the bionic swimming robot. The bionic swimming robot moves according to the decision , bionic swimming trajectory and action mapping function, mapping the action sequence number into bionic synchronous and staggered swimming gaits, making the joints of the bionic swimming robot execute actions, the bionic swimming robot performs swimming actions, and at the same time generates the next state of the bionic swimming robot and inputs it into the D3QN model; repeating the above steps, the bionic swimming robot can achieve stable and continuous swimming in water.

[0150] The bionic swimming robot is an existing device, including a trunk, bionic hind feet, a controller, and a pitch sensor. The hind feet generate power when they move, and the pitch sensor is used to detect the pitch angle of the trunk. When swimming synchronously, Figure 2 As shown in the figure, the two hind legs move forward or backward at the same time, keeping the same trajectory. Figure 2 As shown, one hind foot moves forward (pulling the leg) and the other hind foot moves backward (paddling).

[0151] Experimental verification

[0152] The present invention sets up synchronous and staggered swimming experiments of beavers to verify the stability of the pitch posture of the bionic swimming robot when swimming. Because the robot's environment is partially observable, and the robot's swimming environment is highly random, the loss value of the deep network has large fluctuations, but it is generally convergent. We use the sliding average method to smooth the loss function curve with a sliding factor of 0.9. As shown in Figure 6, the loss function is stable at about the 500th training. In terms of the effect of stable swimming in pitch posture, the synchronous swimming and staggered swimming of the present invention can basically achieve stable swimming within the desired pitch angle, but compared with the two, the staggered swimming has a better learning effect because the two legs swing alternately during swimming, which is itself compared with the synchronous swimming method in which the two legs paddle backward at the same time;

[0153] As attached Figure 7 As shown, in order to verify the pitch attitude angle stabilization control effect of the present invention, the pitch angles of the bionic swimming robot are compared in 100 swimming test steps.

[0154] In normal swimming conditions: in synchronous swimming mode, the pitch angle fluctuates between [-3,26]; in staggered swimming mode, the pitch angle is between [-1,16]. The swimming pitch angle fluctuates to a certain extent and cannot adapt to the dynamically changing water flow environment.

[0155] When the present invention swims: in the synchronous swimming strategy, its pitch angle is stable between [-5,5]; in the staggered swimming strategy, its pitch angle is stable between [-3,3], and in the swimming test process, it maintains a relatively stable state. The general swimming situation refers to the biological movement trajectory in nature.

[0156] The experimental results show that the present invention can effectively and stably control the pitch angle of the beaver-like biped swimming robot, and can adapt to the environment according to the change of the water flow environment to achieve stable control of the pitch angle.

[0157] The preferred embodiments of the present invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

Claims

1. A deep reinforcement learning control method for a beaver-like robot, comprising the following steps: Step 1: Build the D3QN model; Step 2: Set the space state and reward function; Step 3: Obtain the bionic swimming trajectory of the beaver's hind foot joint; Step 4: Control the movement of the bionic swimming robot according to the bionic swimming trajectory; Step 5: Train the D3QN model.

2. The beaver-like robot deep reinforcement learning control method according to claim 1, characterized in that: The neural network of the D3QN model includes a value network and a target network; the value network and the target network both include an input layer, a hidden layer, and an output layer.

3. The beaver-like robot deep reinforcement learning control method according to claim 1, characterized in that: In the step 2, State Space : ， Where: Motor angle , which represents the real-time angle of the robot's six joint motors, which are the six motors of the two hind legs; the real-time pitch angle , indicating the real-time pitch angle change of the robot during movement; the training round sign is jumped out , indicating that the robot jumps out of the current training set; reaching the expected goal mark , indicating whether the robot can stay within the desired pitch angle through 5 steps; Reward function for synchronized swimming : , Staggered swimming pattern : , in: Represents the synchronous swimming reward coefficient; The reward value representing the difference between the current pitch angle and the desired pitch angle; The reward value indicating whether the expected condition is met; Represents the reward value compared with the average historical reward value; Represents the reward value compared with the historical pitch angle; Indicates the reward value for comparing the speed of pulling the leg forward with the speed of pulling the leg backward during synchronous movement, to encourage the robot to pull the leg backward faster than the speed of pulling the leg forward during synchronous swimming; Represents the staggered swimming reward coefficient.

4. The beaver-like robot deep reinforcement learning control method according to claim 3 is characterized in that: In the step 2, ， ， ， ， ， ， ， in: The reward weight coefficient representing the reward value of the difference between the current pitch angle and the expected pitch angle; The reward weight coefficient indicating whether the reward value meets the expected condition; The reward weight coefficient representing the reward value compared with the mean of historical reward values; The reward weight coefficient representing the reward value compared with the historical pitch angle; The reward weight coefficient representing the reward value of comparing the forward leg retraction speed with the backward stroke speed during synchronous movement; Indicates the current pitch angle; Indicates the pitch angle of the expected target; Represents the historical average pitch angle; Indicates that the expected condition is met, the value is 0 or 1; Indicates the backward stroke time; Indicates forward stroke time; Indicates the current reward value; Represents the historical reward value.

5. The beaver-like robot deep reinforcement learning control method according to claim 1, characterized in that: The step 3 comprises: 3.1) Observe the video of beavers swimming in nature and obtain the discrete trajectory points of the ankle joint during swimming ; 3.2) Based on the discrete trajectory points, use the B-spline function to fit the ankle joint trajectory of the beaver when swimming. ； 3.3) Establish inverse kinematics analysis of the hind leg joints of the bionic swimming robot; 3.3.1) Establish the geometric relationship diagram of the hind foot motion joints of the bionic robot: hip joint angle , knee angle , ankle joint angle , Footpath length , Thigh length , Thigh length , Thigh length ; 3.3.2) Discrete trajectory points of the ankle joint of the bionic swimming robot in the hip joint coordinate system , establish inverse kinematics analysis.

6. The beaver-like robot deep reinforcement learning control method according to claim 5, characterized in that: The inverse kinematics analysis of step 3.3.2) includes: ， ， ， ， ， 。 7. The beaver-like robot deep reinforcement learning control method according to claim 1, characterized in that: The step 4 comprises: On the basis of obtaining the bionic swimming trajectory of the beaver's hind foot joint, the discrete swimming action sequence number is output, which is output as the bionic swimming trajectory by the action mapping function to control the movement of the bionic swimming robot; The action mapping function is: ， ， ， ， in: represents the action space; The action representing the best performing trajectory; represents the target action obtained; represents the initial angles of the six motors of the two hind legs during synchronous swimming; Represents the initial angles of the six motors of the two hind legs during staggered swimming.

Images