Energy-saving swimming gait learning method and device for bionic underwater robot based on reinforcement learning

Abstract

The present invention provides a method and device for learning energy-saving swimming gait of a bionic underwater robot based on reinforcement learning, which relates to the field of robotics technology, including: imitating the structure of a bionic robot fish, designing and constructing a reinforcement learning simulation environment based on slender body theory; constructing a Markov decision model based on the task objectives and motion state of the robot fish; and learning the efficient and energy-saving intermittent swimming gait of a fish-like vehicle based on reinforcement learning. The present invention can realize the efficient and energy-saving swimming gait learning of a robot fish based on deep reinforcement learning.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a reinforcement learning-based energy-saving swimming gait learning method and device for a bionic underwater robot. Background Art

[0002] At present, various types of bionic underwater robots have been widely used in the fields of ocean and military, such as marine life observation, underwater resource exploration, maritime military strikes, etc. Compared with traditional propeller-propelled underwater vehicles, bionic underwater robots have greater advantages in maneuverability, stability, anti-interference ability, noise, etc. Bionic underwater robots and their motion control methods have gradually attracted the attention of researchers and engineers.

[0003] During long-term underwater missions, bionic underwater robots are limited by the underwater environment and it is difficult to replenish energy. Therefore, it is of great significance to study the energy-saving swimming mode of bionic underwater robots. Swimming gait is very important for bionic underwater robots. Optimizing the swimming gait of bionic underwater robots to achieve energy-saving swimming has great application value.

[0004] There is currently no similar learning method for the learning of efficient and energy-saving swimming gaits through reinforcement learning of robot fish. At present, the industry is mainly developing three main aspects for the optimization of the swimming gait of robot fish. First, the parameters of the CPG central pattern generator of the robot fish are optimized based on the particle swarm optimization method to find the optimal parameter combination. However, this method can only be optimized under the prescribed swimming mode, and the intermittent swimming mode of the robot fish cannot be optimized. Secondly, computational fluid dynamics is used to perform fluid analysis on different artificially prescribed swimming gaits, and the swimming efficiency is analyzed from the aspects of resistance, hydrodynamics, and eddy currents. However, this method is very time-consuming and can only explore the prescribed swimming mode. In theory, it is impossible to find the optimal swimming gait. Summary of the invention

[0005] In order to solve the defects of the existing bionic underwater robot technology, such as high energy consumption for long-term operation, difficulty in modeling energy-saving swimming gait, and difficulty in optimizing swimming from swimming gait, the present invention provides a bionic underwater robot energy-saving swimming gait learning method and device based on reinforcement learning, which can autonomously explore the optimal swimming gait. Compared with the traditional CPG parameter optimization method, this method can explore more possibilities, find more abundant swimming modes, and have better energy-saving effect. At the same time, compared with the method using computational fluid dynamics, this method is less time-consuming.

[0006] Fish do not keep wagging their tails while swimming. Instead, they adopt a swimming method called intermittent swimming, which periodically combines high-speed tail wagging and gliding. Biologists have confirmed that this swimming method can greatly improve the swimming efficiency of robotic fish compared to traditional continuous swimming, and has a very strong reference significance for the research on efficient swimming of robotic fish.

[0007] Reinforcement learning is a widely used learning algorithm that does not rely on precise models. It optimizes control strategies through the interaction between the robot and the environment and has good environmental adaptability. Exploring the energy-saving swimming gait of bionic robot fish through reinforcement learning can effectively solve problems such as the difficulty in defining the energy-saving swimming effect and the difficulty in effectively modeling the gait.

[0008] In order to achieve the above object, the present invention adopts the following technical scheme:

[0009] A bionic underwater robot energy-saving swimming gait learning method based on reinforcement learning comprises the following steps:

[0010] Step 1: The bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory;

[0011] Step 2: Construct a Markov decision model based on the task objectives and motion state of the robot fish;

[0012] Step 3: Based on reinforcement learning, a learning framework is constructed, wherein the learning framework includes a simulation environment for reinforcement learning and a Markov decision model, so as to realize the efficient and energy-saving intermittent swimming gait learning of the bionic underwater robot.

[0013] Furthermore, step 1 includes: obtaining the motion parameters of the bionic underwater robot, inputting them into the central pattern generator of the bionic underwater robot, and obtaining the angle, angular velocity and angular acceleration of each joint thereof; inputting the obtained joint angle, angular velocity and angular acceleration into a dynamic model based on the slender body theory, and obtaining the energy consumption, force and motion state of the robot fish at this moment.

[0014] Furthermore, in step 1, the reinforcement learning simulation environment simulates the physical properties of the robot fish in the simulation, simulating the force conditions of the robot fish in different motion states; the input of the reinforcement learning simulation environment is the control parameters of the robot fish, including the three parameters of the frequency, amplitude, and intermittent factor of the central pattern generator, and the output is the force condition and position information of the robot fish; the reinforcement learning simulation environment constructs an interactive platform for simulating bionic robot fish in computer simulation, so that the reinforcement learning algorithm can be trained, greatly improving the simulation rate.

[0015] Furthermore, step 2 includes: by constructing an energy-saving swimming gait reward module, action space and state space, the motion state of the robot fish is input into the reward module in the simulation, and the optimal swimming gait of the robot fish is optimized, so as to find the optimal swimming gait of the bionic robot fish.

[0016] Furthermore, in step 3, the learning of efficient and energy-saving intermittent swimming gait of the bionic underwater robot is realized, including: imitating the efficient and energy-saving intermittent swimming behavior of fish in nature through the intermittent CPG module, constructing the intermittent swimming CPG control module of the robot fish, and controlling the swimming and gliding of the robot fish; the intermittent CPG module outputs the tail fin oscillation to realize the efficient swimming of the underwater robot.

[0017] Furthermore, the learning framework includes:

[0018] The energy-saving swimming gait optimization reward module inputs the motion state of the robot fish into the energy-saving swimming gait optimization reward module in the simulation, optimizes the optimal swimming gait of the robot fish, and thus finds the optimal swimming gait of the bionic robot fish.

[0019] The present invention also provides a bionic underwater robot energy-saving swimming gait learning device based on reinforcement learning, comprising the following modules:

[0020] In the simulation environment construction module, the bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory;

[0021] The decision model building module builds a Markov decision model based on the task objectives and motion status of the robot fish;

[0022] The gait learning module constructs a learning framework based on reinforcement learning. The learning framework includes a simulation environment for reinforcement learning and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning of a bionic underwater robot.

[0023] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method are implemented.

[0024] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method are implemented.

[0025] Beneficial effects:

[0026] The present invention can autonomously explore the optimal swimming gait of a bionic underwater robot, greatly reduce its swimming energy consumption, and perform more efficient and energy-saving swimming. By summarizing the swimming gait in the learning framework and deploying it on a bionic underwater robot, the energy consumption per unit distance of the bionic underwater robot is greatly reduced. The present invention has a very strong guiding significance for the efficient and energy-saving control of underwater robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a framework diagram of energy-saving swimming gait training provided by the present invention;

[0028] Figure 2 It is a schematic diagram of the network structure provided by the present invention;

[0029] Figure 3 It is a schematic diagram of the energy-saving motion learning method and efficient intermittent swimming strategy of the underwater robot of the present invention;

[0030] Figure 4 is an intermittent energy-saving swimming gait obtained through a reinforcement learning training framework; wherein (a) is a graph showing the frequency variation over time of the efficient intermittent energy-saving swimming gait obtained through the reinforcement learning framework, and (b) is a graph showing the joint angle variation over time of the bionic underwater robot of the efficient intermittent energy-saving swimming gait obtained through the reinforcement learning framework;

[0031] Figure 5 This is the verification result diagram of the robot motion control task.

[0032] The reference numerals are: a first hidden layer 210 , a second hidden layer 220 , and an output layer 230 . DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] like Figure 1 As shown, a bionic underwater robot energy-saving swimming gait learning method based on reinforcement learning of the present invention comprises the following steps:

[0035] Step 1, the bionic underwater robot is preferably a robotic fish, which imitates the structure of the bionic robotic fish and designs and constructs a reinforcement learning simulation environment based on the slender body theory;

[0036] The simulation environment can simulate the physical properties of the bionic robot fish in the simulation, and simulate the force conditions of the bionic robot fish in different motion states. The input of the simulation environment is the control parameters of the robot fish (the frequency, amplitude, and intermittent factor of the CPG central pattern generator), and the output is the force conditions and position information of the bionic robot fish. The role of the simulation environment is to build an interactive platform for simulating bionic robot fish in computer simulation, so that the reinforcement learning algorithm can be trained without being in reality, greatly improving the simulation rate.

[0037] Step 2: Construct a Markov decision model based on the task objectives and motion state of the robot fish. The Markov decision model includes the setting of reinforcement learning reward function and reinforcement learning algorithm.

[0038] Step 3: Based on reinforcement learning, a learning framework is constructed, wherein the learning framework includes a simulation environment for reinforcement learning and a Markov decision model, so as to realize the efficient and energy-saving intermittent swimming gait learning of the bionic underwater robot.

[0039] Specifically, in step 1, the reinforcement learning simulation environment consists of two parts, namely, a central pattern generator of the simulated underwater robot and a dynamic model of the simulated underwater robot. The central pattern generator can control each joint of the bionic underwater robot to perform periodic oscillation.

[0040] For the dynamic model, the robotic fish can be simplified into a series of n-link mechanisms, representing n-1 joints from the head to the tail of the robotic fish.

[0041] The coordinate transformation and speed transformation of the robot fish constitute the kinematic formula, which can be expressed as:

[0042] ;

[0043] in, The robot fish The fish body is segmented. R is the rotation transformation matrix of the robot fish, P is the length matrix, and T is the final joint transformation matrix. , V represents the velocity transformation matrix. The joint angle of each joint of the robot fish is recorded as , represents the yaw angle of the head. The length of each segment of the fish body is recorded as .

[0044] Using the Newton-Euler method, the dynamic formula is constructed, and the external force on each section of the fish body can be expressed as:

[0045] ;

[0046] in, Represents the force exerted by the previous section of the fish on this section of the fish. is the Coriolis force and centripetal force on each fish segment, It is The inertia matrix of the fish, It is The hydrodynamics of a fish. Representative of the robot fish The length of the center of mass of the connecting rod compared to the starting position of the connecting rod, Representative of the robot fish The mass of the connecting rod, Representative The moment of inertia of the joints, Representative The superscript T indicates the transpose of the matrix. Represents the 3×3 identity matrix. is the unit vector of each joint of the robot fish along the x direction.

[0047] Based on the above theory, the present invention establishes a dynamic model of the robotic fish, as shown in the following formula:

[0048] ;

[0049] in, It is the mapping relationship of the dynamic model. and Represents the speed of the robot fish head unit at this moment t and the next moment t+1; variable and Represents the angle of each joint between this moment t and the next moment t+1, Represents the total external force acting on each segment of the robot fish.

[0050] Specifically, the step 2 includes:

[0051] Construct a reinforcement learning Markov decision model for energy-saving swimming gait learning. At each time step, the observed state obtained from the environment Defined as:

[0052] ;

[0053] in, is the observation state of reinforcement learning at time t, Indicates the deviation angle from the specified direction, represents the three-axis speed in the robot coordinate system, Indicates the joint angles of the robot fish. Indicates action.

[0054] For each observed state , the policy network generates an action , corresponding to the three parameters of the CPG (central pattern generator) network, namely frequency F, amplitude M, and intermittent factor λ. Figure 3 In the figure, B is the bias and Φ is the phase difference.

[0055] For the energy-saving swimming gait learning task, define a dense reward function :

[0056] ,

[0057] ;

[0058] in, is the weight of each sub-reward, i=1,2,3,4,5, represents the speed of the robot fish at time t, and Represent the torque and angular velocity of each joint at time t, Reward the robot fish to swim forward at a certain speed; Reward the robot fish to move in the specified direction and maintain the yaw angle; Reward the robot fish for swimming with low energy consumption; The robot fish will be punished for its lazy behavior and encouraged to explore freely; Smooth motion of the robot fish will be encouraged, and exp represents the exponential function. is the deviation angle between the current heading angle of the robot fish and the target heading angle, and the superscript t represents the time t.

[0059] Specifically, the step 3 includes:

[0060] After building the simulation environment and Markov decision model, we can start learning the energy-saving swimming gait. The input is sent to the pre-built simulation environment, and the driving control model in the simulation environment outputs the frequency, offset, and phase difference of the robot's tail fin swing. Input the dynamics model, kinematics model, and Markov decision model to get the reward for the current time step , and the robot's next time step state . And determine the next time step state as the current time step state.

[0061] Figure 2 It is a structural diagram of the initial residual model provided by the present invention, such as Figure 2As shown, the initial residual model is a multi-layer neural network, which is composed of a first hidden layer 210, a second hidden layer 220 and an output layer 230, wherein the first hidden layer 210 contains 800 neurons, the second hidden layer 220 contains 600 neurons, and the output layer 230 contains 3 neurons. The ReLU function is used as the hidden layer activation function, and the Tanh function is used as the output layer activation function. The current motion state and task goal of the robot are input into the initial residual model to obtain the second control action output by the initial residual model.

[0062] The experience array Store in the experience pool. Randomly extract arrays from the experience pool for offline training of the policy network of the initial residual model of the Soft Actor-Critic (SAC) reinforcement learning algorithm , until the training is completed and the optimal energy-saving swimming gait of the robot fish is obtained.

[0063] Figure 3 This is a schematic diagram of the CPG central pattern generator that imitates fish in nature. The intermittent swimming behavior of fish includes a burst phase and a gliding phase. The switching between burst and gliding is achieved through the intermittent factor.

[0064] Figure 4 The figure is a schematic diagram of the swimming gait obtained by the robot fish through efficient and energy-saving swimming. Figure 4 (a) is a graph showing the frequency variation over time of the efficient intermittent energy-saving swimming gait obtained through the reinforcement learning framework. Figure 4 (b) is a graph showing the variation of joint angles over time for a bionic underwater robot with an efficient intermittent energy-saving swimming gait obtained through a reinforcement learning framework.

[0065] To verify the effectiveness of the obtained energy-saving swimming gait, e.g. Figure 5 As shown, the robot motion control task verification can be carried out in an indoor pool of 5m×4m×1.1m. The energy consumption measurement module installed inside the robot fish can measure the energy consumption information of the robot fish in real time. It can be seen from the verification results that compared with the traditional continuous swimming method of existing robot fish and the energy-saving swimming method obtained by reinforcement learning, the method provided by the embodiment of the present invention can enable the underwater robot to save an average of 24.2% of energy consumption while maintaining the same swimming speed.

[0066] The present invention also provides a bionic underwater robot energy-saving swimming gait learning device based on reinforcement learning, comprising the following modules:

[0067] In the simulation environment construction module, the bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory;

[0068] The decision model building module builds a Markov decision model based on the task objectives and motion status of the robot fish;

[0069] The gait learning module constructs a learning framework based on reinforcement learning. The learning framework includes a simulation environment for reinforcement learning and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning of a bionic underwater robot.

[0070] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method are implemented.

[0071] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method are implemented.

[0072] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0073] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0077] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A bionic underwater robot energy-saving swimming gait learning method based on reinforcement learning, characterized in that: The steps include: Step 1: The bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory; The reinforcement learning simulation environment simulates the physical properties of the robotic fish in the simulation, simulating the force conditions of the robotic fish under different motion states; The input of the reinforcement learning simulation environment is the control parameters of the robot fish, including the frequency, amplitude, and intermittent factor of the central pattern generator, and the output is the force and position information of the robot fish. The reinforcement learning simulation environment builds an interactive platform for simulating bionic robotic fish in computer simulation, allowing the reinforcement learning algorithm to be trained, greatly improving the simulation rate; Step 2: Construct a Markov decision model based on the mission objectives and motion state of the robot fish, including: By constructing an energy-saving swimming gait reward module, action space, and state space, the motion state of the robot fish is input into the reward module in the simulation, and the optimal swimming gait of the robot fish is optimized, thereby finding the optimal swimming gait of the bionic robot fish; Step 3: Based on reinforcement learning, a learning framework is constructed, wherein the learning framework includes a simulation environment for reinforcement learning and a Markov decision model, so as to realize the efficient and energy-saving intermittent swimming gait learning of the bionic underwater robot; Realizing efficient and energy-saving intermittent swimming gait learning of bionic underwater robots includes: imitating the efficient and energy-saving intermittent swimming behavior of fish in nature through the intermittent CPG module, building the intermittent swimming CPG control module of the robot fish, and controlling the swimming and gliding of the robot fish; the intermittent CPG module outputs tail fin oscillation to realize efficient swimming of the underwater robot.

2. According to the reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method of claim 1, it is characterized in that: The step 1 includes: obtaining the motion parameters of the bionic underwater robot, inputting them into the central pattern generator of the bionic underwater robot, and obtaining the angle, angular velocity and angular acceleration of each joint thereof; inputting the obtained joint angle, angular velocity and angular acceleration into a dynamic model based on the slender body theory, and obtaining the energy consumption, force and motion state of the robot fish at this moment.

3. The energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning according to claim 1 is characterized in that: The learning framework includes: The energy-saving swimming gait optimization reward module inputs the motion state of the robot fish into the reward module in the simulation, optimizes the optimal swimming gait of the robot fish, and thus finds the optimal swimming gait of the bionic robot fish.

4. A bionic underwater robot energy-saving swimming gait learning device based on reinforcement learning that implements the bionic underwater robot energy-saving swimming gait learning method based on reinforcement learning as described in claim 1, characterized in that: Includes the following modules: In the simulation environment construction module, the bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory; The decision model building module builds a Markov decision model based on the task objectives and motion status of the robot fish; The gait learning module constructs a learning framework based on reinforcement learning. The learning framework includes a simulation environment for reinforcement learning and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning of a bionic underwater robot.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning are implemented as described in any one of claims 1-3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the reinforcement learning-based energy-saving swimming gait learning method of a bionic underwater robot as described in any one of claims 1 to 3 are implemented.

Images