Bionic underwater robot energy-saving swimming gait learning method and device based on reinforcement learning

Abstract

The invention provides an energy-saving swimming gait learning method and device for a bionic underwater robot based on reinforcement learning, and relates to the technical field of robots, and the method comprises the steps: simulating the structure of a bionic robotic fish, and designing and constructing a reinforcement learning simulation environment based on a slender body theory; constructing a Markov decision model based on the task target and the motion state of the robotic fish; and based on reinforcement learning, learning the efficient and energy-saving intermittent swimming gait of the fish-imitating aircraft of a framework. According to the invention, efficient and energy-saving swimming gait learning of the robotic fish based on deep reinforcement learning can be realized.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method and apparatus for learning the energy-saving swimming gait of a biomimetic underwater robot based on reinforcement learning. Background Technology

[0002] Currently, various biomimetic underwater robots have been widely used in marine and military fields, such as marine life observation, underwater resource exploration, and maritime military operations. Compared with traditional propeller-driven underwater vehicles, biomimetic underwater robots have greater advantages in terms of maneuverability, stability, anti-interference ability, and noise reduction. Biomimetic underwater robots and their motion control methods are gradually attracting the attention of researchers and engineers.

[0003] During extended underwater missions, biomimetic underwater robots face limitations in energy replenishment due to the underwater environment. Therefore, research on energy-efficient swimming methods for biomimetic underwater robots is of great significance. Swimming gait is crucial for biomimetic underwater robots, and optimizing their gait to achieve energy-efficient swimming has significant application value.

[0004] Currently, there is no similar learning method for learning efficient and energy-saving swimming gaits for robotic fish using reinforcement learning. Current industry research on optimizing robotic fish swimming gaits mainly focuses on three aspects. First, it uses particle swarm optimization (PSO) to optimize the parameters of the robotic fish's central pattern generator (CPG) to find the optimal parameter combination. However, this method can only optimize under specified swimming patterns and cannot optimize the intermittent swimming patterns of robotic fish. Second, it uses computational fluid dynamics (CFD) to perform fluid analysis on different artificially defined swimming gaits, analyzing swimming efficiency from aspects such as resistance, hydrodynamics, and eddies. However, this method is very time-consuming and can only explore specified swimming patterns, theoretically failing to find the optimal swimming gait. Summary of the Invention

[0005] To address the shortcomings of existing biomimetic underwater robot technologies, such as high energy consumption during long-term operation, difficulty in modeling energy-efficient swimming gait, and difficulty in optimizing swimming movement based on gait, this invention provides a reinforcement learning-based method and device for learning energy-efficient swimming gait in biomimetic underwater robots. This method can autonomously explore the optimal swimming gait. Compared to traditional methods that optimize CPG parameters, this approach can explore more possibilities, find a wider variety of swimming patterns, and achieve better energy efficiency. Furthermore, compared to methods using computational fluid dynamics, this approach is less time-consuming.

[0006] Fish do not continuously wag their tails while swimming. Instead, they employ a swimming technique called intermittent swimming, periodically combining high-speed tail wagging with gliding. Biological studies have confirmed that this swimming method significantly improves the efficiency of robotic fish compared to traditional continuous swimming, and it has great reference value for research on efficient swimming in robotic fish.

[0007] Reinforcement learning, as a widely used learning algorithm, does not rely on precise models. It optimizes control strategies through robot-environment interaction and exhibits good environmental adaptability. Exploring the energy-efficient swimming gait of biomimetic robotic fish using reinforcement learning can effectively solve problems such as the difficulty in defining energy-efficient swimming effects and the challenge in effectively modeling gait.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A reinforcement learning-based method for energy-saving swimming gait learning in biomimetic underwater robots includes the following steps: Step 1: Select a robotic fish as the biomimetic underwater robot, imitate the structure of the biomimetic robotic fish, and design and construct a reinforcement learning simulation environment based on the theory of slender bodies. Step 2: Construct a Markov decision model based on the robotic fish's mission objectives and motion state; Step 3: Based on reinforcement learning, construct a learning framework, which includes a reinforcement learning simulation environment and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning for biomimetic underwater robots.

[0009] Further, step 1 includes: acquiring the motion parameters of the biomimetic underwater robot, inputting them into the central mode generator of the biomimetic underwater robot, acquiring the angle, angular velocity and angular acceleration of each joint; inputting the acquired joint angle, angular velocity and angular acceleration into the dynamic model based on the slender body theory, acquiring the energy consumption, force conditions and motion state of the robotic fish at this moment.

[0010] Furthermore, in step 1, the reinforcement learning simulation environment simulates the physical properties of the robotic fish and the force conditions under different motion states of the robotic fish. The input of the reinforcement learning simulation environment is the control parameters of the robotic fish, including the frequency, amplitude, and intermittent factor of the central pattern generator. The output is the force conditions and position information of the robotic fish. The reinforcement learning simulation environment constructs an interactive platform simulating the bionic robotic fish in the computer simulation, enabling the reinforcement learning algorithm to be trained and greatly improving the simulation speed.

[0011] Furthermore, step 2 includes: constructing an energy-saving swimming gait reward module, an action space, and a state space; inputting the motion state of the robotic fish into the reward module during simulation; optimizing the optimal swimming gait of the robotic fish; and thus finding the optimal swimming gait of the biomimetic robotic fish.

[0012] Furthermore, in step 3, the efficient and energy-saving intermittent swimming gait learning of the biomimetic underwater robot includes: imitating the efficient and energy-saving intermittent swimming behavior of fish in nature through the intermittent CPG module, constructing an intermittent swimming CPG control module for the robotic fish to control the swimming and gliding of the robotic fish; and outputting tail fin oscillations by the intermittent CPG module to achieve efficient swimming of the underwater robot.

[0013] Furthermore, the learning framework includes: The energy-saving swimming gait optimization reward module inputs the motion state of the robotic fish into the simulation, optimizes the optimal swimming gait of the robotic fish, and thus finds the optimal swimming gait of the biomimetic robotic fish.

[0014] This invention also provides a reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning device, comprising the following modules: The simulation environment construction module selects a robotic fish as the biomimetic underwater robot, imitates the structure of the biomimetic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory. The decision model building module constructs a Markov decision model based on the robot fish's task objectives and motion state. The gait learning module, based on reinforcement learning, constructs a learning framework that includes a reinforcement learning simulation environment and a Markov decision model, enabling efficient and energy-saving intermittent swimming gait learning for biomimetic underwater robots.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning method.

[0016] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning method.

[0017] Beneficial effects: This invention enables a biomimetic underwater robot to autonomously explore its optimal swimming gait, significantly reducing its swimming energy consumption and achieving more efficient and energy-saving movement. By summarizing the swimming gait patterns within the learning framework and deploying them on the biomimetic underwater robot, the energy consumption per unit distance was substantially reduced. This invention has significant guiding implications for the efficient and energy-saving control of underwater robots. Attached Figure Description

[0018] Figure 1 This is a diagram of the energy-saving swimming gait training framework provided by the present invention; Figure 2 This is a schematic diagram of the network structure provided by the present invention; Figure 3 This is a schematic diagram of the energy-saving motion learning method and efficient intermittent swimming strategy for underwater robots of the present invention; Figure 4 It is an intermittent energy-saving swimming gait obtained through a reinforcement learning training framework; where (a) is a graph showing the frequency of the efficient intermittent energy-saving swimming gait obtained through the reinforcement learning framework as a function of time, and (b) is a graph showing the joint angle of the biomimetic underwater robot obtained through the efficient intermittent energy-saving swimming gait as a function of time. Figure 5 This is a diagram showing the verification results of the robot motion control task.

[0019] The attached diagram is labeled as follows: first hidden layer 210, second hidden layer 220, and output layer 230. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0021] like Figure 1 As shown, the present invention provides a reinforcement learning-based method for energy-saving swimming gait learning of biomimetic underwater robots, comprising the following steps: Step 1: The biomimetic underwater robot is preferably a robotic fish. By imitating the structure of the biomimetic robotic fish, a reinforcement learning simulation environment based on the theory of slender bodies is designed and constructed. This simulation environment can simulate the physical properties of a biomimetic robotic fish, including the forces acting on it under different motion states. The input to the simulation environment consists of the robotic fish's control parameters (frequency, amplitude, and intermittent factor of the CPG central pattern generator), and the output includes the forces acting on the robotic fish and its position information. The purpose of this simulation environment is to build an interactive platform for simulating a biomimetic robotic fish in a computer simulation, allowing reinforcement learning algorithms to be trained without real-world conditions, thus significantly improving the simulation speed.

[0022] Step 2: Construct a Markov decision model based on the robot fish's task objective and motion state. The Markov decision model includes the setting of the reinforcement learning reward function and the reinforcement learning algorithm. Step 3: Based on reinforcement learning, construct a learning framework, which includes a reinforcement learning simulation environment and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning for biomimetic underwater robots.

[0023] Specifically, in step 1, the reinforcement learning simulation environment consists of two parts: a central pattern generator for simulating an underwater robot and a dynamic model of the underwater robot. The central pattern generator can control each joint of the bionic underwater robot to oscillate periodically.

[0024] For the dynamic model, the robotic fish can be simplified as a series of n-bar linkages, representing n-1 joints from the head to the tail of the robotic fish.

[0025] The coordinate and velocity transformations of the robotic fish constitute the kinematic formula, which can be expressed as: ; in, The first part represents the robotic fish. The robot fish's body is represented by segment R, where R is the rotation transformation matrix, P is the length matrix, and T is the final joint transformation matrix. V represents the velocity transformation matrix. The joint angle of each joint of the robotic fish is denoted as... , The yaw angle represents the head. The length of each segment of the fish's body is denoted as... .

[0026] Using the Newton-Euler method, a dynamic formula is constructed, and the external forces acting on each segment of the fish can be expressed as: ; in, This represents the force exerted on the current segment of the fish by the previous segment. It is the Coriolis force and centripetal force in each segment of the fish. It is the first The inertial matrix on the fish's body It is the first The hydrodynamics of the fish. Representing the robotic fish The center of mass of each link is relative to its length at the starting position. Representing the robotic fish The mass of each link. Representing the Moment of inertia of each joint Representing the The angular velocities of each joint. The superscript T denotes the transpose of the matrix. This represents a 3×3 identity matrix. Let x be the unit vector of each joint of the robotic fish along the x-direction.

[0027] Based on the above theory, this invention establishes a dynamic model of the robotic fish, as shown in the following equation: ; in, This represents the mapping relationship of the dynamic model. and Represents the velocity of the robot fish's head unit between time t and the next time t+1; variable and This represents the angle of each joint between this moment t and the next moment t+1. This represents the net external force acting on each segment of the robotic fish.

[0028] Specifically, step 2 includes: Construct a reinforcement learning Markov decision model for energy-efficient gait learning. At each time step, the observed state from the environment is used. Defined as: ; in, To reinforce the learning of the observed state at time t, Indicates the angle of deviation from the specified direction. This represents the velocity of the three axes in the robot's coordinate system. This indicates the angles of each joint of the robotic fish. Indicates an action.

[0029] For each observation state The policy network generates an action. The three parameters corresponding to the CPG (Central Pattern Generator) network are frequency F, amplitude M, and intermittent factor λ. Figure 3 In this context, B represents the bias, and Φ represents the phase difference.

[0030] For the energy-saving gait learning task, a dense reward function is defined. : , ; in, These are the weights of each sub-reward, i=1,2,3,4,5. This represents the velocity of the robotic fish at time t. and These represent the torque and angular velocity of each joint at time t, respectively. The robotic fish is rewarded for maintaining a certain speed as it swims forward. The robot fish is rewarded by moving in the designated direction while maintaining its yaw angle. The robotic fish is rewarded for swimming with low energy consumption; The robot fish's laziness will be punished, and it will be encouraged to explore freely. This will encourage smooth movement of the robotic fish, where exp represents the exponential function. The deviation angle between the current heading angle and the target heading angle of the robotic fish is represented by the superscript t, which indicates the time t.

[0031] Specifically, step 3 includes: After constructing the simulation environment and Markov decision model, the learning of energy-efficient swimming gait can begin. The movements... The input is fed into a pre-built simulation environment, and after passing through the drive control model in the simulation environment, the output is the frequency, bias, and phase difference of the robot's tail fin oscillation. Then... Input the dynamic model, kinematic model, and Markov decision model to obtain the reward at the current time step. The state of the robot at the next time step And the state of the next time step is determined as the state of the current time step.

[0032] Figure 2 This is a schematic diagram of the structure of the initial residual model provided by the present invention, as shown below. Figure 2 As shown, the initial residual model is a multi-layer neural network, consisting of a first hidden layer 210, a second hidden layer 220, and an output layer 230, all fully connected. The first hidden layer 210 contains 800 neurons, the second hidden layer 220 contains 600 neurons, and the output layer 230 contains 3 neurons. The ReLU function is used as the activation function for the hidden layers, and the Tanh function is used as the activation function for the output layer. The robot's current motion state and the task objective are input into the initial residual model to obtain the second control action output by the initial residual model.

[0033] experience array Stored in the experience pool. An array is randomly drawn from the experience pool for offline training of the policy network of the initial residual model in the Soft Actor-Critic (SAC) reinforcement learning algorithm. This continues until the training is complete and the optimal energy-efficient swimming gait of the robotic fish is obtained.

[0034] Figure 3 This is a schematic diagram of a CPG central pattern generator that mimics the behavior of fish in nature. The intermittent swimming behavior of fish includes burst phases and gliding phases. Switching between burst and gliding is achieved through an intermittent factor.

[0035] Figure 4 This is a schematic diagram of the swimming gait of a robotic fish achieved through highly efficient and energy-saving swimming. Among them, Figure 4 (a) is a graph showing the frequency variation over time of the efficient intermittent energy-saving swimming gait obtained through the reinforcement learning framework. Figure 4 (b) is a graph showing the change of joint angles over time for a biomimetic underwater robot with an efficient intermittent energy-saving swimming gait obtained through a reinforcement learning framework.

[0036] To verify the effectiveness of the obtained energy-saving gait, for example, such as Figure 5 As shown, robot motion control tasks can be verified in an indoor pool measuring 5m × 4m × 1.1m. An energy consumption measurement module installed inside the robotic fish can measure its energy consumption in real time. The verification results show that, compared to existing robotic fish's traditional continuous swimming methods and energy-saving swimming methods obtained through reinforcement learning, the method provided in this embodiment of the invention enables the underwater robot to save an average of 24.2% of energy while maintaining the same swimming speed.

[0037] This invention also provides a reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning device, comprising the following modules: The simulation environment construction module selects a robotic fish as the biomimetic underwater robot, imitates the structure of the biomimetic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory. The decision model building module constructs a Markov decision model based on the robot fish's task objectives and motion state. The gait learning module, based on reinforcement learning, constructs a learning framework that includes a reinforcement learning simulation environment and a Markov decision model, enabling efficient and energy-saving intermittent swimming gait learning for biomimetic underwater robots.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning method.

[0039] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based biomimetic underwater robot energy-saving swimming gait learning method.

[0040] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0041] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0042] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0044] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0045] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A bionic underwater robot energy-saving swimming gait learning method based on reinforcement learning, characterized in that: The steps include: Step 1: The bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory; Step 2: Construct a Markov decision model based on the task objectives and motion state of the robot fish; Step 3: Based on reinforcement learning, a learning framework is constructed, wherein the learning framework includes a simulation environment for reinforcement learning and a Markov decision model, so as to realize the efficient and energy-saving intermittent swimming gait learning of the bionic underwater robot.

2. According to the reinforcement learning-based bionic underwater robot energy-saving swimming gait learning method of claim 1, it is characterized in that: The step 1 comprises: obtaining the motion parameters of the bionic underwater robot, inputting them into the central pattern generator of the bionic underwater robot, and obtaining the angle, angular velocity and angular acceleration of each joint thereof; inputting the obtained joint angle, angular velocity and angular acceleration into a dynamic model based on the slender body theory, and obtaining the energy consumption, force and motion state of the robot fish at this moment.

3. The energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning according to claim 1 is characterized in that: In step 1, the reinforcement learning simulation environment simulates the physical properties of the robotic fish in the simulation, simulating the force conditions of the robotic fish in different motion states; The input of the reinforcement learning simulation environment is the control parameters of the robot fish, including the frequency, amplitude, and intermittent factor of the central pattern generator, and the output is the force and position information of the robot fish. The reinforcement learning simulation environment builds an interactive platform for simulating bionic robot fish in computer simulation, allowing the reinforcement learning algorithm to be trained, greatly improving the simulation rate.

4. The energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning according to claim 1 is characterized in that: The step 2 includes: by constructing an energy-saving swimming gait reward module, an action space and a state space, the motion state of the robot fish is input into the reward module in the simulation, and the optimal swimming gait of the robot fish is optimized, so as to find the optimal swimming gait of the bionic robot fish.

5. The energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning according to claim 1 is characterized in that: In step 3, the efficient and energy-saving intermittent swimming gait learning of the bionic underwater robot is realized, including: imitating the efficient and energy-saving intermittent swimming behavior of fish in nature through the intermittent CPG module, constructing the intermittent swimming CPG control module of the robot fish, and controlling the swimming and gliding of the robot fish; the intermittent CPG module outputs the tail fin oscillation to realize the efficient swimming of the underwater robot.

6. The energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning according to claim 5 is characterized in that: The learning framework includes: The energy-saving swimming gait optimization reward module inputs the motion state of the robot fish into the reward module in the simulation, optimizes the optimal swimming gait of the robot fish, and thus finds the optimal swimming gait of the bionic robot fish.

7. A bionic underwater robot energy-saving swimming gait learning device based on reinforcement learning, characterized in that: Includes the following modules: In the simulation environment construction module, the bionic underwater robot selects a robotic fish, imitates the structure of the bionic robotic fish, and designs and constructs a reinforcement learning simulation environment based on the slender body theory; The decision model building module builds a Markov decision model based on the task objectives and motion status of the robot fish; The gait learning module constructs a learning framework based on reinforcement learning. The learning framework includes a simulation environment for reinforcement learning and a Markov decision model to achieve efficient and energy-saving intermittent swimming gait learning of a bionic underwater robot.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the energy-saving swimming gait learning method of a bionic underwater robot based on reinforcement learning are implemented as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the reinforcement learning-based energy-saving swimming gait learning method of a bionic underwater robot are implemented as described in any one of claims 1 to 6.

Images