Biomimetic robotic fish control method, device and storage medium based on deep reinforcement learning

Abstract

This invention provides a deep reinforcement learning-based biomimetic robotic fish control method, device, and storage medium, belonging to the field of biomimetic robot control technology. It solves the problem in existing technologies of lacking a deep reinforcement learning CPG network for joint motion control of biomimetic robotic fish. The invention includes S1: constructing an outer biomimetic robotic fish information network through deep learning, and providing initial commands through interaction with the environment; S2: constructing an inner CPG network based on the initial commands, and providing specific joint motion commands by constructing a motion model based on a central pattern generator. This invention has advantages such as the ability to adjust the biomimetic fish in complex underwater environments.

Description

Technical Field

[0001] This invention belongs to the field of biomimetic robot control technology, and specifically relates to a deep reinforcement learning-based biomimetic robotic fish control method, device and storage medium. Background Technology

[0002] Deep reinforcement learning primarily consists of two parts: deep learning and reinforcement learning. The concept of deep learning originated from artificial neural networks (ANNs). Its models typically consist of multiple layers of nonlinear computational units, using the output of lower layers as input to higher layers, thereby learning abstract feature representations from large amounts of training data and discovering the distributed characteristics of the data. Deep learning theory can effectively mine deep-level features of data. One important branch, graph neural networks, has effectively broken the constraints of traditional neural networks on image processing requirements, making it one of the most important research directions today. CPG (Central Pattern Generator) is a neural network capable of generating coordinated patterns of rhythmic activity without requiring any rhythmic input from sensory feedback or a higher-level control center. Due to its excellent performance, CPG-based controls have been widely used to generate various swimming patterns, such as forward swimming, backward swimming, and rotation. Although many CPG model methods have been proposed, these methods are relatively simple and difficult to cope with complex underwater environments. In addition, the level of intelligence of these control methods is low. At present, the research on biomimetic robotic fish motion control integrating deep reinforcement learning CPG is still in its infancy. There is a lack of biomimetic robotic fish joint motion control methods based on deep reinforcement learning CPG networks for joint swimming control, and many studies lack clear and specific implementation plans. Summary of the Invention

[0003] The purpose of this invention is to address the aforementioned problems in the prior art by providing a deep reinforcement learning-based biomimetic robotic fish control method, device, and storage medium.

[0004] The first objective of this invention can be achieved through the following technical solution: a biomimetic robotic fish control method based on deep reinforcement learning, characterized by comprising the following steps:

[0005] S1: Construct an outer biomimetic robotic fish information network through deep learning, and give initial instructions through interaction with the environment;

[0006] S2: Construct an inner CPG network for the initial instructions, and give specific joint motion instructions by constructing a motion model based on the central pattern generator;

[0007] The working principle of this invention: This invention has good applicability to joint motion control of biomimetic robotic fish with multiple joints or multiple degrees of freedom in complex underwater environments. The method of this invention combines a deep reinforcement learning network and proposes a joint motion co-control method for biomimetic robotic fish that integrates the outer biomimetic robotic fish information network and the inner CPG network model. Using this method, intelligent, autonomous and efficient swimming control of biomimetic robotic fish can be achieved.

[0008] In the aforementioned deep reinforcement learning-based biomimetic robotic fish control method, the outer biomimetic robotic fish information network includes input information processed using a cooperative transformation method. The input information is used to generate a deep reinforcement learning network for initial instructions and an inner CPG network transmission interface.

[0009] In the aforementioned deep reinforcement learning-based biomimetic robotic fish control method, the collaborative conversion method includes associating and labeling four consecutive frames of images acquired by the external sensors of the biomimetic robotic fish with depth, distance, and two or more other data. The collaborative conversion method packages multi-dimensional data into structured data that can be directly processed by deep networks as input to subsequent deep reinforcement learning networks.

[0010] In the aforementioned deep reinforcement learning-based biomimetic robotic fish control method, the deep reinforcement learning network is constructed using a deep reinforcement Q-learning network. Through the good processing mechanism and good interaction capability with the external environment of the deep reinforcement Q-learning network, the initial instructions for the movement of the biomimetic robotic fish are generated. After the deep network generates the initial instructions, they are input into the inner CPG network interface.

[0011] In the above-mentioned deep reinforcement learning-based biomimetic robotic fish control method, the deep reinforcement learning network utilizes the DQN algorithm to construct a deep reinforcement learning framework. The multivariate data is input into the deep reinforcement learning framework, and the deep reinforcement learning framework generates the corresponding Q-values ​​of the input multivariate data by setting a target reward value. The Q-value generation formula is shown in equation (1):

[0012] Q * (s, a)=∑p a (s,s′)(R a (s,s′)+γmax a′ Q * (s′,a′)) (1)

[0013] Where P a (s,s,) represents the probability of transitioning from the current state s to the next state s', R(s,s') represents the reward after performing the action in the current state, γ is the decay coefficient, and maxQ *(s′,a′) represents the operation of selecting the current largest Q value, generating an estimate of Q through the deep network, and updating the parameters of the deep network by the difference between the Q value and the Q estimate, as shown in equation (2):

[0014] L(θ)=E((R+γmax a′ (s′, a′, θ)-Q(s, a, θ)) 2 (2)

[0015] Where L(θ) represents the loss function and E represents the expectation operation.

[0016] In the above-mentioned deep reinforcement learning-based bionic robotic fish control method, the inner CPG network interface can convert the preliminary instructions and transmit them to the inner CPG network to realize the specific joint movements of the bionic robotic fish. The generative formula of the specific joint movement model of the bionic robotic fish is as shown in formula (3):

[0017]

[0018] in t is the direction control parameter, and θ is the phase difference between neurons. ε represents the different phases of the i-th neuron. i ω represents the amplitude and frequency of the neuron. Furthermore, P... u P v Let P be the disturbance term, where P is the disturbance term. v =c2u i+ 1sinθ+c1u i+1 cosθ, P u =c1u i-1 cosθ-c2v i-1 sinθ, c1, and c2 are the neuron coupling coefficients. After the CPG model is constructed, the phase output is converted and input to each joint of the bionic robotic fish, as shown in equation (4):

[0019] Γ i =ζ i v i +Θ i (4)

[0020] Where Γ i ζ represents the input of the i-th joint. i The conversion coefficient corresponding to the joint is determined by the corresponding motor, Θ i The initial instruction coefficients generated for the upper-layer network are ultimately used to complete the interaction and intelligent, efficient swimming control of the biomimetic robotic fish in complex environments through the collaborative cooperation of the outer deep reinforcement learning network and the inner CPG network.

[0021] In the aforementioned deep reinforcement learning-based biomimetic robotic fish control method, the inner CPG network interface includes a preliminary instruction decomposition, calibration, and transmission mechanism.

[0022] In the aforementioned deep reinforcement learning-based biomimetic robotic fish control method, the inner CPG network includes a motion model based on a central pattern generator and a specific joint motion command conversion and transmission mechanism.

[0023] The second objective of this invention can be achieved through the following technical solution: a biomimetic robotic fish joint motion control device based on a deep reinforcement learning CPG network, comprising:

[0024] Computer side;

[0025] Controller;

[0026] One or more processors;

[0027] Memory;

[0028] And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a biomimetic robotic fish joint motion control method for executing a deep reinforcement learning CPG network as described above.

[0029] The third objective of this invention can be achieved through the following technical solution: a storage medium storing a computer program for use with a computer terminal and a display, wherein the computer program can be executed by a processor to perform a bionic robotic fish joint motion control method using a deep reinforcement learning CPG network as described above.

[0030] Compared with existing technologies, this invention has the advantages of being able to adjust the bionic fish in complex underwater environments and having sensitive adjustment capabilities. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the joint motion control of the biomimetic robotic fish using deep reinforcement learning CPG according to the present invention.

[0032] Figure 2 This is a schematic diagram of the angle input of the biomimetic robotic fish based on the CPG network in the inner layer of this invention. Detailed Implementation

[0033] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0034] like Figure 1-2 As shown, the biomimetic robotic fish control method based on deep reinforcement learning is characterized by the following steps:

[0035] S1: Construct an outer biomimetic robotic fish information network through deep learning, and give initial instructions through interaction with the environment;

[0036] S2: Construct an inner CPG network for the initial instructions, and give specific joint motion instructions by constructing a motion model based on the central pattern generator;

[0037] The outer layer of the bionic robotic fish information network interacts with the environment through a deep reinforcement learning-based bionic robotic fish information network to generate initial commands for the joint movements of the bionic robotic fish. The inner layer of the CPG network transforms the initial commands into the movement angles of the specific joints of the bionic robotic fish by constructing a CPG-based rhythmic motion network, thereby realizing the joint movement control of the bionic robotic fish.

[0038] To elaborate further, the outer biomimetic robotic fish information network includes input information processed using a cooperative transformation method. This input information is used to generate a deep reinforcement learning network for initial instructions, as well as an inner CPG network transmission interface. The input information is processed using a cooperative transformation method and used as input. The initial motion instructions for the biomimetic robotic fish are realized through the DQN algorithm framework and then input to the lower interface to realize the motion angles of the specific joints of the biomimetic robotic fish by constructing a CPG-based rhythmic motion network.

[0039] To elaborate further, the collaborative conversion method involves associating and labeling four consecutive frames of images acquired by the external sensors of the biomimetic robotic fish with depth, distance, and two or more other data. The collaborative conversion method packages multi-dimensional data into structured data that can be directly processed by deep networks as input to subsequent deep reinforcement learning networks. It associates image information with sensor data such as depth and distance, packages it into structured data that can be processed by deep networks, and then uses it as input information for deep reinforcement learning networks.

[0040] To elaborate further, the deep reinforcement learning network uses a deep reinforcement Q-learning network to construct the deep network. Through the good processing mechanism of the deep reinforcement Q-learning network and its good interaction with the external environment, it generates the initial instructions for the movement of the biomimetic robotic fish. After the deep network generates the initial instructions, they are input into the inner CPG network interface.

[0041] To elaborate further, the deep reinforcement learning network utilizes the DQN algorithm to construct a deep reinforcement learning framework. Multivariate data is input into the deep reinforcement learning framework, which then generates the corresponding Q-values ​​of the input multivariate data by setting a target reward value. The formula for generating the Q-values ​​is shown in equation (1).

[0042] Q * (s, a)=∑p a (s,s′)(R a (s,s′)+γmax a′ Q *(s′,a′)) (1)

[0043] Where P a (s,s,) represents the probability of transitioning from the current state s to the next state s', R(s,s') represents the reward after performing the action in the current state, γ is the decay coefficient, and maxQ * (s′,a′) represents the operation of selecting the current largest Q value, generating an estimate of Q through the deep network, and updating the parameters of the deep network by the difference between the Q value and the Q estimate, as shown in equation (2):

[0044] L(θ)=E((R+γmax a′ (s′, a′, θ)-Q(s, a, θ)) 2 (2)

[0045] Where L(θ) represents the loss function, E represents the expectation operation, and the network architecture also employs mechanisms such as memory playback and target network.

[0046] To elaborate further, the inner CPG network interface can convert the initial instructions and transmit them to the inner CPG network to realize the specific joint movements of the bionic robotic fish. The generative formula of the specific joint movement model of the bionic robotic fish is as shown in formula (3):

[0047]

[0048] in t is the direction control parameter, and θ is the phase difference between neurons. ε represents the different phases of the i-th neuron. i ω represents the amplitude and frequency of the neuron. Furthermore, P... u P v Let P be the disturbance term, where P is the disturbance term. v =c2u i+ 1sinθ+c1u i+1 cosθ, P u =c1u i-1 cosθ-c2v i-1 sinθ, c1, and c2 are the neuron coupling coefficients. After the CPG model is constructed, the phase output is converted and input to each joint of the bionic robotic fish, as shown in equation (4):

[0049] Γ i =ζ i v i +Θ i (4)

[0050] Where Γ i ζ represents the input of the i-th joint. i The conversion coefficient corresponding to the joint is determined by the corresponding motor, Θi The initial instruction coefficients generated for the upper-layer network are ultimately used to complete the interaction and intelligent, efficient swimming control of the biomimetic robotic fish in complex environments through the collaborative cooperation of the outer deep reinforcement learning network and the inner CPG network.

[0051] To elaborate further, the inner CPG network interface includes a mechanism for the decomposition, calibration, and transmission of preliminary instructions. For example, "turn right quickly" is decomposed into "turn right" and then "quickly" is transmitted to the inner CPG network.

[0052] To elaborate further, the inner CPG network includes a motion model based on a central pattern generator and a specific joint motion command conversion and transmission mechanism.

[0053] The second objective of this invention can be achieved through the following technical solution: a biomimetic robotic fish joint motion control device based on a deep reinforcement learning CPG network, comprising:

[0054] Computer side;

[0055] Controller;

[0056] One or more processors;

[0057] Memory;

[0058] And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including a biomimetic robotic fish joint motion control method for executing a deep reinforcement learning CPG network as described above.

[0059] The third objective of this invention can be achieved through the following technical solution: a storage medium for storing a computer program that is used in conjunction with a computer terminal and a display. The computer program can be executed by a processor to perform a biomimetic robotic fish joint motion control method using a deep reinforcement learning CPG network as described above.

[0060] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

[0061] Although this document uses a large number of technical terms, the possibility of using other terms is not excluded. These terms are used merely for the convenience of describing and explaining the essence of the invention; interpreting them as any additional limitation would be contrary to the spirit of the invention.

Claims

1. A method for controlling a biomimetic robotic fish based on deep reinforcement learning, the method comprising: It comprises the following steps: S1: constructing an outer layer biomimetic robotic fish information network through deep learning, and giving preliminary instructions through interaction with the environment; S2: constructing an inner layer CPG network for the preliminary instructions, and giving specific joint motion instructions through constructing a motion model based on the central pattern generator; The inner layer CPG network interface can convert the preliminary instructions and transmit them into the inner layer CPG network to realize the specific joint motion of the biomimetic robotic fish, and the specific joint motion model of the biomimetic robotic fish is generated as formula (3): （3） wherein , t is the direction control parameter, θ is the phase difference between neurons, represents the different phase of the i-th neuron, εi, ω represents the amplitude and frequency of the neuron, in addition, Pu, Pv are perturbation terms, wherein Pv=c2ui+1sinθ+c1ui+1cosθ, Pu=c1ui-1cosθ-c2vi-1sinθ, c1, c2 are the coupling coefficients of the neurons, after the completion of the CPG model construction, the phase output is converted and input to each joint of the biomimetic robot fish, as formula (4): Γi = ζiv i + Θi (4) Where Γi represents the input of the i-th joint, ζi is the corresponding conversion coefficient of the joint, which is determined by the corresponding motor, and θi is the preliminary instruction coefficient generated by the upper layer network. Finally, the outer layer deep reinforcement learning network and the inner layer CPG network cooperate to complete the interaction and intelligent efficient swimming control of the biomimetic robotic fish in a complex environment; the collaborative conversion method comprises associating and labeling the continuous 4 frames of images collected by the external sensors of the biomimetic robotic fish with depth, distance and more than two kinds of data, and the collaborative conversion method packs the multi-element data into structured data that can be directly processed by the deep network as input of the subsequent deep reinforcement learning network.

2. The method of claim 1, wherein: The outer layer biomimetic robotic fish information network comprises input information processed by the collaborative conversion method, and the input information is used for the deep reinforcement learning network for generating preliminary instructions and the transmission interface of the inner layer CPG network.

3. The method of claim 1, wherein the method further comprises: The deep reinforcement learning network adopts a deep reinforcement Q learning network to construct a deep network, and generates preliminary instructions for the motion of the biomimetic robotic fish through the good processing mechanism of the deep reinforcement Q learning network and the good interaction ability with the external environment. The deep network generates preliminary instructions and inputs the inner layer CPG network interface.

4. The method of claim 1, wherein: The inner layer CPG network interface comprises a decomposition, calibration and transmission mechanism of the preliminary instructions.

5. The method of claim 1, wherein: The inner layer CPG network comprises a motion model based on the central pattern generator and a specific joint motion instruction conversion and transmission mechanism.

6. A control device characterized by comprising: It comprises: a computer terminal; a controller; one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs comprise a biomimetic robotic fish joint motion control method for executing the deep reinforcement learning CPG network of any one of claims 1-5.

7. A storage medium characterized by: A computer program is stored for use in combination with a computer terminal and a display, and the computer program can be executed by a processor to complete the biomimetic robotic fish joint motion control method of the deep reinforcement learning CPG network of any one of claims 1-5.

Images