A manta ray machine fish depth control method based on q learning

Abstract

The application relates to a manta ray machine fish depth control method based on Q learning, compared with a traditional control method, the application does not need to establish a control object model, through collecting experimental data, training a Q table offline, then transplanting the table into a prototype controller, obtaining a control variable through table lookup, and controlling the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, the application has low requirements on hardware resources, small space demand, low power consumption and convenient realization. The application has the beneficial effects that the realization of the method has low requirements on the hardware of the controller, and an ordinary single-chip microcomputer can realize the method, which not only saves the cost, but also saves the space and energy. The application is simple to realize and has strong generalization ability, and can be applied to a traditional underwater AUV as well as a bionic underwater robot.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of underwater biomimetic robots, and relates to a depth control method for a manta ray-inspired robotic fish based on Q-learning. Background Technology

[0002] With the development of social science and technology and the increasingly severe ecological and environmental problems facing humanity, the demand for marine environmental exploration and resource extraction is increasing. Autonomous underwater vehicles (AUVs) have promising scientific and engineering applications, enabling tasks such as underwater environmental monitoring, scientific research, underwater archaeology, and resource development in near and far seas. The manta ray-inspired underwater robotic fish is a novel type of biomimetic AUV. Compared to traditional propeller propulsion systems, it exhibits better environmental compatibility, higher maneuverability, and stronger adaptability to complex environments. In recent years, the control problems of biomimetic underwater robots have attracted the research interest and enthusiasm of scholars, and related scientific and technological issues have also seen significant development.

[0003] Controlling underwater biomimetic robots is one of the challenges in the field. Depth control, as the most fundamental attitude control problem, is the basis for underwater biomimetic robots to perform other tasks and is one of the basic requirements for underwater control. Currently, underwater depth control methods mainly include model-based methods and model-free methods. Model-based methods require establishing a mathematical model of the controlled object. However, the dynamic model of a biomimetic underwater robot is more complex than that of a traditional rigid rotating body (AUV), and the coupling effect between the flexible body and the fluid is more pronounced, making model establishment more difficult. Therefore, model-free control methods are a better choice.

[0004] Fuzzy control methods have been successfully applied to biomimetic underwater robots. The advantage of fuzzy control is that it does not require building a model of the controlled object; instead, it only requires designing a fuzzy rule base based on human experience, and then controlling the object according to these rules. The disadvantages of fuzzy control are its reliance on expert experience and relatively low control accuracy. Compared to fuzzy control, the most popular model-free control method currently being researched is reinforcement learning (RL). In 1956, Bellman proposed dynamic programming (DP), in 1988 Sutton proposed the temporal difference (TD) algorithm, and in 1992, Watkins proposed the Q-learning algorithm. In 2013, Mnih et al. combined deep learning and reinforcement learning to propose the first deep reinforcement learning algorithm, which was successfully applied to Atari games. Some games played using RL algorithms have surpassed the level of professional human players. In 2016, Deep Blue Academy's reinforcement learning algorithm, AlphaGo, successfully defeated the human world champion.

[0005] Currently, there are still many challenges in applying RL methods to real-world prototypes. One of these is how to implement reinforcement learning control in systems with relatively low computing power. For example, many deep reinforcement learning algorithms require industrial control computers with high GPU computing power as the control system hardware, while currently many controllers still use microcontrollers such as single-chip microcomputers as the main control unit. Summary of the Invention

[0006] Technical problems to be solved

[0007] To avoid the shortcomings of existing technologies, this invention proposes a depth control method for a manta ray-inspired robotic fish based on Q-learning.

[0008] Technical solution

[0009] A depth control method for a manta ray-inspired robotic fish based on Q-learning, characterized by the following steps:

[0010] Step 1: Establish a finite Markov model for the depth control problem of the manta ray-inspired robotic fish:

[0011] 1. Establish a discrete state space S: The depth in the actual environment ranges from (-∞, +∞).

[0012] S={s -n ,s -n+1 ,…,s -1 ,s0,s1,…,s n-1 ,s n}

[0013] Where: s -n =(-∞,Δz) -n ], s -n+1 =(Δz) -n ,Δz -n+1 ], s n-1 =(Δz) n-1 ,Δa n ],

[0014] s n =(Δz) n ,+∞],

[0015] The Δz represents the difference between the actual depth and the target depth, Δz = z now -z ref , z now The z-axis represents the current depth reported by the sensor. ref Indicates the reference target depth;

[0016] 2. Establish the discrete action space A: A = {a -n ,a -n+1 ,…,a -1 ,a0,a1,…,an-1 ,a n}

[0017] Where 'a' represents an element of the action space, representing the fixed angle of the tail fin's swing;

[0018] 3. Design the reward function:

[0019]

[0020] Where δ is a parameter for adjusting the depth control accuracy of the manta ray-inspired robotic fish;

[0021] The r t+1 Let the state be s at time t. t The action is a t The state at the next moment is s. t+1 The reward received;

[0022] Step 2: Create the action-value function table Q(s,a):

[0023]

[0024] Step 3: Train the Q-table:

[0025] 1. Training data samples:

[0026] Calculate the reward function r at time t using the original depth and pose change data. t+1 (s t ,a t );

[0027] Data for training the Q-table test Let the storage size be N, which stores state transition information (Transition).

[0028] Among them: Data test ={Transition1,Transition2,…,Transition N},

[0029] Transition i ={s t ,a t ,s t+1 ,r t+1};

[0030] 2. Q(s) n ,a n The updated formula is:

[0031]

[0032] Where α is the learning rate and γ is the discount factor;

[0033] 3. Randomly sample data from the training dataset, according to Q(s) n ,a n The formula is updated and the table is updated cyclically until the error in the change of the Q value in the Q table is less than the specified value ε.

[0034] Step 4: Design control strategy:

[0035]

[0036] This indicates that when the state is s, the action 'a' is chosen to maximize Q(s,α).

[0037] The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

[0038] Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows:

[0039] 1) Initialize the learning rate α and the discount factor γ;

[0040] 2) Assign an initial value of 0 to all Q(s,a) tables in Q;

[0041] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0042] 4) Repeat steps 5-7:

[0043] 5) Randomly sample transitions from the experience pool and perform operations according to Q(s). n ,a n Update the formula to update Q(s,a);

[0044] 6) Calculation

[0045] 7) When The loop terminates;

[0046] 8) Output all Q(s,a) in the Q table, and training ends.

[0047] The value of ε is adjusted according to the training situation.

[0048] The value of ε is ε = 0.01.

[0049] The value of δ is 3cm.

[0050] Beneficial effects

[0051] This invention proposes a depth control method for a manta ray-inspired robotic fish based on Q-learning. Compared with traditional control methods, this invention does not require building a model of the controlled object. It collects experimental data, trains a Q-table offline, and then imports the table into the prototype controller. The control variables are obtained by looking up the table, controlling the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, it has lower hardware resource requirements, smaller space requirements, lower power consumption, and is easier to implement. Its beneficial effects are:

[0052] 1) The implementation of the method of the present invention has low hardware requirements for the controller, and can be achieved by ordinary microcontrollers. In addition to saving costs, it also saves space and energy.

[0053] 2) Compared with the traditional classical PID control method, it does not require the establishment of a mathematical model of the controlled object. Since biomimetic models are more difficult to establish, a lot of time and cost can be saved.

[0054] 3) Compared with depth control methods based on fuzzy control, this method can achieve depth control of the robotic fish without the need for expert experience to build a rule base. The Q-table will learn "expert experience" on its own during training.

[0055] 4) This invention is simple to implement and has strong generalization ability. In addition to being applied to biomimetic underwater robots, it can also be applied to traditional underwater AUVs. Attached Figure Description

[0056] Figure 1 This is a flowchart of the depth control method for the manta ray robotic fish of the present invention;

[0057] Figure 2 This is a flowchart of the Q-table training program of the present invention.

[0058] Figure 3 This is a schematic diagram of the depth control method for the manta ray robotic fish of the present invention.

[0059] Figure 4 This is a diagram showing the experimental results of the manta ray-inspired robotic fish for depth control in this invention. Detailed Implementation

[0060] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0061] The purpose of this invention is to solve the problem of applying reinforcement learning methods in practical systems, and to propose a depth control method for a manta ray-inspired robotic fish based on Q-learning, characterized by the following steps:

[0062] 1. Establishing a finite Markov model for the depth control problem of a manta ray-inspired robotic fish:

[0063] Step 1-1) Establish the discrete state space S:

[0064] In real-world environments, the depth ranges from (-∞, +∞). To simplify the state space, we reduce it to 2n+1 elements, i.e., let the state space be...

[0065] S={s -n ,s -n+1 ,…,s -1 ,s0,s1,…,s n-1 ,s n},

[0066] in,

[0067] s -n =(-∞,Δz) -n ],

[0068] s -n+1 =(Δz) -n ,Δz -n+1 ],

[0069] ……,

[0070] s -1 =(Δz) -2 ,Δz -1 ],

[0071] s0=(Δz -1 ,Δz1),

[0072] s1=(Δz1,Δz2],

[0073] ……,

[0074] s n-1 =(Δz) n-1 ,Δz n ]

[0075] s n =(Δz) n ,+∞]

[0076] in

[0077] Δz represents the difference between the actual depth and the target depth, calculated as follows:

[0078] Δz=z mow -z ref (1)

[0079] in

[0080] z now The z-axis represents the current depth reported by the sensor. ref Indicates the reference target depth;

[0081] Step 1-2) Establish the discrete action space A:

[0082] The actual motion space for depth control of the manta ray robotic fish is a continuous motion space [-a] min ,+a max To simplify the model, while meeting the control requirements, the control variables are discretized into 2n+1 actions. Let the discrete action space be...

[0083] A={a -n ,a -n+1 ,…,a -1 ,a0,a1,…,a n-1 ,a n}

[0084] Where 'a' represents an element of the action space, representing the fixed angle of the tail fin's swing;

[0085] Steps 1-3) Design the reward function calculation method:

[0086] At time t, the state is s. t The action is a t The state at the next moment is s. t+1 The reward received is r t+1 ,

[0087] r t+1 The calculation formula is:

[0088]

[0089] δ is a small quantity that affects the depth control accuracy of the manta ray-inspired robotic fish;

[0090] 2: Establish the action-value function table Q(s,a);

[0091] Table 1

[0092]

[0093] 3: Training the Q-table;

[0094] Step 3-1) Training data sample preparation, the specific steps include:

[0095] 1) Experimental data such as original depth and attitude changes obtained through experimental methods or by previous experiments;

[0096] 2) Calculate the reward function r at time t according to the formula in step 1. t+1 (s t ,a t );

[0097] 3) Create the dataset for training the Q-table. testLet the storage size be N, which stores state transition information (Transition).

[0098] in

[0099] Data test ={Transition1,Transition2,…,Transition N},

[0100] Transition i ={s t ,a t ,s t+1 ,r t+1};

[0101] Step 3-2) Q(s) n ,a n The updated formula is:

[0102]

[0103] Where α is the learning rate and γ is the discount factor;

[0104] Step 3-3) Randomly extract data from the training dataset and update the table iteratively according to the calculation formula in step 3-2) until the error of the change in the Q value in the Q table is less than the specified value ε;

[0105] The pseudocode for the program flowchart of training Q-tables using Python is as follows:

[0106] 1) Initialize the learning rate α and the discount factor γ;

[0107] 2) Assign an initial value of 0 to all Q(s,a) tables in Q;

[0108] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0109] 4) Loop:

[0110] 5) Randomly sample Transitions from the experience pool and update Q(s,a) according to the formula in 3-2);

[0111] 6) Calculation

[0112] 7) When The loop terminates;

[0113] 8) Output all Q(s,a) in the Q-table to end training.

[0114] 4. Design control strategy:

[0115]

[0116] This indicates that when the state is s, the action 'a' is selected to maximize Q(s,a). This relates to Xi's depth control method for a manta ray-inspired robotic fish. Specific implementation examples:

[0118] like Figure 1 As shown, the first step of this invention is to model the depth control problem of the manta ray robotic fish as a Markov decision model: 1) establish the state space S of the depth control of the manta ray robotic fish; 2) establish the action space A of the depth control; 3) reward function.

[0119] 1. Establish a Markov decision model for depth control of a manta ray-inspired robotic fish:

[0120] 1-1) Establish the state space:

[0121] S={s -n ,s -n+1 ,…,s -1 ,s0,s1,…,s n-1 ,s n},

[0122] Let n = 5, and set another...

[0123] s -5 =(-∞,-40cm],

[0124] s -4 =(-40cm, -20cm],

[0125] s -3 = (-20cm, -10cm),

[0126] s -2 =(-10cm, -5cm],

[0127] s -1 = (-5cm, -3cm)

[0128] s0 = (-3cm, +3cm)

[0129] s1 = (3cm, 5cm)

[0130] s2 = (5cm, 10cm)

[0131] s3 = (10cm, 20cm)

[0132] s4 = (20cm, 40cm)

[0133] s5=(40cm,+∞)

[0134] 1-2) Establish the motion space A for depth control of the manta ray-inspired robotic fish:

[0135] In this embodiment, the tail fin of the manta ray-like robotic fish is adjustable within a range of [-40°, +40°]. Taking n = 20, then...

[0136] A={a -20 ,a -19 ,…,a -1 ,a0,a1,…,a 19 ,a 20}

[0137] in,

[0138] a -20 = -40°, a -19 = -38°, ..., a -1 =-2°, a0=0°, a1=2°,…,a 19 =38°, a 20 =40°

[0139] 1-3) Design the reward function

[0140] According to formula (2), taking δ = 3cm, the reward function is:

[0141]

[0142] 2: Establish the action-value function table Q(s,a);

[0143] Based on Table 1, the state space, and the action space, construct the Q-table:

[0144]

[0145] 3: Training the Q-table;

[0146] Step 3-1) Training data sample preparation (obtain data on depth changes and tail fin wagging angle changes through experiments);

[0147] Step 3-2) Establish a training environment based on Python;

[0148] Step 3-3) Update the Q-table based on the Q-algorithm. The algorithm flow is as follows: Figure 2 As shown:

[0149] 1) Program initialization: Initialize Q for any state and action. t (s t ,a t =0; Initial learning rate α = 0.01, discount factor γ = 0.9, training cutoff condition ε = 0.01, experience pool storage capacity N = 2000;

[0150] 2) Data Processing: Read the training data and calculate the reward function, converting the data into {s} t ,a t ,r t+1 ,s t+1}form;

[0151] 3) Store the data in experience pool D;

[0152] 4) Loop:

[0153] Randomly sample data from experience pool D;

[0154] Update Q t (s t ,a t According to formula (3)

[0155] The calculation is performed for all states and actions according to formula (4).

[0156] if If the value is less than ε, exit the loop; otherwise, continue the loop.

[0157] 5) Output the Q-table; training ends.

[0158] 4. Design control strategy:

[0159] like Figure 3 The diagram shows the principle of the depth control method of the present invention. The depth sensor provides the current depth of the controlled object, and then compares it with the target depth to output a depth error Δz. Δz is input to the controller for calculation to determine the current state s. The controller then determines the current state s according to the control strategy.

[0160]

[0161] In the Q table, find the action 'a' that maximizes the state value function and use it as the output of the control variable.

[0162] The experimental results were verified with a water depth of 55cm, and the experimental results are shown in the figure below. Figure 4 As shown.

Claims

1. A depth control method for a manta ray-inspired robotic fish based on Q-learning, characterized by the following steps: Step 1: Establish a finite Markov model for the depth control problem of the manta ray-inspired robotic fish: 1.1 Establishing the Discrete State Space The range of depth values ​​in a real-world environment is: in: ， ， ， The This represents the difference between the actual depth and the target depth. , This indicates the current depth reported by the sensor. Indicates the reference target depth; 1.2 Establishing a Discrete Action Space : in Elements representing the movement space, representing the fixed angle of the tail fin's swing; 1.3 Design the reward function: in To adjust the parameters for depth control accuracy of the manta ray-inspired robotic fish; The ; Step 2: Create an action-value function table : Step 3: Train the Q-table: 3.1 Training data samples: Calculate using raw depth and attitude change data Time-based reward function ; Establish training The table's dataset Let the storage capacity be... It stores state transition information. ; in: , ； 3.2 The updated formula is: in, For learning rate, Discount factor; 3.

3. Randomly extract data from the training dataset, according to... The formula is updated and the table is updated iteratively until the error in the change of the Q value in the Q table is less than the specified value ε. Step 4: Design control strategy: This indicates that when the state is s, the choice is made such that... Action a that achieves the maximum value; Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows: 1) Initialize the learning rate and discount factor ; 2) Assign values ​​to all tables in Q. The initial value is 0; 3) Set the experience pool storage capacity Read in the raw training data, process it, and then store it in the experience pool. ; 4) Repeat steps 5-7: 5) Randomly sample from the experience pool And in accordance with Formula update ; 6) Calculation 7) When The loop terminates; 8) Output all values ​​in the Q table. Training is over; The value of ε is adjusted according to the training situation; The value of ε is ε=0.01, the Take the value 3cm.

2. The depth control method for the manta ray-inspired robotic fish based on Q-learning according to claim 1, characterized in that: The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

Images