Manta ray robot fish heading control method based on q-learning and pectoral fin phase difference

Abstract

The application relates to a manta ray robot fish heading control method based on Q learning and pectoral fin phase difference. Compared with a traditional control method, the application does not need to establish a control object model, collects experimental data, trains a Q table offline, then transplants the table into a prototype controller, obtains a control variable through table lookup, and controls the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, the application has low requirements on hardware resources, small space demand, low power consumption and convenient realization. The application has the beneficial effects that the application has low requirements on the hardware of a controller, and an ordinary single-chip microcomputer can be used to realize the application, thereby saving cost, space and energy, saving a large amount of time cost, being simple to realize, having strong generalization ability, and being applicable to not only a bionic underwater robot but also a traditional underwater AUV.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of underwater biomimetic robots, and relates to a heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference. Background Technology

[0002] With the development of social science and technology and the increasingly severe ecological and environmental problems facing humanity, the demand for marine environmental exploration and resource extraction is increasing. Autonomous underwater vehicles (AUVs) have promising scientific and engineering applications, enabling tasks such as underwater environmental monitoring, scientific research, underwater archaeology, and resource development in near and far seas. The manta ray-inspired underwater robotic fish is a novel type of biomimetic AUV. Compared to traditional propeller propulsion systems, it exhibits better environmental compatibility, higher maneuverability, and stronger adaptability to complex environments. In recent years, the control problems of biomimetic underwater robots have attracted the research interest and enthusiasm of scholars, and related scientific and technological issues have also seen significant development.

[0003] Control is one of the challenges in underwater biomimetic robots, and heading control, as part of attitude control, is fundamental for underwater biomimetic robots to perform other tasks and is one of the basic requirements for underwater operational control. Currently, underwater heading control methods can be categorized into model-based and model-free methods based on whether a model is required. Model-based methods require establishing a mathematical model of the controlled object. However, the dynamic model of a biomimetic underwater robot is more complex than that of a traditional rigid rotating AUV, and the coupling effect between the flexible body and the fluid is more pronounced, making model establishment more difficult. Therefore, model-free control methods are a better choice.

[0004] Fuzzy control methods have been successfully applied to biomimetic underwater robots. The advantage of fuzzy control is that it does not require building a model of the controlled object; instead, it only requires designing a fuzzy rule base based on human experience, and then controlling the object according to these rules. The disadvantages of fuzzy control are its reliance on expert experience and relatively low control accuracy. Compared to fuzzy control, the most popular model-free control method currently being researched is reinforcement learning (RL). In 1956, Bellman proposed dynamic programming (DP), in 1988 Sutton proposed the temporal difference (TD) algorithm, and in 1992, Watkins proposed the Q-learning algorithm. In 2013, Mnih et al. combined deep learning and reinforcement learning to propose the first deep reinforcement learning algorithm, which was successfully applied to Atari games. Some games played using RL algorithms have surpassed the level of professional human players. In 2016, Deep Blue Academy's reinforcement learning algorithm, A1pha-go, successfully defeated the human world champion.

[0005] Currently, there are still many challenges in applying RL methods to real-world prototypes. One of these is how to implement reinforcement learning control in systems with relatively low computing power. For example, many deep reinforcement learning algorithms require industrial control computers with high GPU computing power as the control system hardware, while currently many controllers still use microcontrollers such as single-chip microcomputers as the main control unit. Summary of the Invention

[0006] Technical problems to be solved

[0007] To avoid the shortcomings of existing technologies and solve the problems of applying reinforcement learning methods in practical systems, this invention proposes a heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference, specifically a model-free heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference.

[0008] Technical solution

[0009] A heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference, characterized by the following steps:

[0010] Step 1: Establish a finite Markov model for the heading control problem of the manta ray-inspired robotic fish:

[0011] 1. Establish the discrete state space S for heading control:

[0012] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n}

[0013] in:

[0014] s -n =(-∞, Δψ) -n ],

[0015] s -n+1 =(Δψ -n , Δψ -n+1 ],

[0016] ……,

[0017] s -1 =(Δψ -2 , Δψ -1 ],

[0018] s v =(Δψ -1 ,Δψ1),

[0019] s1=(Δψ1,Δψ2],

[0020] ……,

[0021] s n-1 =(Δψ n-1 , Δψ n ]

[0022] s n =(Δψ n ,+∞]

[0023] Where, Δψ i The range of values ​​for i can be adjusted according to the actual situation, i∈{-n, -n+1, ​​..., -1, 0, 1, ..., n-1, n};

[0024] 2. Establish discrete motion space A: The motion space for the heading control of the manta ray-inspired robotic fish is a continuous motion space [-a] min +a max Discretize the control variables into 2n+1 actions, and define the discrete action space.

[0025] Where: 'a' represents an element of the motion space, representing the phase difference of a fixed pectoral fin oscillation, and 0 ≤ a ≤ a max ;

[0026] 3. Design the reward function:

[0027]

[0028] Where: δ is a parameter for adjusting the depth control accuracy of the manta ray-inspired robotic fish;

[0029] The r t+1 Let the state be s at time t. t The action is a t The state at the next moment is s t+1 The reward received;

[0030] Step 2: Create the action-value function table Q(s, a):

[0031]

[0032] Step 3: Train the Q-table:

[0033] 1. Training data samples: Using the original depth and pose change data, calculate the reward function r at time t. t+1 (s t a t );

[0034] Data for training the Q-table test Let the storage size be N, which stores state transition information (Transition).

[0035] in:

[0036] Data test ={Transition1, Transition2,…,Transition N}

[0037] Transition i ={s t a t s t+1 r t+1}

[0038] 2. Q(s) n a n The updated formula is:

[0039]

[0040] Where α is the learning rate and γ is the discount factor;

[0041] 3. Randomly sample data from the training dataset, according to Q(s) n a n The formula is updated and the table is updated cyclically until the error in the change of the Q value in the Q table is less than the specified value ε.

[0042] Step 4: Design control strategy:

[0043]

[0044] When the state is s, choose action a that maximizes Q(s, a).

[0045] The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

[0046] The roll deviation The result is: feedback of the roll angle obtained through the sensor is... Assuming the roll angle fed back by the sensor is the true roll value, the desired roll value range is: Therefore, it can be deduced that...

[0047] The when At that time, based on the 360° period of angle change, it is equivalent to Similarly, when At that time, it is equivalent to therefore The range of values ​​can be equivalent to

[0048] Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows:

[0049] 1) Initialize the learning rate α and the discount factor γ;

[0050] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0051] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0052] 4) Repeat steps 5-7:

[0053] 5) Randomly sample transitions from the experience pool and perform operations according to Q(s). n a n Update the formula to update Q(s, a);

[0054] 6) Calculation

[0055] 7) When The loop terminates;

[0056] 8) Output all Q(s, a) in the Q table, and training ends.

[0057] The value of ε is adjusted according to the training situation.

[0058] The value of ε is ε = 0.1.

[0059] The value of δ is 3°.

[0060] Beneficial effects

[0061] This invention proposes a heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference. Compared with traditional control methods, this invention does not require building a model of the controlled object. It collects experimental data, trains a Q-table offline, and then imports the table into the prototype controller. The control variables are obtained by looking up the table, controlling the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, it has lower hardware resource requirements, smaller space requirements, lower power consumption, and is easier to implement. Its beneficial effects are:

[0062] 1) The implementation of the method of the present invention has low hardware requirements for the controller, and can be achieved by ordinary microcontrollers. In addition to saving costs, it also saves space and energy.

[0063] 2) Compared with the traditional classical PID control method, it does not require the establishment of a mathematical model of the controlled object. Since biomimetic models are more difficult to establish, a lot of time and cost can be saved.

[0064] 3) Compared with fuzzy control-based heading control methods, this method does not require expert experience to build a rule base to achieve heading control of the robotic fish. The Q-table learns "expert experience" on its own during training.

[0065] 4) This invention is simple to implement and has strong generalization ability. In addition to being applied to biomimetic underwater robots, it can also be applied to traditional underwater AUVs. Attached Figure Description

[0066] Figure 1 This is a flowchart of the manta ray robotic fish heading control method of the present invention;

[0067] Figure 2 This is a flowchart of the Q-table training program of the present invention.

[0068] Figure 3 This is a schematic diagram of the heading control method for the manta ray robotic fish of the present invention.

[0069] Figure 4 This is a diagram showing the experimental results of the manta ray-inspired robotic fish's heading control based on the pectoral fin phase difference, as presented in this invention. Detailed Implementation

[0070] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0071] This invention proposes a model-free manta ray-inspired robotic fish heading control method based on Q-learning, characterized by the following steps:

[0072] 1. Establish a finite Markov model for the heading control problem of a manta ray-inspired robotic fish:

[0073] Step 1-1) Establish the discrete state space S for heading control:

[0074] The heading angle feedback obtained from the sensor is ψ∈[0°, 360°). Let the sensor-feedback heading be the true heading value, and the expected heading range be ψ′∈[0°, +360°). From this, the range of the heading deviation Δψ can be derived:

[0075] Δψ=ψ-ψ′, Δψ∈(-360°, +360°).

[0076] When Δψ∈(-360°, -180°), based on the 360° period of angle change, it is equivalent to Δψ∈(0°, 180°). Similarly, when Δψ∈(180°, 360°), it is equivalent to Δψ∈(-180°, 0°). Therefore, the range of values ​​for Δψ can be equivalent to Δψ∈[-180°, 180°].

[0077] To discretize the state space, we simplify it to 2n+1 elements, that is, let the state space be...

[0078] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n},

[0079] in,

[0080] s -n =(-∞, Δψ) -n ],

[0081] s -n+1 =(Δψ -n , Δψ -n+1 ],

[0082] ……,

[0083] s -1 =(Δψ -2 , Δψ -1 ],

[0084] s0=(Δψ -1 ,Δψ1),

[0085] s1=(Δψ1,Δψ2],

[0086] ……,

[0087] s n-1 =(Δψ n-1 , Δψ n ]

[0088] s n =(Δψ n ,+∞]

[0089] Where, Δψ i The range of values ​​for i can be adjusted according to the actual situation, i∈{-n, -n+1, ​​..., -1, 0, 1, ..., n-1, n}

[0090] Step 1-2) Establish the discrete action space A:

[0091] The actual motion space for the heading control of the manta ray-inspired robotic fish is a continuous motion space [-a] min +a max To simplify the model, while meeting the control requirements, the control variables are discretized into 2n+1 actions. Let the discrete action space be...

[0092] A={a -n a -n+1 , ..., a -1 a0, a1, ..., a n-1 a n}

[0093] Where 'a' represents an element of the motion space, signifying the phase difference of a fixed pectoral fin oscillation, and 0 ≤ a ≤ a max ;

[0094] Steps 1-3) Design the reward function calculation method:

[0095] At time t, the state is s. t The action is a t The state at the next moment is s t+1 The reward received is r t+1 r t+1 The calculation formula is

[0096]

[0097] Where δ is a small quantity, the value of which is near the desired control accuracy, and it affects the heading control accuracy of the manta ray-inspired robotic fish. When the desired control accuracy is 3°, we can take δ = 3°.

[0098] 2: Establish the action-value function table Q(s, a);

[0099] Table 1

[0100]

[0101] 3: Training the Q-table;

[0102] Step 3-1) Training data sample preparation, the specific steps include:

[0103] 1) Raw depth and attitude change experimental data obtained through experimental methods or by previous experiments;

[0104] 2) Calculate the reward function r at time t according to the formula in step 1. t+1 (s t a t );

[0105] 3) Create the dataset for training the Q-table. test Let the storage size be N, which stores state transition information (Transition).

[0106] in

[0107] Data test ={Transition1, Transition2,…,Transition N},

[0108] Transition i ={s t at s t+1 r t+1};

[0109] Step 3-2) Q(s) n a n The updated formula is:

[0110]

[0111] Where α is the learning rate and γ is the discount factor;

[0112] Step 3-3) Randomly extract data from the training dataset and update the table iteratively according to the calculation formula in step 3-2) until the error of the change in the Q value in the Q table is less than the specified value ε = 0.1;

[0113] The pseudocode for the program flowchart of training Q-tables using Python is as follows:

[0114] 1) Initialize the learning rate α and the discount factor γ;

[0115] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0116] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0117] 4) Start the loop:

[0118] 5) Randomly sample Transitions from the experience pool and update Q(s, a) according to the formula in 3-2);

[0119] 6) Calculation

[0120] 7) When The loop terminates;

[0121] 8) Output all Q(s, a) in the Q-table to end training.

[0122] 4. Design control strategy:

[0123]

[0124] This indicates that when the state is s, the action 'a' is chosen to maximize Q(s, a).

[0125] The embodiments of the present invention are described in detail below. The accompanying drawings of the embodiments are shown in the drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0126] like Figure 1As shown, the first step of this invention is to model the heading control problem of the manta ray-inspired robotic fish as a Markov decision model:

[0127] 1) Establish the state space S for the heading control of the manta ray-inspired robotic fish; 2) Establish the action space A for heading control; 3) Develop a reward function.

[0128] 1. Establish a Markov decision model for the heading control of a manta ray-inspired robotic fish:

[0129] 1-1) Establish the state space:

[0130] S={s -n s -n+1 , ..., s -1 s o ,s1,...,s n-1 s n},

[0131] Let n = 10, and set another...

[0132] s -10 = [-180°, -30°],

[0133] s -9 = (-30°, -27°)

[0134] s -8 = (-27°, -24°)

[0135] s -2 = (-9°, -6°)

[0136] s -1 = (-6°, -3°)

[0137] s0 = (-3°, 3°),

[0138] s1 = [3°, 6°),

[0139] s2 = [6°, 9°),

[0140] …,

[0141] s9 = [27°, 30°),

[0142] s 10 = [30°, 180°)

[0143] 1-2) Establish the motion space A for the heading control of the manta ray-inspired robotic fish:

[0144] In this embodiment, the pectoral fin amplitude of the manta ray-inspired robotic fish is adjustable within the range of [0°, +50°]. When swimming straight, both pectoral fins maintain a constant amplitude of 50°. When turning left, the amplitude of the right pectoral fin remains unchanged, while the amplitude of the left pectoral fin changes; similarly, when turning right, the amplitude of the left pectoral fin remains unchanged, while the amplitude of the right pectoral fin changes.

[0145] Let the action space A = (a0, a1, a2, a3, a4, a5)

[0146] For the prototype test, the following parameters are used: a0 = 50°, a1 = 40°, a2 = 30°, a3 = 20°, a4 = 10°, and a5 = 0°. These parameters can be changed according to the test results and accuracy requirements.

[0147] 1-3) Design the reward function

[0148] According to formula (2), taking δ = 3°, the reward function is:

[0149]

[0150] 2: Establish the action-value function table Q(s, a);

[0151] Based on Table 1, the state space, and the action space, construct the Q-table:

[0152] Table 2:

[0153]

[0154] 3: Training the Q-table;

[0155] Step 3-1) Training data sample preparation (obtain data on changes in heading and pectoral fin sway angle through experiments);

[0156] Step 3-2) Establish a training environment based on Python;

[0157] Step 3-3) Update the Q-table based on the Q-algorithm. The algorithm flow is as follows: Figure 2 As shown:

[0158] 1) Program initialization: Initialize Q for any state and action. t (s t a t =0; Initial learning rate α = 0.1, discount factor γ = 0.9, training cutoff condition ε = 0.1, experience pool storage capacity N = 2800;

[0159] 2) Data Processing: Read the training data and calculate the reward function, converting the data into {s} t a t r t+1 s t+1}form;

[0160] 3) Store the data in experience pool D;

[0161] 4) Start the loop:

[0162] Randomly sample data from experience pool D;

[0163] Update Q t (s t a t According to formula (3)

[0164] The calculation is performed for all states and actions according to formula (4).

[0165] if If the value is less than ε, exit the loop; otherwise, continue the loop.

[0166] 5) Output the Q-table; training ends.

[0167] 4. Design control strategy:

[0168] like Figure 3 The diagram shows the principle of the heading control method of the present invention. The depth sensor provides the current depth of the controlled object, which is then compared with the target heading to output a depth error Δψ. Δψ is input to the controller for calculation to determine the current state s. The controller then determines the current state s according to the control strategy.

[0169]

[0170] The action 'a' that maximizes the state value function is retrieved from the Q table and used as the output of the pectoral fin heading control.

[0171] The test results verified that the set heading was 10°, and the test results were as follows: Figure 4 As shown.

Claims

1. A heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference, characterized by the following steps: Step 1: Establish a finite Markov model for the heading control problem of the manta ray-inspired robotic fish: 1.1 Establishing the discrete state space for heading control : in: ， ， ……， ， ， ， ……， in, The range of values ​​can be adjusted according to the actual situation. ; 1.2 Establishing a Discrete Action Space The motion space for the heading control of the manta ray-inspired robotic fish is a continuous motion space. Discretize the control variables into 2n+1 actions, and define the discrete action space. in The elements representing the motion space represent the phase difference of a fixed pectoral fin oscillation, and ; 1.3 Design the reward function: in: To adjust the parameters for depth control accuracy of the manta ray-inspired robotic fish; ； Step 2: Create an action-value function table : Step 3: Train the Q-table: 3.1 Training Data Samples: Using the original depth and pose change data, calculate... Time-based reward function ; Establish training The table's dataset Let the storage capacity be... It stores state transition information. ; in: 3.2 The updated formula is: in, For learning rate, Discount factor; 3.

3. Randomly extract data from the training dataset, according to... The formula is updated and the table is updated iteratively until the error in the change of the Q value in the Q table is less than the specified value ε. Step 4: Design control strategy: When the state is s, choose the option that makes Action a that achieves the maximum value; The roll deviation The result is: feedback of the roll angle obtained through the sensor is... Let the roll angle fed back by the sensor be the true roll value, and the expected range of roll values ​​be... Therefore, it can be deduced that , . The when At that time, the period of angle change is It can be seen that it is equivalent to Similarly, when At that time, it is equivalent to ,therefore The range of values ​​can be equivalent to ; The The value is 3°.

2. The heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference according to claim 1, characterized in that: The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

3. The heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference according to claim 1, characterized in that: Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows: 1) Initialize the learning rate and discount factor ; 2) Assign values ​​to all tables in Q. The initial value is 0; 3) Set the experience pool storage capacity Read in the raw training data, process it, and then store it in the experience pool. ; 4) Repeat steps 5-7: 5) Randomly sample from the experience pool And in accordance with Formula update ; 6) Calculation 7) When The loop terminates; 8) Output all values ​​in the Q table. Training is over.

4. The heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference according to claim 1 or 3, characterized in that: The value of ε is adjusted according to the training situation.

5. The heading control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin phase difference according to claim 1 or 3, characterized in that: The value of ε is ε=0.1.

Images