A bionic fish pitch control method based on q-learning and pectoral fin bar bias

Abstract

The present application relates to a kind of based on Q learning and bionic fish fin bar bias PLOVER control method, compared with traditional control method, the present application does not need to establish control object model, by collecting experimental data, offline training Q table, then table is transplanted to prototype controller, control variable is obtained by table lookup, control prototype is swimming in fixed depth.Compared with other reinforcement learning control method, it is low to hardware resource requirement, and space demand is small, power consumption is small, and it is easy to realize.It is beneficial to have in that the hardware requirement of controller is low, and ordinary single-chip microcomputer can be realized, in addition to saving cost, it also reaches the effect of saving space and energy saving.No need to establish the mathematical model of control object, since bionic model is more difficult to establish, therefore, a large amount of time cost can be saved.No need to establish rule base with expert experience to achieve the pitch control of robotic fish, Q table will learn "expert experience" when training.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of underwater biomimetic robots, and relates to a biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset. Background Technology

[0002] With the development of social science and technology and the increasingly severe ecological and environmental problems facing humanity, the demand for marine environmental exploration and resource extraction is increasing. Autonomous underwater vehicles (AUVs) have promising scientific and engineering applications, enabling tasks such as underwater environmental monitoring, scientific research, underwater archaeology, and resource development in near and far seas. The manta ray-inspired underwater robotic fish is a novel type of biomimetic AUV. Compared to traditional propeller propulsion systems, it exhibits better environmental compatibility, higher maneuverability, and stronger adaptability to complex environments. In recent years, the control problems of biomimetic underwater robots have attracted the research interest and enthusiasm of scholars, and related scientific and technological issues have also seen significant development.

[0003] Controlling underwater biomimetic robots is one of the challenges in this field. Rapid pitch control, as part of high-maneuverability attitude control, is essential for underwater biomimetic robots to quickly avoid obstacles and is a fundamental requirement for controlling operations in confined underwater areas. Currently, underwater rapid pitch control methods can be categorized into model-based and model-free methods, depending on whether a model is required. Model-based methods require establishing a mathematical model of the controlled object. However, the dynamic model of a biomimetic underwater robot is more complex than that of a traditional rigid rotating AUV, with a more pronounced coupling effect between the flexible body and the fluid, making model establishment more difficult. Therefore, model-free control methods are a better choice.

[0004] Fuzzy control methods have been successfully applied to biomimetic underwater robots. The advantage of fuzzy control is that it does not require building a model of the controlled object; instead, it only requires designing a fuzzy rule base based on human experience, and then controlling the object according to these rules. The disadvantages of fuzzy control are its reliance on expert experience and relatively low control accuracy. Compared to fuzzy control, the most popular model-free control method currently being researched is reinforcement learning (RL). In 1956, Bellman proposed dynamic programming (DP), in 1988 Sutton proposed the temporal difference (TD) algorithm, and in 1992, Watkins proposed the Q-learning algorithm. In 2013, Mnih et al. combined deep learning and reinforcement learning to propose the first deep reinforcement learning algorithm, which was successfully applied to Atari games. Some games played using RL algorithms have surpassed the level of professional human players. In 2016, Deep Blue Academy's reinforcement learning algorithm, AlphaGo, successfully defeated the human world champion.

[0005] Currently, there are still many challenges in applying RL methods to real-world prototypes. One of these is how to implement reinforcement learning control in systems with relatively low computing power. For example, many deep reinforcement learning algorithms require industrial control computers with high GPU computing power as the control system hardware, while currently many controllers still use microcontrollers such as single-chip microcomputers as the main control unit. Summary of the Invention

[0006] Technical problems to be solved

[0007] To avoid the shortcomings of existing technologies and solve the problems of applying reinforcement learning methods in practical systems, this invention proposes a model-free rapid pitch control method for a manta ray robotic fish based on Q-learning and pectoral fin ray amplitude bias.

[0008] Technical solution

[0009] A biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset, characterized by the following steps:

[0010] Step 1: Establish a finite Markov model for the rapid pitch control problem of the manta ray-inspired robotic fish:

[0011] 1. The discrete state space S of pitch control, the pitch angle feedback is θ∈[0°, 360°), the sensor feedback pitch angle is the true pitch value, and the expected pitch value range is θ′∈[0°, +360°).

[0012] The range of pitch deviation Δθ is: Δθ=θ-θ′, Δθ∈(-360°,+360°)

[0013] State space discretization: = {s -n ,s -n+1 ,…,s -1 ,s0,s1,…,s n-1 ,s n}

[0014] in:

[0015] s -n =(-180°,Δθ) -n ],

[0016] s -n+1 =(Δθ) -n ,Δθ -n+1 ],

[0017] ……,

[0018] s -1 =(Δθ) -2 ,Δθ -1 ],

[0019] s0=(Δθ -1 ,Δθ1),

[0020] s1=(Δθ1,Δθ2],…,

[0021] s n -1=(Δθ n -1, Δθ n ]

[0022] s n =(Δθ) n (180°)

[0023] Where, Δθ i The range of values ​​can be adjusted according to the actual situation. One simple way is to take Δθ. i =360° / n, i∈{-n,-n+1,…,-1,0,1,…,n-1,n};

[0024] 2. Establish the discrete action space A:

[0025] A={a -n a -n+1 , ..., a -1 a0, a1, ..., a n-1 a n}

[0026] Where 'a' represents an element of the action space, representing the sum of the angles of the fixed pre- and post-pectoral fin rays, and -180°≤a≤180°, the left and right pectoral fins use symmetrical offset angles as the response of the control quantity;

[0027] 3. Design the reward function:

[0028]

[0029] Where: r t+1 Let the state be s at time t. t The action is a t The state at the next moment is s t+1 The reward received; δ is a parameter for adjusting the pitch control accuracy of the manta ray-inspired robotic fish;

[0030] Step 2: Create the action-value function table Q(s, a):

[0031]

[0032] Step 3: Train the Q-table:

[0033] 1. Using the original depth and attitude change data, calculate the reward function r at time t. t+1 (s ta t );

[0034] Data set for training the Q table test Let the storage size be N, which stores state transition information (Transition).

[0035] Among them: Data test ={Transition1, Transition2,…,Transition N}

[0036] Transition i ={s t a t s t+1 r t+1}

[0037] 2. Q(s) n a n The updated formula is:

[0038]

[0039] Where α is the learning rate and γ is the discount factor;

[0040] 3. Randomly sample data from the training dataset, based on Q(s) from the previous step. n a n The formula is updated and the table is updated cyclically until the error in the change of the Q value in the Q table is less than the specified value ε.

[0041] Step 4: Design control strategy:

[0042]

[0043] When the state is s, choose action a that maximizes Q(s, a).

[0044] The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

[0045] The range of the pitch deviation Δθ is as follows: when Δθ∈(-360°, -180°), based on the 360° angle change period, it is equivalent to Δθ∈(0°, 180°). Similarly, when Δθ∈(180°, 360°), it is equivalent to Δθ∈(-180°, 0°). Therefore, the range of Δθ is equivalent to Δθ∈[-180°, 180°].

[0046] Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows:

[0047] 1) Initialize the learning rate α and the discount factor γ;

[0048] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0049] 3) Set the experience pool storage capacity N, read in the original training data, process it, and store it in the experience pool D;

[0050] 4) Start the loop;

[0051] 5) Randomly sample transitions from the experience pool and perform operations according to Q(s). n a n Update the formula to update Q(s, a);

[0052] 6) Calculation

[0053] 7) When The loop terminates;

[0054] 8) Output all Q(s, a) in the Q table, and training ends.

[0055] The value of ε is adjusted according to the training situation.

[0056] The value of ε is ε = 0.1.

[0057] The value of δ is 3°.

[0058] Beneficial effects

[0059] This invention proposes a biomimetic fish pitch control method based on Q-learning and pectoral fin ray bias. Compared with traditional control methods, this invention does not require building a controlled object model. It collects experimental data, trains a Q-table offline, and then imports the table into the prototype controller. The control variables are obtained by looking up the table, controlling the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, it has lower hardware resource requirements, smaller space requirements, lower power consumption, and is easier to implement. Its beneficial effects are:

[0060] 1) The implementation of the method of the present invention has low hardware requirements for the controller, and can be achieved by ordinary microcontrollers. In addition to saving costs, it also saves space and energy.

[0061] 2) Compared with the traditional classical PID control method, it does not require the establishment of a mathematical model of the controlled object. Since biomimetic models are more difficult to establish, a lot of time and cost can be saved.

[0062] 3) Compared with pitch control methods based on fuzzy control, this method can achieve pitch control of the robotic fish without the need for expert experience to build a rule base. The Q-table will learn "expert experience" on its own during training. Attached Figure Description

[0063] Figure 1 This is a flowchart of the rapid pitch control method for the manta ray-inspired robotic fish of the present invention;

[0064] Figure 2 This is a flowchart of the Q-table training program of the present invention;

[0065] Figure 3 This is a schematic diagram of the rapid pitch control method for the manta ray-inspired robotic fish of the present invention. Detailed Implementation

[0066] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0067] This invention proposes a model-free, manta ray-inspired robotic fish fast pitch control method based on Q-learning, characterized by the following steps:

[0068] 1. Establish a finite Markov model for the rapid pitch control problem of a manta ray-inspired robotic fish:

[0069] Step 1-1) Establish the discrete state space S for fast pitch control:

[0070] The rapid pitch angle feedback obtained from the sensor is θ∈[0°, 360°). Let the rapid pitch angle feedback from the sensor be the true pitch value, and let the range of the desired pitch be θ′∈[0°, +360°). From this, the range of the pitch deviation Δθ can be derived:

[0071] Δθ=θ-θ′, Δθ∈(-360°, +360°).

[0072] When Δθ∈(-360°, -180°), based on the 360° period of angle change, it is equivalent to Δθ∈(0°, 180°). Similarly, when Δθ∈(180°, 360°), it is equivalent to Δθ∈(-180°, 0°). Therefore, the range of Δθ is equivalent to Δθ∈[-180°, 180°].

[0073] In practical applications, the pitch angle range of the manta ray-inspired robotic fish is (-180°, 180°). The state space is discretized and simplified to 2n+1 elements. That is, let the state space...

[0074] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n},

[0075] in,

[0076] s -n= (-180°, Δθ) -n ],

[0077] s -n+1 =(Δθ) -n , Δθ -n+1 ],

[0078] ……,

[0079] s -1 =(Δθ) -2 , Δθ -1 ],

[0080] s o =(Δθ) -1 ,Δθ1),

[0081] s1=(Δθ1,Δθ2],

[0082] ……,

[0083] s n-1 =(Δθ) n-1 , Δθ n ]

[0084] s n =(Δθ) n (180°)

[0085] Where, Δθ i The range of values ​​can be adjusted according to the actual situation. One simple way is to take Δθ. i =360° / n, i∈{-n,-n+1,…,-1,0,1,…,n-1,n}.

[0086] Step 1-2) Establish the discrete action space A:

[0087] The actual motion space for rapid pitch control of the manta ray-inspired robotic fish is a continuous motion space [-a] min +a max To simplify the model, while meeting the control requirements, the control variables are discretized into 2n+1 actions. Let the discrete action space be...

[0088] A={a -n a -n+1 , ..., a -1 a0, a1, ..., a n-1 a n}

[0089] Where 'a' represents an element of the motion space, representing the sum of the fixed angles of the pre- and post-pectoral fin rays, and -180°≤a≤180°. The left and right pectoral fins use symmetrical offset angles as the response of the control quantity. For example, when a=90°, it means that the amplitude of the pre-pectoral fin ray is offset by 45° and the amplitude of the post-pectoral fin ray is offset by 45°.

[0090] Steps 1-3) Design the reward function calculation method:

[0091] At time t, the state is s. t The action is a t The state at the next moment is s t+1 The reward received is r t+1 ,

[0092] r t+1 The calculation formula is:

[0093]

[0094] δ is a small quantity that takes values ​​near the desired control accuracy and affects the pitch control accuracy of the manta ray-inspired robotic fish. When the desired control accuracy is 3°, δ can be taken as 3°.

[0095] 2: Establish the action-value function table Q(s, a);

[0096] Table 1

[0097]

[0098] 3: Training the Q-table;

[0099] Step 3-1) Training data sample preparation, the specific steps include:

[0100] 1) Experimental data such as original depth and attitude changes obtained through experimental methods or by previous experiments;

[0101] 2) Calculate the reward function r at time t according to the formula in step 1. t+1 (s t a t );

[0102] 3) Create the dataset for training the Q-table. test Let the storage size be N, which stores state transition information (Transition).

[0103] in

[0104] Data test ={Transition1, Transition2,…,Transition N},

[0105] Transition i ={s t a t s t+1 r t+1};

[0106] Step 3-2) Q(s) n a n The updated formula is:

[0107]

[0108] Where α is the learning rate and γ is the discount factor;

[0109] Step 3-3) Randomly extract data from the training dataset and update the table iteratively according to the calculation formula in step 3-2) until the error of the change in the Q value in the Q table is less than the specified value ε;

[0110] The pseudocode for the program flowchart of training Q-tables using Python is as follows:

[0111] 1) Initialize the learning rate α and the discount factor γ;

[0112] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0113] 3) Set the experience pool storage capacity N, read in the original training data, process it, and store it in the experience pool D;

[0114] 4) Start the loop:

[0115] 5) Randomly sample Transitions from the experience pool and update Q(s, a) according to the formula in 3-2);

[0116] 6) Calculation

[0117] 7) When The loop terminates;

[0118] 8) Output all Q(s, a) in the Q-table, training ends.

[0119] 4. Design control strategy:

[0120]

[0121] This indicates that when the state is s, the action 'a' is chosen to maximize Q(s, a).

[0122] The embodiments of the present invention are described in detail below. The accompanying drawings of the embodiments are shown in the drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0123] like Figure 1 As shown, the first step of this invention is to model the pitch control problem of the manta ray robotic fish as a Markov decision model: 1) establish the state space S of the pitch control of the manta ray robotic fish; 2) establish the action space A of the pitch control; 3) establish the reward function.

[0124] 1. Establish a Markov decision model for pitch control of a manta ray-inspired robotic fish:

[0125] 1-1) Establish the state space:

[0126] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n},

[0127] Let n = 10. Furthermore, based on actual application conditions, the pitch angle adjustment range of the robotic fish is set to (-90°, 90°).

[0128] s -10 = (-90°, 50°),

[0129] s -9 = (-50°, -45°)

[0130] s -8 = (-45°, -40°)

[0131] s -2 = (-15°, -10°)

[0132] s -1 = (-10°, -5°)

[0133] s0 = (-5°, 5°),

[0134] s1 = [5°, 10°),

[0135] s2 = [10°, 15°),

[0136] …,

[0137] s9 = [45°, 50°),

[0138] s 10 = [50°, 90°)

[0139] 1-2) Establish the motion space A for pitch control of the manta ray-inspired robotic fish:

[0140] In this embodiment, the manta ray-inspired robotic fish has three levels of adjustable fin rays. Our adjustment strategy is to adjust the amplitude of the first-level fin rays and the single-sided pectoral fin within the range of [-80°, +80°]. Pitch control is achieved by adjusting the offset of the first and third-level pectoral fin rays while swimming straight. When swimming straight, both pectoral fins maintain a constant 50° amplitude, therefore the adjustable range of the offset is [-30°, +30°]. When pitching upwards, the first-level pectoral fin ray offset amplitude is positive, the second-level pectoral fin ray offset remains unchanged, and the third-level pectoral fin ray offset is negative. When pitching downwards, the first-level pectoral fin ray offset amplitude is negative, the second-level pectoral fin ray offset remains unchanged, and the third-level pectoral fin ray offset is positive.

[0141] Let the action space A = {a -5 a -4 a -3 a -2 a -1 ,a0,a1,a2,a3,a4,a5},

[0142] in,

[0143] a -5 = -50°, a -4 = -40°, a -3 = -30°, a -2 = -20°, a -1 =-10°, a0=0°, a1=10°, a2=20°, a3=30°, a4=40°, a5=50°. It should be noted that these parameters can be changed according to the experimental results and accuracy requirements.

[0144] 1-3) Design the reward function

[0145] According to formula (2), taking δ = 5°, the reward function is:

[0146]

[0147] 2: Establish the action-value function table Q(s, a);

[0148] Based on Table 1, the state space, and the action space, construct the Q-table:

[0149] Table 2:

[0150]

[0151] 3: Training the Q-table;

[0152] Step 3-1) Training data sample preparation (obtain data on changes in heading and pectoral fin swing angle through experiments);

[0153] Step 3-2) Establish a training environment based on Python;

[0154] Step 3-3) Update the Q-table based on the Q-algorithm. The algorithm flow is as follows: Figure 2 As shown:

[0155] 1) Program initialization: Initialize Q for any state and action. t (s t a t =0; Initialize learning rate α = 0.1, discount factor γ = 0.9, training cutoff condition ε = 0.1, experience pool storage capacity N = 3000;

[0156] 2) Data Processing: Read the training data and calculate the reward function, converting the data into {s} t a t r t+1 s t+1}form;

[0157] 3) Store the data in experience pool D;

[0158] 4) Loop:

[0159] Randomly sample data from experience pool D;

[0160] Update Q t (s t a t According to formula (3)

[0161] The calculation is performed for all states and actions according to formula (4).

[0162] if If the value is less than ε, exit the loop; otherwise, continue the loop.

[0163] 5) Output the Q-table; training ends.

[0164] 4. Design control strategy:

[0165] like Figure 3 The diagram shows the principle of the pitch control method of this invention. The depth sensor provides the current depth of the controlled object, which is then compared with the target heading to output a depth error Δθ. Δθ is input to the controller for calculation to determine the current state s. The controller then determines the current state s according to the control strategy.

[0166]

[0167] The action 'a' that maximizes the state value function is retrieved from the Q table and used as the output of the pectoral fin pitch control.

Claims

1. A biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset, characterized by the following steps: Step 1: Establish a finite Markov model for the rapid pitch control problem of the manta ray-inspired robotic fish: 1.1 Discrete State Space for Pitch Control The pitch angle feedback is The sensor feedback pitch angle is the true pitch value, and the desired pitch value range is... ; Pitch deviation The range of values ​​for: , State space discretization: in: ， ， ……， ， ， ， ……， in, The range of values ​​can be adjusted according to the actual situation; one simple way is to take... , ; 1.2 Establishing a Discrete Action Space : in The element representing the movement space represents the sum of the angles of the fixed anterior and posterior pectoral fin rays, and The left and right pectoral fins use symmetrical offset angles as the control quantity response; 1.3 Design the reward function: in: For at any time The status is The action is The state at the next moment is The reward received; To adjust the parameters for the pitch control accuracy of the manta ray-inspired robotic fish; Step 2: Create an action-value function table : Step 3: Train the Q-table: 3.1 Calculate using the original depth and attitude change data. Time-based reward function ; Establish training The table's dataset Let the storage capacity be... It stores state transition information. ; in: 3.2 The updated formula is: in, For learning rate, Discount factor; 3.3 Randomly extract data from the training dataset, based on the previous step. The formula is updated and the table is updated iteratively until the error in the change of the Q value in the Q table is less than the specified value ε. Step 4: Design control strategy: When the state is s, choose the option that makes Action a that achieves the maximum value; pitch deviation The range of values ​​for: when At that time, the period of angle change is It is equivalent to Similarly, when At that time, it is equivalent to ,therefore The range of values ​​is equivalent to ; The The value is 3°.

2. The biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset according to claim 1, characterized in that: The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

3. The biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset according to claim 1, characterized in that: Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows: 1) Initialize the learning rate and discount factor ; 2) Assign values ​​to all tables in Q. The initial value is 0; 3) Set the experience pool storage capacity Read in the raw training data, process it, and then store it in the experience pool. ; 4) Start the loop; 5) Randomly sample from the experience pool And in accordance with Formula update ; 6) Calculation 7) When The loop terminates; 8) Output all values ​​in the Q table. Training is over.

4. The biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset according to claim 1 or 3, characterized in that: The value of ε is adjusted according to the training situation.

5. The biomimetic fish pitch control method based on Q-learning and pectoral fin ray offset according to claim 1 or 3, characterized in that: The value of ε is ε=0.1.

Images