Manta ray robot fish roll control method based on q-learning and pectoral fin amplitude bias

Abstract

The application relates to a manta ray robot fish roll control method based on Q learning and pectoral fin amplitude bias, compared with a traditional control method, the application does not need to establish a control object model, collects experimental data, trains a Q table offline, then transplants the table into a prototype controller, obtains a control variable through table lookup, and controls the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, the application has low requirements on hardware resources, small space demand, low power consumption and convenient realization. The application has the beneficial effects that the application has low requirements on the hardware of a controller, and an ordinary single-chip microcomputer can be used for realization, thereby saving cost, space and energy. The application does not need to establish a mathematical model of a control object, can save a large amount of time cost, and is difficult to establish a bionic model. The application can realize roll control of the robot fish without expert experience for establishing a rule base, and the Q table can learn "expert experience" during training.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of underwater biomimetic robots, and relates to a roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias. Background Technology

[0002] With the development of social science and technology and the increasingly severe ecological and environmental problems facing humanity, the demand for marine environmental exploration and resource extraction is increasing. Autonomous underwater vehicles (AUVs) have promising scientific and engineering applications, enabling tasks such as underwater environmental monitoring, scientific research, underwater archaeology, and resource development in near and far seas. The manta ray-inspired underwater robotic fish is a novel type of biomimetic AUV. Compared to traditional propeller propulsion systems, it exhibits better environmental compatibility, higher maneuverability, and stronger adaptability to complex environments. In recent years, the control problems of biomimetic underwater robots have attracted the research interest and enthusiasm of scholars, and related scientific and technological issues have also seen significant development.

[0003] Controlling underwater biomimetic robots is one of the challenges in this field. Roll control, as part of high-maneuverability attitude control, is essential for underwater biomimetic robots to navigate through narrow areas and is a fundamental requirement for controlling operations in confined underwater environments. Currently, underwater roll control methods can be categorized into model-based and model-free methods, depending on whether a model is required. Model-based methods require establishing a mathematical model of the controlled object. However, the dynamic models of biomimetic underwater robots are more complex than those of traditional rigid rotating AUVs, with more pronounced coupling between the flexible body and the fluid, making model creation more difficult. Therefore, model-free control methods are a better choice.

[0004] Fuzzy control methods have been successfully applied to biomimetic underwater robots. The advantage of fuzzy control is that it does not require building a model of the controlled object; instead, it only requires designing a fuzzy rule base based on human experience, and then controlling the object according to these rules. The disadvantages of fuzzy control are its reliance on expert experience and relatively low control accuracy. Compared to fuzzy control, the most popular model-free control method currently being researched is reinforcement learning (RL). In 1956, Bellman proposed dynamic programming (DP), in 1988 Sutton proposed the temporal difference (TD) algorithm, and in 1992, Watkins proposed the Q-learning algorithm. In 2013, Mnih et al. combined deep learning and reinforcement learning to propose the first deep reinforcement learning algorithm, which was successfully applied to Atari games. Some games played using RL algorithms have surpassed the level of professional human players. In 2016, Deep Blue Academy's reinforcement learning algorithm, AlphaGo, successfully defeated the human world champion.

[0005] Currently, there are still many challenges in applying RL methods to real-world prototypes. One of these is how to implement reinforcement learning control in systems with relatively low computing power. For example, many deep reinforcement learning algorithms require industrial control computers with high GPU computing power as the control system hardware, while currently many controllers still use microcontrollers such as single-chip microcomputers as the main control unit. Summary of the Invention

[0006] Technical problems to be solved

[0007] To avoid the shortcomings of existing technologies, this invention proposes a roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias.

[0008] Technical solution

[0009] A roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias is characterized by the following steps:

[0010] Step 1: Establish a finite Markov model for the roll control problem of the manta ray-inspired robotic fish:

[0011] 1. Establish the discrete state space S of roll control:

[0012] S={s -n s -n+1 , ..., s -1 s o ,s1,...,s n-1 s n}

[0013] in:

[0014]

[0015]

[0016] ……,

[0017]

[0018]

[0019]

[0020] ……,

[0021]

[0022] Among them, the sensor obtains the roll angle deviation. The range of values ​​is

[0023] 2. Establish discrete motion space A: The motion space for the roll control of the manta ray robotic fish is a continuous motion space [-a] min +a max Discretize the control variables into 2n+1 actions, and create a discretized action space:

[0024] A={a -n a -n+1 , ..., a -1 a0, a1, ..., a n-1 a n}

[0025] Where 'a' represents an element of the motion space, representing the sum of the fixed angles of the left and right pectoral fin amplitude offsets, and -180°≤a≤180°, the left and right pectoral fins use symmetrical offset angles as the response of the control quantity;

[0026] 3. Design the reward function:

[0027]

[0028] Where: δ is a parameter for adjusting the depth control accuracy of the manta ray-inspired robotic fish;

[0029] The r t+1 Let the state be s at time t. t The action is a t The state at the next moment is s t+1 The reward received;

[0030] Step 2: Create the action-value function table Q(s, a):

[0031]

[0032] Step 3: Train the Q-table:

[0033] 1. Training data samples: Using the original depth and pose change data, calculate the reward function r at time t. t+1 (s t a t ); Create the dataset Data for training the Q table. test Let the storage size be N, which stores state transition information (Transition).

[0034] in:

[0035] Data test ={Transition1, Transition2,…,Transition N}

[0036] Transitioni ={s t a t s t+1 r t+1}

[0037] 2. Q(s) n a n The updated formula is:

[0038]

[0039] Where α is the learning rate and γ is the discount factor;

[0040] 3. Randomly sample data from the training dataset, according to Q(s) n a n The formula is updated and the table is updated cyclically until the error in the change of the Q value in the Q table is less than the specified value ε.

[0041] Step 4: Design control strategy:

[0042]

[0043] When the state is s, choose action a that maximizes Q(s, a).

[0044] The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

[0045] The roll deviation The result is: feedback of the roll angle obtained through the sensor is... Assuming the roll angle fed back by the sensor is the true roll value, the desired roll value range is: Therefore, it can be deduced that...

[0046] The when At that time, based on the 360° period of angle change, it is equivalent to Similarly, when At that time, it is equivalent to therefore The range of values ​​can be equivalent to

[0047] Step 3, training the Q-table, uses p y The pseudocode for training Q-tables with thon is shown below:

[0048] 1) Initialize the learning rate α and the discount factor γ;

[0049] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0050] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0051] 4) Repeat steps 5-7:

[0052] 5) Randomly sample transitions from the experience pool and perform operations according to Q(s). n a n Update the formula to update Q(s, a);

[0053] 6) Calculation

[0054] 7) When The loop terminates;

[0055] 8) Output all Q(s, a) in the Q table, and training ends.

[0056] The value of ε is adjusted according to the training situation.

[0057] The value of ε is ε = 0.1.

[0058] The value of δ is 2° to 5°.

[0059] Beneficial effects

[0060] This invention proposes a roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias. Compared with traditional control methods, this invention does not require building a model of the controlled object. It collects experimental data, trains a Q-table offline, and then imports the table into the prototype controller. The control variables are obtained by looking up the table, controlling the prototype to swim at a fixed depth. Compared with other reinforcement learning control methods, it has lower hardware resource requirements, smaller space requirements, lower power consumption, and is easier to implement. Its beneficial effects are:

[0061] 1) The implementation of the method of the present invention has low hardware requirements for the controller, and can be achieved by ordinary microcontrollers. In addition to saving costs, it also saves space and energy.

[0062] 2) Compared with the traditional classical PID control method, it does not require the establishment of a mathematical model of the controlled object. Since biomimetic models are more difficult to establish, a lot of time and cost can be saved.

[0063] 3) Compared with roll control methods based on fuzzy control, this method does not require expert experience to build a rule base to achieve roll control of the robotic fish. The Q table will learn "expert experience" on its own during training. Attached Figure Description

[0064] Figure 1 This is a flowchart of the manta ray robotic fish roll control method of the present invention;

[0065] Figure 2 This is a flowchart of the Q-table training program of the present invention;

[0066] Figure 3 This is a schematic diagram of the manta ray-inspired robotic fish roll control method of the present invention. Detailed Implementation

[0067] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0068] This invention proposes a model-free manta ray-inspired robotic fish roll control method based on Q-learning and pectoral fin amplitude bias, characterized by the following steps:

[0069] 1. Establishing a finite Markov model for the roll control problem of a manta ray-inspired robotic fish:

[0070] Step 1-1) Establish the discrete state space S of the roll control:

[0071] Feedback on the roll angle obtained through sensors is as follows: Assuming the roll angle fed back by the sensor is the true roll value, the desired roll value range is: From this, we can deduce the roll deviation. The range of values ​​for:

[0072] when At that time, based on the 360° period of angle change, it is equivalent to Similarly, when At that time, it is equivalent to therefore The range of values ​​can be equivalent to

[0073] In practical applications, the roll angle range of the manta ray-inspired robotic fish is (-180°, 180°). The state space is discretized and simplified to 2n+1 elements. That is, let the state space...

[0074] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n},

[0075] in,

[0076]

[0077]

[0078] ……,

[0079]

[0080]

[0081]

[0082] ……,

[0083]

[0084] in, The range of values ​​can be adjusted according to the actual situation; one simple way is to take...

[0085] Step 1-2) Establish the discrete action space A:

[0086] The actual motion space for the roll control of the manta ray robotic fish is a continuous motion space [-a] min +a max To simplify the model, while meeting the control requirements, the control variables are discretized into 2n+1 actions. Let the discrete action space be A = {a -n a -n+1 , ..., a -1 a0, a1, ..., a n-1 a n}

[0087] Where 'a' represents an element of the motion space, representing the sum of fixed angles of the left and right pectoral fin amplitude offsets, and -180°≤a≤180°. The left and right pectoral fins use symmetrical offset angles as the response of the control quantity. For example, when a=90°, it means that the left pectoral fin amplitude is offset by -45° and the right pectoral fin amplitude is offset by 45°.

[0088] Steps 1-3) Design the reward function calculation method:

[0089] At time t, the state is s. t The action is a t The state at the next moment is s t+1 The reward received is r t+1 ,

[0090] r t+1 The calculation formula is:

[0091]

[0092] δ = 2°, which affects the roll control accuracy of the manta ray-inspired robotic fish; its size can be changed according to the control accuracy.

[0093] 2: Establish the action-value function table Q(s, a); Table 1

[0094]

[0095] 3: Training the Q-table;

[0096] Step 3-1) Training data sample preparation, the specific steps include:

[0097] 1) Experimental data such as original depth and attitude changes obtained through experimental methods or by previous experiments;

[0098] 2) Calculate the reward function r at time t according to the formula in step 1. t+1 (s t a t );

[0099] 3) Create the dataset for training the Q-table. test Let the storage size be N, which stores state transition information (Transition).

[0100] in

[0101] Data test ={Transition1, Transition2,…,Transition N},

[0102] Transition i ={s t a t s t+1 r t+1};

[0103] Step 3-2) Q(s) n a n The updated formula is:

[0104]

[0105] Where α is the learning rate and γ is the discount factor;

[0106] Step 3-3) Randomly extract data from the training dataset and update the table iteratively according to the calculation formula in step 3-2) until the error of the change in the Q value in the Q table is less than the specified value ε = 0.1;

[0107] The pseudocode for the program flowchart of training Q-tables using Python is as follows:

[0108] 1) Initialize the learning rate α and the discount factor γ;

[0109] 2) Assign an initial value of 0 to all Q(s, a) tables in Q;

[0110] 3) Set the experience pool storage capacity N, read in the original training data, process it, and then store it in the experience pool.

[0111] 4) Loop:

[0112] 5) Randomly sample Transitions from the experience pool and update Q(s, a) according to the formula in 3-2);

[0113] 6) Calculation

[0114] 7) When The loop terminates;

[0115] 8) Output all Q(s, a) in the Q-table, training ends.

[0116] 4. Design control strategy:

[0117]

[0118] This indicates that when the state is s, the action 'a' is chosen to maximize Q(s, a).

[0119] The embodiments of the present invention are described in detail below. The accompanying drawings of the embodiments are shown in the drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0120] like Figure 1 As shown, the first step of this invention is to model the roll control problem of the manta ray robotic fish as a Markov decision model: 1) establish the state space S of the roll control of the manta ray robotic fish; 2) establish the action space A of the roll control; 3) establish the reward function.

[0121] 1. Establish a Markov decision model for the roll control of a manta ray-inspired robotic fish:

[0122] 1-1) Establish the state space:

[0123] S={s -n s -n+1 , ..., s -1 s0, s1, ..., s n-1 s n},

[0124] Let n = 10, and set another...

[0125] s- 10 = (-180°, -50°)

[0126] s-9 = (-50°, -45°)

[0127] s-8 = (-45°, -40°)

[0128] …

[0129] s -2 = (-15°, -10°)

[0130] s -1 = (-10°, -5°)

[0131] s0 = (-5°, 5°),

[0132] s1 = [5°, 10°),

[0133] s2 = [10°, 15°),

[0134] …,

[0135] s9 = [45°, 50°),

[0136] s 10 = [50°, 180°)

[0137] 1-2) Establish the motion space A for the roll control of the manta ray-inspired robotic fish:

[0138] In this embodiment, the adjustable range of the amplitude of the pectoral fin on one side of the manta ray-inspired robotic fish is [-80°, +80°]. Roll control adjusts the pectoral fin offset based on straight swimming. When swimming straight, both pectoral fins maintain a constant amplitude of 50°, therefore the adjustable range of offset is [-40°, +40°]. When rolling to the left, the left pectoral fin amplitude offset is positive, and the right pectoral fin amplitude offset is negative, with the absolute values ​​of the left and right offsets being equal; similarly, when rolling to the right, the right pectoral fin offset is positive, and the left pectoral fin offset is negative, with the absolute values ​​of the left and right offsets being equal.

[0139] Let the motion space A = {a0, a1, a2, a3, a4, a5}, where a0 = 10°, a1 = 20°, a2 = 30°, a3 = 40°, a4 = 50°, and a5 = 60°. These parameters can be changed according to experimental results and accuracy requirements. When performing a left roll, if a = a1 = 20°, it means that the amplitude offset of the left pectoral fin is 10° and the offset of the right pectoral fin is -10°.

[0140] 1-3) Design the reward function

[0141] According to formula (2), taking δ = 5°, the reward function is:

[0142]

[0143] 2: Establish the action-value function table Q(s, a);

[0144] Based on Table 1, the state space, and the action space, construct the Q-table:

[0145] Table 2:

[0146]

[0147] Step 3-2) Establish a training environment based on Python;

[0148] Step 3-3) Update the Q-table based on the Q-algorithm. The algorithm flow is as follows: Figure 2 As shown:

[0149] 1) Program initialization: Initialize Q for any state and action. t (s t a t =0; Initialize learning rate α = 0.1, discount factor γ = 0.9, training cutoff condition ε = 0.1, experience pool storage capacity N = 3000;

[0150] 2) Data Processing: Read the training data and calculate the reward function, converting the data into {s} t a t r t+1 s t+1}form;

[0151] 3) Store the data in experience pool D;

[0152] 4) Loop:

[0153] Randomly sample data from experience pool D;

[0154] Update Q t (s t a t According to formula (3)

[0155] The calculation is performed for all states and actions according to formula (4).

[0156] if If the value is less than ε, exit the loop; otherwise, continue the loop.

[0157] 5) Output the Q-table; training ends.

[0158] 4. Design control strategy:

[0159] like Figure 3 The diagram shown illustrates the principle of the roll control method of this invention. A depth sensor provides the current depth of the controlled object, which is then compared with the target heading to output the depth error. The input is fed into the controller for calculation, and the current state s is determined. The controller then follows the control strategy.

[0160]

[0161] The action 'a' that maximizes the state value function is retrieved from the Q table and used as the output of the pectoral fin roll control.

[0162] The test results confirmed that the heading was set at 10°.

Claims

1. A roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias, characterized by the following steps: Step 1: Establish a finite Markov model for the roll control problem of the manta ray-inspired robotic fish: 1.1 Establishing the discrete state space of roll control : in: ， ， ……， ， ， ， ……， Among them, the sensor obtains the roll angle deviation. The range of values ​​is , ; 1.2 Establishing a Discrete Action Space The motion space for the roll control of the manta ray-inspired robotic fish is a continuous motion space. Discretize the control variables into 2n+1 actions, and create a discretized action space: in The element representing the motion space represents the sum of the angles of the fixed amplitude offset of the left and right pectoral fins, and The left and right pectoral fins use symmetrical offset angles as the control quantity response; 1.3 Design the reward function: in: To adjust the parameters for depth control accuracy of the manta ray-inspired robotic fish; ； Step 2: Create an action-value function table : Step 3: Train the Q-table: 3.1 Training Data Samples: Using the original depth and pose change data, calculate... Time-based reward function ; Establish training The table's dataset Let the storage capacity be... It stores state transition information. ; in: 3.2 The updated formula is: in, For learning rate, Discount factor; 3.

3. Randomly extract data from the training dataset, according to... The formula is updated and the table is updated iteratively until the error in the change of the Q value in the Q table is less than the specified value ε. Step 4: Design control strategy: When the state is s, choose the option that makes Action a that achieves the maximum value; The roll deviation The result is: feedback of the roll angle obtained through the sensor is... Let the roll angle fed back by the sensor be the true roll value, and the expected range of roll values ​​be... Therefore, it can be deduced that , ; The when At that time, the period of angle change is It can be seen that it is equivalent to Similarly, when At that time, it is equivalent to ,therefore The range of values ​​can be equivalent to ; The The value ranges from 2° to 5°.

2. The roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias according to claim 1, characterized in that: The original depth and attitude change experimental data were obtained through experimental methods or by previous experiments.

3. The roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias according to claim 1, characterized in that: Step 3, training the Q-table, uses Python to train the Q-table. The pseudocode of the program flowchart is as follows: 1) Initialize the learning rate and discount factor ; 2) Assign values ​​to all tables in Q. The initial value is 0; 3) Set the experience pool storage capacity Read in the raw training data, process it, and then store it in the experience pool. ; 4) Repeat steps 5-7: 5) Randomly sample from the experience pool And in accordance with Formula update ; 6) Calculation 7) When The loop terminates; 8) Output all values ​​in the Q table. Training is over.

4. The roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias according to claim 1 or 3, characterized in that: The value of ε is adjusted according to the training situation.

5. The roll control method for a manta ray-inspired robotic fish based on Q-learning and pectoral fin amplitude bias according to claim 1 or 3, characterized in that: The value of ε is ε=0.1.

Images