Underwater glider motion control method based on multi-agent reinforcement learning

Abstract

The invention discloses an underwater glider motion control method based on multi-agent reinforcement learning, and the method comprises the steps: deploying a final first decision network # imgabs0 # and a final second decision network # imgabs1 # in an underwater glider, and calculating an optimal control quantity: inputting a target operation speed (u, v, w) into # imgabs2 #, and outputting a buoyancy adjustment control quantity Bb and a pitching adjustment control quantity lp; inputting (u, v, w, Bb, lp) into # imgabs3, outputting a rolling adjustment control quantity, # imgabs4, and completing the navigation of the current section according to # imgabs5; judging whether the observation task is completed; calculating the ocean current interference speed in the current profile; compensating (u, v, w) according to the ocean current interference speed; and taking the compensated speed (u ', v, w') as the target running speed of the next section. According to the invention, the motion control of the underwater glider is realized, the precision of the motion control is improved, and the coupling problem among the control quantities of the underwater glider is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater robot control, and in particular relates to an underwater glider motion control method based on multi-agent reinforcement learning. Background Art

[0002] In recent decades, with the development of marine technology and the growing demand for underwater detection, underwater gliders have attracted the interest of more and more researchers and have become one of the most effective underwater unmanned ocean observation platforms.

[0003] Underwater gliders have the characteristics of long flight time. Under normal circumstances, underwater gliders achieve zigzag gliding motion in the longitudinal section by adjusting their own net buoyancy and pitch angle, and with the help of the hydrodynamic force acting on the fixed wings. Underwater gliders achieve turning motion by changing their roll attitude. Underwater gliders have the advantages of low energy consumption, wide navigation range, long endurance time and low noise.

[0004] Due to the highly nonlinear and cross-coupled system dynamics of underwater gliders, unpredictable external disturbances caused by ocean currents, and the uncertain internal dynamics of underwater gliders, it is difficult to establish an accurate dynamic model for each underwater glider and perform motion control on the underwater glider. In addition, the system dynamics, external disturbances, and internal dynamic factors all need to be accurately processed, otherwise the maneuverability of the underwater glider will be affected, which will lead to unstable motion control of the underwater glider. Traditional control methods require accurate determination of the dynamic model before they can achieve motion control of the underwater glider. Moreover, the accuracy of the motion control of the underwater glider by traditional control methods is low, which cannot meet the requirements of high-precision control, and the motion control process is complex. In summary, the traditional control method based on the dynamic model requires the construction of an accurate and complex dynamic model. This traditional control method is easily disturbed by external environmental noise, and the accuracy of the motion control achieved is low. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a multi-agent reinforcement learning model.

[0006] Another object of the present invention is to provide a training method for a multi-agent reinforcement learning model.

[0007] Another object of the present invention is to provide an underwater glider motion control method based on multi-agent reinforcement learning.

[0008] The underwater glider motion control method of the present invention is based on the trained multi-agent reinforcement learning model (the final first decision network and finally the second decision network ) calculates the optimal control quantity for the underwater glider under the current profile, and compensates the target running speed of the next profile according to the ocean current interference speed of the current profile, thereby achieving high-precision motion control of the underwater glider.

[0009] The present invention is achieved through the following technical solutions.

[0010] A multi-agent reinforcement learning model, comprising: a first decision network of agent 1, a first evaluation network of agent 1, a second decision network of agent 2, and a second evaluation network of agent 2;

[0011] Input of the first decision network is the velocity u, v, and w of the underwater glider along the X-axis, Y-axis, and Z-axis in the inertial coordinate system in time step t; the output of the first decision network is the buoyancy control quantity B of the underwater glider in time step t b and pitch adjustment control amount l p ;

[0012] The input of the first evaluation network is (u, v, w, B b , l p ); the output of the first evaluation network is the first evaluation value

[0013] The input of the second decision network is The output of the second decision network is the roll adjustment control quantity of the underwater glider in time step t

[0014] The input of the second evaluation network is The output of the second evaluation network is the second evaluation value

[0015] Among them, the parameters of the first decision network are Parameters of the second decision network The parameters of the first evaluation network are The parameters of the second evaluation network are

[0016] In the above technical solution, the first decision network, the first evaluation network, the second decision network and the second evaluation network all adopt fully connected neural networks.

[0017] A training method for a multi-agent reinforcement learning model, comprising: using a Markov chain dataset D as input of the multi-agent reinforcement learning model, training the multi-agent reinforcement learning model N times, and obtaining a first decision network of agent 1 as a final first decision network. and the second decision network of agent 2 as the final second decision network

[0018] The Markov chain dataset D consists of multiple Markov chain sample data, each of which includes: and

[0019] Among them, the underwater glider randomly performs real gliding motion for the i-th time for The underwater glider randomly performs real gliding motion for the jth time for

[0020] A training method for a multi-agent reinforcement learning model is described. The specific steps of the training process using a set of Markov chain sample data as input to the multi-agent reinforcement learning model are as follows:

[0021] Step 1: Convert the Markov chain sample data into and Input to the first evaluation network, the first evaluation network outputs the first evaluation value; according to the loss function of the first evaluation network, the first evaluation value and the reward value r in the Markov chain sample data are used t (1) Calculate the loss function value of the first evaluation network and adjust the parameters of the first evaluation network Update and get parameters Then the updated first evaluation network Q′1 is obtained;

[0022] The Markov chain sample data and Input to the second evaluation network, the second evaluation network outputs the second evaluation value; according to the loss function of the second evaluation network, the second evaluation value and the reward value r in the Markov chain sample data are used t (2) Calculate the loss function value of the second evaluation network and adjust the parameters of the second evaluation network Update and get parameters Then the updated second evaluation network Q′2 is obtained;

[0023] Step 2: Convert the Markov chain sample data Input to the first decision network, the first decision network output Will and As Input to the second decision network, the second decision network output

[0024] Step 3: Convert the Markov chain sample data into and Input to the updated first evaluation network, the updated first evaluation network outputs the first evaluation value Will and Input to the updated second evaluation network of agent 2, the updated second evaluation network outputs the second evaluation value

[0025] Step 4: The first evaluation value Substitute into the gradient calculation formula of the first decision network to calculate the gradient value of the first decision network Using gradient values Parameters of the first decision network Update and get parameters Then the updated first decision network μ′1 is obtained;

[0026] The second evaluation value Substitute into the gradient calculation formula of the second decision network to calculate the gradient value of the second decision network Using gradient values Parameters of the second decision network Update and get parameters Then the updated second decision network μ′2 is obtained;

[0027] Step 5: Convert the Markov chain sample data into Input to the updated first decision network, the output of the updated first decision network is Will and As Input to the updated second decision network, the output of the updated second decision network is

[0028] Step 6, and the Markov chain sample data Substitute into the guided loss function of agent 1, calculate the guided loss function value, and update the parameters of the first decision network Update again to get the parameters Then we get the first decision network after the second update

[0029] Will and the Markov chain sample data Substitute into the guided loss function of agent 2, calculate the guided loss function value, and update the parameters of the second decision network Update again to get the parameters Then we get the second decision network after the second update

[0030] Among them, the guided loss function of agent 1 is

[0031]

[0032] The guided loss function of agent 2 is

[0033]

[0034] Represents the optimization of the parameters of the first decision network minimize and the differences between; Represents the optimization of the parameters of the second decision network minimize and The difference between.

[0035] A motion control method for an underwater glider based on multi-agent reinforcement learning, comprising:

[0036] Step 1: The final first decision network of agent 1 and the final second decision network of Agent 2 Deployed in the actual observation mission of the underwater glider, the optimal control quantity is calculated:

[0037] In the process of the underwater glider actually performing the observation mission, before each section enters the water, the target running speed (u, v, w) in the observation mission is used as the final first decision network The input of the first decision network The output is the buoyancy adjustment control quantity B b and pitch adjustment control amount l p ;(u,v,w,B b , l p ) as the final second decision network The input of the second decision network Output roll adjustment control amount B b , l p , Input to the underwater glider, the underwater glider according to B b , l p , Complete the navigation of the current section;

[0038] Step 2: Determine whether the underwater glider has completed the observation mission in the current profile: If the straight-line distance between the underwater glider's position and the observation mission target point is less than s, the underwater glider has completed the navigation mission and the motion control of the underwater glider is stopped; otherwise, execute step 3;

[0039] Step 3: After completing the navigation of the current profile, calculate the ocean current interference speed f of the underwater glider in the current profile;

[0040] Step 4: Compensate the speed of the underwater glider in the X-axis and Z-axis directions of the next profile according to the ocean current interference speed f calculated for the current profile. No speed compensation is performed for the first profile movement.

[0041] Substitute the compensated speed (u′, v, w′) as the target running speed of the next profile into step 1.

[0042] A motion control method for an underwater glider based on multi-agent reinforcement learning is proposed. The calculation formula for the ocean current interference speed f of the underwater glider in the current profile is:

[0043]

[0044] Wherein, x is the expected navigation displacement of the underwater glider along the X-axis in the section, x′ is the actual navigation displacement of the underwater glider along the X-axis in the section; z is the expected navigation displacement of the underwater glider along the Z-axis in the section, z′ is the actual navigation displacement of the underwater glider along the Z-axis in the section; T is the operating time of the underwater glider in the section.

[0045] In the above technical solution, the calculation formula for the target running speed in the X-axis direction and the Z-axis direction after compensation is:

[0046] u′=u+f u

[0047] w′=w+f w

[0048] Among them, u′ is the speed of the current profile along the X-axis after compensation, u is the target running speed of the current profile along the X-axis, and f u is the component of the ocean current interference velocity f in the X-axis direction; w′ is the velocity of the current profile along the Z-axis after compensation, w is the target running speed of the current profile along the Z-axis direction, f w is the component of the ocean current interference velocity f in the Z-axis direction.

[0049] The present invention has the following advantages due to the adoption of the above technical solution:

[0050] 1. The underwater glider motion control method of the present invention establishes and trains a multi-agent reinforcement learning model, and uses the trained multi-agent reinforcement learning model to predict the control quantity to achieve motion control of the underwater glider, thereby avoiding the traditional control method of directly establishing a complex dynamic model of the underwater glider and solving the dynamic model.

[0051] 2. The underwater glider motion control method of the present invention adopts a multi-agent reinforcement learning model to solve the control quantity of the underwater glider, and uses two agents in the multi-agent reinforcement learning model to output the control quantity, among which the decision network of agent 1 outputs the buoyancy adjustment control quantity and the pitch adjustment control quantity, and the decision network of agent 2 outputs the roll adjustment control quantity, thereby realizing the motion control of the underwater glider, improving the accuracy of the motion control of the underwater glider, and avoiding the coupling problem between the various control quantities of the underwater glider.

[0052] 3. The underwater glider motion control method of the present invention has a small amount of computational complexity. The decision network and evaluation network of the two agents in the multi-agent reinforcement learning model do not require complex fully connected neural networks to achieve high-precision error control. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is the process of simulating different motion states of the underwater glider and building a real database in the present invention;

[0054] Figure 2 The process of solving the optimal control variables for the multi-agent reinforcement learning model of the present invention;

[0055] Figure 3 This is the overall flow chart of the motion control of the underwater glider according to the present invention;

[0056] Figure 4 The figure is a comparison diagram of the expected path and the actual path of the present invention during an actual sea trial. DETAILED DESCRIPTION

[0057] The following is a detailed description of an underwater glider motion control method based on multi-agent reinforcement learning of the present invention with reference to the accompanying drawings.

[0058] Example 1

[0059] A multi-agent reinforcement learning model (Arulkumaran K, Deisenroth MP, Brundage M, et al. Deep reinforcement learning: A brief survey [J]. IEEE Signal Processing Magazine, 2017, 34(6): 26-38.) is used to output the optimal control quantity for an underwater glider. The multi-agent reinforcement learning model includes: a first decision network of agent 1, a first evaluation network of agent 1, a second decision network of agent 2, and a second evaluation network of agent 2;

[0060] Input of the first decision network is the velocity u along the X axis, the velocity v along the Y axis, and the velocity w along the Z axis of the underwater glider in the inertial coordinate system within the t time step, that is, The output of the first decision network is the buoyancy control quantity B of the underwater glider in time step t b and pitch adjustment control amount l p ,Right now

[0061] The input of the first evaluation network is (u, v, w, B b , l p )(Right now ); the output of the first evaluation network is the first evaluation value

[0062] The input of the second decision network is The output of the second decision network is the roll adjustment control quantity of the underwater glider in time step t Right now

[0063] The input of the second evaluation network is The output of the second evaluation network is the second evaluation value

[0064] Among them, the parameters of the first decision network are Parameters of the second decision network The parameters of the first evaluation network are The parameters of the second evaluation network are

[0065] The structures of the first decision network, the first evaluation network, the second decision network, and the second evaluation network in the multi-agent reinforcement learning model are as follows:

[0066] The first decision network uses a fully connected neural network, including: an input layer, two hidden layers, and an output layer connected in sequence. The activation function from the input layer to the hidden layer is the ReLu function, and the activation function from the hidden layer to the output layer is the Tanh function. The number of neurons in the input layer of the first decision network is 3, the number of neurons in both hidden layers of the first decision network is 200, the number of neurons in the output layer of the first decision network is 2, and the number of parameters of the first decision network is 81602. The calculation formula is: (3×200+200)+(200×200+200)+(200×200+200)+(200×2+2);

[0067] The first evaluation network adopts a fully connected neural network, including: an input layer, two hidden layers, and an output layer connected in sequence. The activation function from the input layer to the hidden layer is the ReLu function, and the activation function from the hidden layer to the output layer is the Tanh function. The number of neurons in the input layer of the first evaluation network is 5, the number of neurons in the two hidden layers of the first evaluation network is 200, the number of neurons in the output layer of the first evaluation network is 1, and the number of parameters of the first evaluation network is 81801. The calculation formula is: (5×200+200)+(200×200+200)+(200×200+200)+(200×1+1);

[0068] The second decision network uses a fully connected neural network, consisting of an input layer, two hidden layers, and an output layer. The activation function from the input layer to the hidden layer of the second decision network is the ReLu function, and the activation function from the hidden layer to the output layer of the second decision network is the Tanh function. The input layer of the second decision network has 5 neurons, the two hidden layers of the second decision network have 200 neurons each, and the output layer of the second decision network has 1 neuron. The number of parameters of the second decision network is: (5×200+200)+(200×200+200)+(200×200+200)+(200×1+1).

[0069] The second evaluation network is constructed using a fully connected neural network, including: one input layer, two hidden layers and one output layer. The activation function from the input layer to the hidden layer of the second evaluation network is the ReLu function, and the activation function from the hidden layer to the output layer of the second evaluation network is the Tanh function; the number of neurons in the input layer of the second evaluation network is 6, the number of neurons in the two hidden layers of the second evaluation network is 200, the number of neurons in the output layer of the second evaluation network is 1, and the number of parameters of the second evaluation network is 82001. The calculation formula is: (6×200+200)+(200×200+200)+(200×200+200)+(200×1+1).

[0070] Example 2

[0071] The method for obtaining the Markov chain data set D includes S1 to S2, and the specific steps are as follows:

[0072] S1, perform the following operation h times to obtain h samples and form a real database: Figure 1 As shown, the buoyancy control amount B of the underwater glider is adjusted b , pitch adjustment control amount l p and roll adjustment control amount Within the allowed range, a set of B is randomly generated bi ′, l pi ′, (B bi ′ is the randomly generated buoyancy adjustment control quantity, l pi ′ is the randomly generated pitch adjustment control value, is the randomly generated roll adjustment control amount), so that the underwater glider follows the group B bi ′, l pi ′, Perform a gliding motion, observe and record the velocity u along the X-axis in the inertial coordinate system during the underwater glider's motion. i ′, velocity v along the Y axis i ′ and the velocity w along the Z axis i ′, get a sample corresponding to the gliding motion

[0073] In this embodiment, h=5×10 4 , buoyancy adjustment control amount B b The allowable range is -7N to 7N, and the pitch adjustment control amount is l p The allowable range is -0.2m to 0.2m, and the roll adjustment control amount The allowed range is

[0074] S2, repeat steps S2-1 to S2-4 n times to obtain n Markov chain sample data, and form a Markov chain data set D, where n≤h 2 , each Markov chain sample data is independent and identically distributed;

[0075] S2-1, randomly select two samples from the real database and

[0076] in, and They represent the i-th sample and the j-th sample in the real database respectively. The value range of i and j is an integer between 1 and h, and i and j are not equal;

[0077] S2-2, the sample (u i ,v i ,w i ) as the input of the first decision network, that is (B bj , l pj ) as the output of the first decision network, i.e.

[0078] will (u i , v i , w i , B bj , l pj ) as the input of the second decision network, that is Will As the output of the second decision network,

[0079] S2-3, according to (u i , v i , w i ) and (u j , v j , w j ), calculate the reward value r of agent 1 and agent 2 respectively t (1) and reward value r t (2) , the calculation formula is as follows:

[0080] r t (1) =-(|u i -u j |+|v i -v j |) 2

[0081] r t (2) =-|w i -w j |2

[0082] S2-4, make S2-2 and r in S2-3 t (1) and r t (2) A set of Markov chain sample data is formed, wherein the Markov chain length is set to 1, that is, one Markov chain sample data represents one round.

[0083] Example 3

[0084] Based on Example 2, a training method for a multi-agent reinforcement learning model includes:

[0085] Step 1: Use the Markov chain dataset D as the input of the multi-agent reinforcement learning model, train the multi-agent reinforcement learning model N times, and obtain the first decision network of agent 1 as the final first decision network. and the second decision network of agent 2 as the final second decision network

[0086] In this embodiment, the number of training times N is set to 1×10 4 .

[0087] Take a set of Markov chain sample data As the input of the multi-agent reinforcement learning model, the specific steps of the training process are as follows:

[0088] Step 1-1, the Markov chain sample data and Input to the first evaluation network, the first evaluation network outputs the first evaluation value; according to the loss function of the first evaluation network, the first evaluation value and the reward value r in the Markov chain sample data are used t (1) Calculate the loss function value of the first evaluation network and adjust the parameters of the first evaluation network Update and get parameters Then the updated first evaluation network Q′1 is obtained;

[0089] The Markov chain sample data and Input to the second evaluation network, the second evaluation network outputs the second evaluation value; according to the loss function of the second evaluation network, the second evaluation value and the reward value r in the Markov chain sample data are used t (2) Calculate the loss function value of the second evaluation network and adjust the parameters of the second evaluation network Update and get parameters Then the updated second evaluation network Q′2 is obtained;

[0090] Among them, the loss function of the first evaluation network is constructed according to the temporal difference method (Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533.). The loss function of the first evaluation network is:

[0091]

[0092] The loss function of the second evaluation network is constructed according to the temporal difference method. The loss function of the second evaluation network is

[0093]

[0094] in, is the first evaluation value output by the first evaluation network, is the second evaluation value output by the second evaluation network, r t (1) and r t (2) is the reward value in the Markov chain sample data, Represents expectations.

[0095] Step 1-2: convert the Markov chain sample data into Input to the first decision network, the first decision network output Will and As Input to the second decision network, the second decision network output

[0096] Step 1-3, the Markov chain sample data and Input to the updated first evaluation network, the updated first evaluation network outputs the first evaluation value

[0097] Will and Input to the updated second evaluation network of agent 2, the updated second evaluation network outputs the second evaluation value

[0098] Step 1-4: The first evaluation value Substitute into the gradient calculation formula of the first decision network to calculate the gradient value of the first decision network Using gradient values Parameters of the first decision network Update and get parameters Then the updated first decision network μ′1 is obtained;

[0099] The second evaluation value Substitute into the gradient calculation formula of the second decision network to calculate the gradient value of the second decision network Using gradient values Parameters of the second decision network Update and get parameters Then the updated second decision network μ′2 is obtained;

[0100] Among them, the gradient value of the first decision network The calculation formula is:

[0101]

[0102] Gradient value of the second decision network The calculation formula is:

[0103]

[0104] in, is the first evaluation value of the first evaluation network after update Parameters of the first decision network The gradient value of is the second evaluation value of the updated second evaluation network Parameters of the second decision network The gradient value of The first evaluation network after update The gradient value of The second evaluation network after update The gradient value of for Parameters of the first decision network The gradient value of represent Parameters of the second decision network The gradient value of .

[0105] Steps 1-5: convert the Markov chain sample data into Input to the updated first decision network, the output of the updated first decision network is Will and As Input to the updated second decision network, the output of the updated second decision network is

[0106] Steps 1-6, and the Markov chain sample data Substitute into the guided loss function of agent 1, calculate the guided loss function value, and update the parameters of the first decision network Update again to get the parameters Then we get the first decision network after the second update

[0107] Will and the Markov chain sample data Substitute into the guided loss function of agent 2, calculate the guided loss function value, and update the parameters of the second decision network Update again to get the parameters Then we get the second decision network after the second update

[0108] Among them, the guided loss function of agent 1 is

[0109]

[0110] The guided loss function of agent 2 is

[0111]

[0112] Represents the optimization of the parameters of the first decision network minimize and the differences between; Represents the optimization of the parameters of the second decision network minimize and The difference between.

[0113] Example 4

[0114] On the basis of Example 3, in order to improve the training efficiency of the multi-agent reinforcement learning model, 256 groups of Markov chain sample data are selected from the Markov chain data set D as input for training each time to obtain a trained multi-agent reinforcement learning model.

[0115] Example 5

[0116] Based on Example 4, a method for underwater glider motion control based on multi-agent reinforcement learning includes the following steps:

[0117] Step 1: The final first decision network of agent 1 and the final second decision network of Agent 2 Deployed in the actual observation mission of the underwater glider, the optimal control quantity is calculated:

[0118] In the process of the underwater glider actually performing the observation mission, before each section enters the water, the target running speed (u, v, w) in the observation mission is used as the final first decision network The input of the first decision network The output is the buoyancy adjustment control quantity B b and pitch adjustment control amount l p;(u,v,w,B b ,l p ) as the final second decision network The input of the second decision network Output roll adjustment control amount B b ,l p , Input to the underwater glider, the underwater glider according to B b , l p , Complete navigation of the current section.

[0119] Among them, through the final first decision network and finally the second decision network Get buoyancy adjustment control value B b , pitch adjustment control amount l p and roll adjustment control amount The process is as follows Figure 2 shown.

[0120] Step 2: Determine whether the underwater glider has completed the observation mission in the current profile: If the straight-line distance between the underwater glider's position and the observation mission target point is less than s, the underwater glider has completed the navigation mission and the motion control of the underwater glider is stopped; otherwise, execute step 3; in this embodiment, s = 2000 meters;

[0121] Step three, such as Figure 3 As shown in the figure, after completing the navigation of the current profile, the ocean current interference speed f of the underwater glider in the current profile is calculated:

[0122] In the marine environment, the influence of the ocean current in the Y-axis direction on the motion of the underwater glider is usually ignored. According to the running time of the underwater glider in the current profile, the navigation displacement error along the X-axis direction and the navigation displacement error along the Z-axis direction, the speed error Δu of the underwater glider along the X-axis direction and the speed error Δw along the Z-axis direction are calculated. Then, the calculation formula of the ocean current interference speed f of the underwater glider in the current profile is:

[0123]

[0124] Wherein, x is the expected navigation displacement of the underwater glider along the X-axis in the cross section, x′ is the actual navigation displacement of the underwater glider along the X-axis in the cross section; z is the expected navigation displacement of the underwater glider along the Z-axis in the cross section, z′ is the actual navigation displacement of the underwater glider along the Z-axis in the cross section; T is the operating time of the underwater glider in the cross section;

[0125] Step 4: Compensate the underwater glider's speed along the X-axis and Z-axis for the next profile based on the ocean current interference speed f calculated for the current profile. No speed compensation is performed for the first profile movement. Substitute the compensated speed (u′, v, w′) as the target operating speed for the next profile and execute step 1.

[0126] The calculation formula for the target running speed in the X-axis direction and the Z-axis direction after compensation is:

[0127] u′=u+f u

[0128] w′=w+f w

[0129] Among them, u′ is the speed of the current profile along the X-axis after compensation, u is the target running speed of the current profile along the X-axis, and f u is the component of the ocean current interference velocity f in the Y-axis direction; w′ is the velocity of the current profile along the Z-axis after compensation, w is the target running speed of the current profile along the Z-axis direction, f w is the component of the ocean current interference velocity f in the Z-axis direction;

[0130] Example 6

[0131] In the underwater glider simulation platform, 200 underwater glider observation tasks were performed according to the underwater glider motion control method of Example 5. Each underwater glider observation task was as follows:

[0132] The angle between the initial heading angle of the underwater glider and the X-axis is set to 0. According to the speed value range of the underwater glider along the (X, Y, Z) axes in the inertial coordinate system (0-0.5606, 0-0.6464, -0.0291-0.0291) m / s, a group of speed values ​​(u, v, w) are randomly selected as the target running speeds of the first profile motion, and the underwater glider motion control method of Example 5 is used to calculate the optimal control quantity, and the underwater glider is motion-controlled according to the optimal control quantity.

[0133] The speed error between the expected speed and the actual speed of each section of the underwater glider in each observation mission is calculated, and the speed error of each section is summed to obtain the mean speed error of the observation mission. The mean speed errors of the 200 underwater glider observation missions are then summed and averaged to obtain the average control speed error of (0.5195, 0.4703, 0.2149)×10 -4 m / s.

[0134] Comparative Example 1

[0135] The 200 sets of speed values ​​(u, v, w) described in Example 6 are respectively used as target running speeds, and an underwater glider motion control method (Juan R, Wang T, Liu S, et al. High-precision motion control of underwater gliders based on reinforcement learning [J]. Ocean Engineering, 2024, 310: 118603.) is used to control the motion of the underwater glider on an underwater glider simulation platform.

[0136] When the maximum diving depth of the underwater glider is 600m, the calculation amount of Example 6 is reduced by 67% compared with the calculation amount of Comparative Example 1. When the maximum diving depth of the underwater glider is 900m, the calculation amount of Example 6 is reduced by 79% compared with the calculation amount of Comparative Example 1. When the maximum diving depth of the underwater glider is 1300m, the calculation amount of Example 6 is reduced by 85% compared with the calculation amount of Comparative Example 1.

[0137] In summary, as the maximum diving depth of the underwater glider increases, the amount of calculation of the underwater glider motion control method of the present invention is reduced more than that of the comparative example 1.

[0138] The underwater glider motion control method (Example 6) of the present invention achieves a speed control error for the underwater glider that is similar to that of the underwater glider in Comparative Example 1. However, Example 6 has a smaller computational load, which shortens the underwater glider's surface waiting time and, consequently, shortens the total time it takes for the underwater glider to complete its mission.

[0139] Comparative Example 2

[0140] The 200 sets of speed values ​​(u, v, w) described in Example 6 are used as target running speeds. The underwater glider motion control method based on the fully connected neural network structure is used to predict the control amount of the underwater glider. The average control speed error of the 200 underwater glider observation tasks is calculated to be (7.3654, 3.0235, 1.5923)×10 -4 m / s.

[0141] The fully connected neural network structure includes: one input layer, two hidden layers and one output layer. The number of neurons in the input layer is 3, the number of neurons in the two hidden layers is 200, and the number of neurons in the output layer is 3.

[0142] Comparing Example 6, Comparative Example 1 and Comparative Example 2, it can be seen that:

[0143] Example 6 has lower control error. Under the influence of highly dynamic ocean currents, the motion control method proposed in this invention (Example 6) can calculate the optimal control quantity that best suits the current ocean environment. While meeting the requirements of actual sea trial missions, the motion control method proposed in this invention has precise target tracking performance.

[0144] Example 7

[0145] A method for underwater glider motion control based on multi-agent reinforcement learning is basically the same as Example 5.

[0146] The expected trajectory and actual trajectory of the underwater glider motion control method provided by the present invention are compared in actual ocean tests. Figure 4 As shown, during this ocean test, the underwater glider conducted a total of 30 profile movements, among which the average maximum diving depth of the underwater glider was about 700 meters, the average diving time was about 80 minutes, the average surfacing time was about 80 minutes, and the navigation error of the underwater glider was 1.11 km.

[0147] The above is an exemplary description of the present invention. It should be noted that, without departing from the core of the present invention, any simple deformation, modification or other equivalent replacement that can be made by other skilled in the art without expending creative labor falls within the scope of protection of the present invention.

Claims

1. A multi-agent reinforcement learning model, characterized in that: include: The first decision network of agent 1, the first evaluation network of agent 1, the second decision network of agent 2, and the second evaluation network of agent 2; Input of the first decision network is the velocity u, v, and w of the underwater glider along the X-axis, Y-axis, and Z-axis in the inertial coordinate system in time step t; the output of the first decision network is the buoyancy control quantity B of the underwater glider in time step t b and pitch adjustment control amount l p ; The input of the first evaluation network is (u,v,w,B b , l p ); the output of the first evaluation network is the first evaluation value The input of the second decision network is The output of the second decision network is the roll adjustment control quantity of the underwater glider in time step t The input of the second evaluation network is The output of the second evaluation network is the second evaluation value Among them, the parameters of the first decision network are Parameters of the second decision network The parameters of the first evaluation network are The parameters of the second evaluation network are 2. A multi-agent reinforcement learning model according to claim 1, characterized in that: The first decision network, the first evaluation network, the second decision network and the second evaluation network all adopt fully connected neural networks.

3. A training method for a multi-agent reinforcement learning model, characterized in that: include: The Markov chain dataset D is used as the input of the multi-agent reinforcement learning model of claim 1, and the multi-agent reinforcement learning model is trained N times to obtain the first decision network of agent 1 as the final first decision network. and the second decision network of agent 2 as the final second decision network The Markov chain dataset D consists of multiple Markov chain sample data, each of which includes: and Among them, the underwater glider randomly performs real gliding motion for the i-th time for The underwater glider randomly performs real gliding motion for the jth time for 4. A training method for a multi-agent reinforcement learning model, characterized in that: The specific steps of the training process using a set of Markov chain sample data as input for the multi-agent reinforcement learning model are as follows: Step 1: Convert the Markov chain sample data into and Input to the first evaluation network, the first evaluation network outputs the first evaluation value; according to the loss function of the first evaluation network, the first evaluation value and the reward value in the Markov chain sample data are used Calculate the loss function value of the first evaluation network and adjust the parameters of the first evaluation network Update and get parameters Then the updated first evaluation network Q′1 is obtained; The Markov chain sample data and Input to the second evaluation network, the second evaluation network outputs the second evaluation value; according to the loss function of the second evaluation network, the second evaluation value and the reward value in the Markov chain sample data are used Calculate the loss function value of the second evaluation network and adjust the parameters of the second evaluation network Update and get parameters Then the updated second evaluation network Q′2 is obtained; Step 2: Convert the Markov chain sample data Input to the first decision network, the first decision network output Will and As Input to the second decision network, the second decision network output Step 3: Convert the Markov chain sample data into and Input to the updated first evaluation network, the updated first evaluation network outputs the first evaluation value Will and Input to the updated second evaluation network of agent 2, the updated second evaluation network outputs the second evaluation value Step 4: The first evaluation value Substitute into the gradient calculation formula of the first decision network to calculate the gradient value of the first decision network Using gradient values Parameters of the first decision network Update and get parameters Then the updated first decision network μ′1 is obtained; The second evaluation value Substitute into the gradient calculation formula of the second decision network to calculate the gradient value of the second decision network Using gradient values Parameters of the second decision network Update and get parameters Then the updated second decision network μ′2 is obtained; Step 5: Convert the Markov chain sample data into Input to the updated first decision network, the output of the updated first decision network is Will and As Input to the updated second decision network, the output of the updated second decision network is Step 6, and the Markov chain sample data Substitute into the guided loss function of agent 1, calculate the guided loss function value, and update the parameters of the first decision network Update again to get the parameters Then we get the first decision network after the second update Will and the Markov chain sample data Substitute into the guided loss function of agent 2, calculate the guided loss function value, and update the parameters of the second decision network Update again to get the parameters Then we get the second decision network after the second update Among them, the guided loss function of agent 1 is The guided loss function of agent 2 is Represents the optimization of the parameters of the first decision network minimize and the differences between; Represents the optimization of the parameters of the second decision network minimize and The difference between.

5. A method for underwater glider motion control based on multi-agent reinforcement learning, characterized in that: include: Step 1: The final first decision network of agent 1 and the final second decision network of Agent 2 Deployed in the actual observation mission of the underwater glider, the optimal control quantity is calculated: In the process of the underwater glider actually performing the observation mission, before each section enters the water, the target running speed (u, v, w) in the observation mission is used as the final first decision network The input of the first decision network The output is the buoyancy adjustment control quantity B b and pitch adjustment control amount l p ; (u,v,w,B b , l p ) as the final second decision network The input of the second decision network Output roll adjustment control amount Will Input to the underwater glider, the underwater glider according to Complete the navigation of the current section; Step 2: Determine whether the underwater glider has completed the observation mission in the current profile: If the straight-line distance between the underwater glider's position and the observation mission target point is less than s, the underwater glider has completed the navigation mission and the motion control of the underwater glider is stopped; otherwise, execute step 3; Step 3: After completing the navigation of the current profile, calculate the ocean current interference speed f of the underwater glider in the current profile; Step 4: Compensate the speed of the underwater glider in the next profile along the X-axis and the axial direction according to the ocean current interference speed f calculated for the current profile. No speed compensation is performed for the first profile movement. Substitute the compensated speed (u′, v, w′) as the target running speed of the next profile into step 1.

6. The underwater glider motion control method based on multi-agent reinforcement learning according to claim 5 is characterized in that: The calculation formula of the ocean current interference speed f of the underwater glider in the current profile is: Wherein, x is the expected navigation displacement of the underwater glider along the X-axis in the section, x′ is the actual navigation displacement of the underwater glider along the X-axis in the section; z is the expected navigation displacement of the underwater glider along the Z-axis in the section, z′ is the actual navigation displacement of the underwater glider along the Z-axis in the section; T is the operating time of the underwater glider in the section.

7. The underwater glider motion control method based on multi-agent reinforcement learning according to claim 6 is characterized in that: The calculation formula for the target running speed in the X-axis direction and the Z-axis direction after compensation is: u′=u+f u w′=w+f w Among them, u′ is the speed of the current profile along the X-axis after compensation, u is the target running speed of the current profile along the X-axis, and f u is the component of the ocean current interference velocity f in the X-axis direction; w′ is the velocity of the current profile along the Z-axis after compensation, w is the target running speed of the current profile along the Z-axis direction, f w is the component of the ocean current interference velocity f in the Z-axis direction.

Images