A reinforcement learning control method for underwater robots based on spatiotemporal smoothness analysis

Abstract

This invention provides a reinforcement learning control method for underwater robots based on spatiotemporal smoothness analysis. By combining temporal and spatial smoothness analysis, it effectively reduces oscillations in motor signals. By introducing an integral compensator into the reinforcement learning neural network-based controller, it compensates for steady-state errors caused by value function fitting errors, thereby ensuring control stability and accuracy. Furthermore, during the control process, the invention dynamically fits the underwater robot's dynamic model and uses this dynamic model to obtain empirical data for reinforcement learning training. This method improves the learning efficiency of the reinforcement learning algorithm by expanding the training data.

Description

Technical Field

[0001] The present invention belongs to the field of underwater robot control, and in particular relates to an underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis. Background Art

[0002] Underwater robots can assist and even replace human divers in dangerous underwater operations. Currently, underwater robots are being used in marine resource research, marine accident investigations, and marine engineering. Due to the complex coupling relationships between underwater robot propulsion systems and the various complex interferences present in the underwater environment, controlling the robot's attitude and position presents significant challenges, making it difficult for traditional control technologies to achieve ideal control results. In recent years, machine learning techniques such as deep learning and reinforcement learning have enabled the control of complex controlled objects through data-based learning methods. Control techniques based on reinforcement learning have achieved certain results in simulation environments for underwater robot control.

[0003] The goal of underwater robot control methods based on reinforcement learning is to enable the underwater robot to reach the target position and attitude autonomously without human intervention. However, existing methods still suffer from problems such as control quantity oscillation and poor control accuracy. First, because the control actions of reinforcement learning are directly derived from the output of the neural network, the continuity and smoothness of the control quantity are not considered. Therefore, high-frequency oscillations in the control quantity may be output, which may cause excessive wear of the propeller in practical applications. In addition, due to the fitting error of the value function, underwater robot control methods based on reinforcement learning often have steady-state errors in position and attitude, resulting in poor control accuracy. In response to these two shortcomings of existing control schemes, it is urgent to develop an improved underwater robot control method to improve the smoothness of the control quantity and improve the control accuracy to meet the needs of practical underwater robot applications. Summary of the Invention

[0004] To address the issues of uneven motor drive signals in underwater robots, steady-state errors in control, excessive wear and tear on the robot's propellers, and inability to accurately achieve position and attitude control goals, the present invention proposes a reinforcement learning control method for underwater robots based on spatiotemporal smoothness analysis. By combining temporal and spatial smoothness analysis, the oscillation phenomenon in the motor signal is effectively weakened. By introducing an integral compensator into the reinforcement learning neural network-based controller, the steady-state error caused by the value function fitting error is compensated, thereby ensuring control stability and accuracy. Furthermore, during the control process, the present invention dynamically fits the underwater robot's dynamic model and obtains empirical data for reinforcement learning training through the dynamic model. By expanding the training data, the learning efficiency of the reinforcement learning algorithm is improved.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A reinforcement learning control method for an underwater robot based on spatiotemporal smoothness analysis comprises the following steps:

[0007] Step S1: construct a simulation environment for the underwater robot, where the robot's six-degree-of-freedom motion model is expressed as:

[0008]

[0009] where η = [x, y, z, φ, θ, ψ] T ,x,y,z represent the position coordinates of the underwater robot, φ,θ,ψ represent the steering angle of the underwater robot, is its corresponding first-order derivative, v represents the speed and turning rate of the underwater robot in the robot body coordinate system, is its corresponding first-order derivative, J(η) is the coordinate transformation matrix from the inertial coordinate system to the robot body coordinate system, and is a function of η; M is the inertia matrix of the underwater robot; the C(v) matrix describes the influence of the centripetal force and Coriolis force in the movement of the underwater robot, and is a function of v; the D(v) matrix is ​​the damping matrix, which represents the damping characteristics of water on the underwater robot, and is also a function of v; g(η) represents the restoring force matrix of the underwater robot, and is a function of η; τ b represents the propulsion force of the underwater robot thruster;

[0010] Step S2: construct the reward function and initialize the evaluator and controller of the underwater robot reinforcement learning algorithm, as well as the model network for fitting dynamics;

[0011] Step S3, performing multiple rounds of position control experiments in the simulation environment, saving relevant experimental data for subsequent evaluator and controller training, using the experimental data to train the dynamic model network, and using the dynamic model network to expand the data set;

[0012] Step S4, using the data in step S3 to train the evaluator and controller of the underwater robot, and updating the parameters in the network through the policy gradient algorithm;

[0013] Step S5, performing a performance test on the controller obtained in step S4 to determine whether it meets the task indicators;

[0014] Step S6: Use the controller to control the position of the underwater robot in a real environment.

[0015] As a further improvement of the present invention, the specific steps of step S2 are as follows:

[0016] Step S2-1, construct a reward function, which is set as the current position [x, y, z] of the underwater robot at the target position and the target position [xd ,y d ,z d ], expressed as:

[0017]

[0018] Step S2-2: construct the controller of the underwater robot, which is represented by a neural network. The input of the network is the current system state. where x e ,y e ,z e Represent the distance between the current position and the target position respectively. The output of the network is the propulsion control amount a=τ of the underwater robot propeller. b ,The overall controller network is expressed as: a = π(s|μ), where μ represents the weight parameter in the network, and the initial value of μ is a random value;

[0019] Step S2-3, construct the underwater robot evaluator, which is represented by a neural network. The evaluator network contains two neural networks with exactly the same structure, namely the main evaluator network and the target evaluator network. The inputs of the two networks are the current system state and the control quantity. The output is expressed as state-action value. The main evaluator network is expressed as Q(s,a|w1), where w1 represents the weight parameter of the network and the initial value of w1 is a random value. The target evaluator network is expressed as Q(s,a|w2), where w2 represents the weight parameter of the network and the initial value of w2 is equal to w1.

[0020] Step S2-4: construct a model for fitting the underwater robot dynamics, which is represented by a neural network. The input of the network is the current system state and the control quantity. The output is expressed as the system state and the corresponding reward value at the next moment. The model network is expressed as F(s,a|b), where b represents the weight parameter of the network and the initial value of b is a random value.

[0021] As a further improvement of the present invention, the specific steps of step S3 are as follows:

[0022] Step S3-1, randomly initialize the target position of the underwater robot in the simulation environment [x d ,y d ,z d ];

[0023] Step S3-2: input the current system state s to the controller of the underwater robot to obtain the propulsion force control value a of the propeller;

[0024] Step S3-3, using the simulation environment to perform dynamic calculations on the underwater robot, inputting the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment;

[0025] Step S3-4: Save the current system state s and control value a, as well as the next system state s' and reward function value r into the simulation training database;

[0026] Step S3-5: When the simulation time reaches 5 seconds, the current round of experiments ends. After a certain number of experiments, the next step S4 is entered.

[0027] Step S3-6, using the data obtained from the simulation to update the model network of the underwater robot, and using the gradient descent method to update the parameter b of F(s,a|b) according to the existing system state s' at the next moment;

[0028] Step S3-7, randomly initialize the current system state s, input the system state s and the robot control quantity a into the underwater robot dynamics model network to obtain the next moment system state s' and reward function value r, and save [s, a, s', r] to the simulation training database.

[0029] As a further improvement of the present invention, the specific steps of step S4 are as follows:

[0030] Step S4-1, randomly extracting some data from the simulation training database;

[0031] Step S4-2, using the sampled data to update the main evaluator network parameters of the underwater robot, and updating the parameter w1 of the main evaluator network Q(s,a|w1) according to the policy gradient method;

[0032] Step S4-3, update the target evaluator network parameters of the underwater robot, and update the parameter w2 of the target evaluator network Q(s,a|w2) according to the following formula:

[0033] w2←βw i +(1-β)w2

[0034] Where β represents the update rate of the parameters;

[0035] Step S4-4, using the sampled data to update the controller network parameters of the underwater robot, adding an optimization strategy for non-smooth motor drive signals, and introducing the motor utilization rate control parameter P in the controller network update process. U , and the motor drive signal spatial smoothness parameter P S and the temporal smoothness parameter P T , where P U Used to prevent the motor usage from exceeding the limit, PS and P T Used to smooth the motor drive signal, calculated using the following formula:

[0036]

[0037] P T =D T (π(s|μ),π(s'|μ))

[0038] Among them D S ,D T represents the root mean square error function, Represents the system state after noise is applied, and the parameter μ of the controller network π(s|μ) is updated according to the motor driving force optimization strategy and the policy gradient algorithm.

[0039] As a further improvement to the present invention, step S5 involves setting a random target position in a simulation environment, using the trained controller to control the underwater robot's position, and calculating the sum of the reward function values ​​during this control process. If the sum of the average reward function values ​​of the controller during the task, obtained after multiple rounds of training and updates, exceeds a threshold, the simulation training is considered complete and the task can be performed in a real environment, and step S6 is performed. Otherwise, the process returns to step S3 to continue data collection and training.

[0040] As a further improvement of the present invention, the specific steps of step S6 are as follows:

[0041] S6-1, randomly initialize the target position of the underwater robot in the real environment [x d ,y d ,z d ];

[0042] S6-2, add an integral compensator to eliminate the steady-state error in the position control of the underwater robot, and calculate the error term [x e ,y e ,z e ] are integrated to obtain the compensator output [x I ,y I ,z I ], the system state quantity after integration becomes An integral limiting algorithm is added to the integral compensator to prevent integral saturation. When the integral accumulation of the system is too large, the integral operation is stopped.

[0043] S6-3, the integrated system state quantity s I Input to the controller of the underwater robot to obtain the propeller propulsion control value a;

[0044] S6-4, use the simulation environment to perform dynamic calculations on the underwater robot, input the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment;

[0045] S6-5, repeatedly execute steps S6-1 to S6-5 until the task requirements are completed.

[0046] This invention employs a reinforcement learning algorithm based on spatiotemporal smoothness analysis to construct an underwater robot controller. Using a deep neural network, the robot's motion state is mapped to the control variables of its propellers, ultimately resulting in a controller that meets control requirements. An integral compensator is incorporated into the controller to eliminate steady-state errors, and an integral limiting algorithm is incorporated to prevent integral windup. Using spatiotemporal smoothness analysis, the invention smoothes the output motor drive signal, ensuring both spatial and temporal smoothness of the motor drive signal while preventing propeller overdrive.

[0047] Compared with the closest existing technology, the technical solution provided by the present invention has the following beneficial effects:

[0048] 1) The present invention incorporates an optimization strategy for non-smooth motor drive signals to limit and weaken irregular and oscillating motor drive signals, making them conform to real-world requirements and ensuring stability during the underwater robot's position control process.

[0049] 2) In the present invention, an integral compensator is added to the reinforcement learning framework to eliminate the steady-state error in the underwater robot position control task, ensuring the accuracy of the control and enabling the underwater robot to reach the specified position without error;

[0050] 3) The present invention incorporates a model network that fits the dynamics of the underwater robot, interacts with the simulated environment based on the dynamic model, obtains a large amount of additional data that can be used for reinforcement learning training, and accelerates the operating efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the process of the present invention. Specific embodiments

[0052] The following is combined with Figure 1 The present invention is further described in detail with specific implementation methods. This patent provides an underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis.

[0053] Step S1: construct a simulation environment for the underwater robot, where the robot's six-degree-of-freedom motion model is expressed as:

[0054]

[0055] where η = [x, y, z, φ, θ, ψ] T ,x,y,z represent the position coordinates of the underwater robot, φ,θ,ψ represent the steering angle of the underwater robot, is its corresponding first-order derivative, v represents the speed and turning rate of the underwater robot in the robot body coordinate system, Its corresponding first-order derivative. J(η) is the coordinate transformation matrix from the inertial coordinate system to the robot body coordinate system, and is a function of η; M is the inertia matrix of the underwater robot; the C(v) matrix describes the influence of the centripetal force and Coriolis force in the movement of the underwater robot, and is a function of v; the D(v) matrix is ​​the damping matrix, which represents the damping characteristics of water on the underwater robot, and is also a function of v; g(η) represents the restoring force matrix of the underwater robot, and is a function of η; τ b represents the propulsion force of the underwater robot thruster;

[0056] Step S2: construct the reward function, design the underwater robot's evaluator and controller, and the model network for fitting dynamics;

[0057] Step S3, performing multiple rounds of position control experiments in the simulation environment, saving relevant experimental data for subsequent evaluator and controller training, using the experimental data to train the dynamic model network, and using the dynamic model network to expand the data set;

[0058] Step S4, using the data in step S3 to train the evaluator and controller of the underwater robot, and updating the parameters in the network through the policy gradient algorithm;

[0059] Step S5, testing the controller obtained in step S4 to determine whether it meets the task indicators.

[0060] Step S6: Use the controller to control the position of the underwater robot in a real environment.

[0061] The specific steps of step S2 are as follows:

[0062] Step S2-1, construct a reward function, which is set as the current position of the underwater robot and the target position (x d ,y d ,z d ), expressed as:

[0063]

[0064] Step S2-2, construct the controller of the underwater robot, which is represented by a neural network. The network consists of an input layer, two hidden layers and an output layer. The input of the network is the current system state. where xe ,y e ,z e Represent the distance between the current position and the target position respectively. The first hidden layer includes 400 hidden nodes, and the second hidden layer includes 400 hidden nodes. The activation function between the first two layers uses the tanh function. The output of the network is the propulsion control amount a = τ of the underwater robot propeller. b , the activation function is the sigmoid function, and the overall controller network is expressed as: a=π(s|μ), where μ represents the weight parameter in the network and the initial value of μ is a random value.

[0065] Step S2-3, construct the underwater robot evaluator, which is represented by a neural network. The evaluator network contains two neural networks with exactly the same structure, namely the main evaluator network and the target evaluator network. The structure of the two networks includes an input layer, two hidden layers and an output layer. The network input is the current system state and the control quantity. The first hidden layer includes 200 hidden nodes, and the second hidden layer includes 200 hidden nodes. The activation function between the first two layers adopts the tanh function. The network output is expressed as the state-action value. The main evaluator network is expressed as Q(s,a|w1), where w1 represents the weight parameter of the network, and the initial value of w1 is a random value; the target evaluator network is expressed as Q(s,a|w2), where w2 represents the weight parameter of the network, and the initial value of w2 is equal to w1.

[0066] Step S2-4, construct a model network for fitting the underwater robot dynamics, which is represented by a neural network. The input of the network is the current system state and the control quantity. The output is expressed as the system state and the corresponding reward value at the next moment. The model network is expressed as F(s,a|b), where b represents the weight parameter of the network and the initial value of b is a random value.

[0067] The specific steps of step S3 are as follows:

[0068] Step S3-1, randomly initialize the target position of the underwater robot in the simulation environment [x d ,y d ,z d ];

[0069] Step S3-2: input the current system state s into the controller of the underwater robot to obtain the propeller propulsion force control value a;

[0070] Step S3-3, using the simulation environment to perform dynamic calculations on the underwater robot, inputting the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment;

[0071] Step S3-4, save the system state s and control quantity a at the current moment, as well as the system state s' and reward function value r at the next moment into the simulation training database.

[0072] Step S3-5: When the simulation time reaches 5 seconds, the current round of experiments ends. After a certain number of experiments, the next step S4 is entered.

[0073] Step S3-6, using the data obtained from the simulation to update the model network of the underwater robot, and using the gradient descent method to update the parameter b of F(s,a|b) according to the existing system state s' at the next moment.

[0074] Step S3-7, randomly initialize the current system state s, input the system state s and the robot control quantity a into the underwater robot dynamics model network to obtain the next moment system state s' and reward function value r, and save [s, a, s', r] to the simulation training database.

[0075] The specific steps of step S4 are as follows:

[0076] Step S4-1, randomly extracting some data from the simulation training database;

[0077] Step S4-2, using the sampled data to update the main evaluator network parameters of the underwater robot, and updating the parameter w1 of the main evaluator network Q(s,a|w1) according to the policy gradient method;

[0078] Step S4-3, update the target evaluator network parameters of the underwater robot, and update the parameter w2 of the target evaluator network Q(s,a|w2) according to the following formula:

[0079] w2←βw1+(1-β)w2

[0080] Where β represents the update rate of the parameters;

[0081] Step S4-4, using the sampled data to update the controller network parameters of the underwater robot, adding an optimization strategy for the non-smooth motor drive signal, and introducing the motor utilization rate control parameter P in the controller network update process. U , and the motor drive signal spatial smoothness parameter P S and the temporal smoothness parameter P T , where P U Used to prevent the motor usage from exceeding the limit, P S and P T Used to smooth the motor drive signal, calculated using the following formula:

[0082]

[0083] PT =D T (π(s|μ),π(s'|μ))

[0084] Among them D S ,D T represents the root mean square error function, Represents the system state after noise is applied, and the parameter μ of the controller network π(s|μ) is updated according to the motor driving force optimization strategy and the policy gradient algorithm.

[0085] The specific method of step S5 is to set a random target position in the simulation environment, use the trained controller to control the underwater robot's position, and calculate the sum of the reward function values ​​during this control process. If the sum of the average reward function values ​​of the controller in the task, obtained after multiple rounds of training and updates, exceeds a threshold, it is determined that the simulation training has been completed and the task can be performed in the real environment, and step S6 is performed; otherwise, return to step S3 to continue collecting data and training.

[0086] The specific steps of step S6 are as follows:

[0087] S6-1, randomly initialize the target position of the underwater robot in the real environment [x d ,y d ,z d ];

[0088] S6-2, add an integral compensator to eliminate the steady-state error in the position control of the underwater robot, and calculate the error term (x e ,y e ,z e ) are integrated to obtain the compensator output (x I ,y I ,z I ), the system state quantity after integration becomes An integral limiting algorithm is added to the integral compensator to prevent integral saturation. When the integral accumulation of the system is too large, the integral operation is stopped.

[0089] S6-3, the integrated system state quantity s I Input to the controller of the underwater robot to obtain the propeller propulsion control value a;

[0090] S6-4, use the simulation environment to perform dynamic calculations on the underwater robot, input the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment;

[0091] S6-5, repeatedly execute steps S6-1 to S6-5 until the task requirements are completed.

[0092] The above description is merely a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent variation based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A reinforcement learning control method for underwater robots based on spatiotemporal smoothness analysis, characterized by: The steps include: Step S1: construct a simulation environment for the underwater robot, where the robot's six-degree-of-freedom motion model is expressed as: ； in =[x,y,z, , , ] T ,x,y,z represent the position coordinates of the underwater robot, , , represents the steering angle of the underwater robot, is its corresponding first-order derivative, v represents the speed and turning rate of the underwater robot in the robot body coordinate system, Its corresponding first-order derivative, J( ) is the coordinate transformation matrix from the inertial coordinate system to the robot body coordinate system, and is function; M is the inertia matrix of the underwater robot; C(v) matrix describes the influence of centripetal force and Coriolis force in the motion of the underwater robot and is a function of v; D(v) matrix is ​​the damping matrix, which represents the damping characteristics of water on the underwater robot and is also a function of v; g( ) represents the restoring force matrix of the underwater robot, and is function; represents the propulsion force of the underwater robot thruster; Step S2: construct the reward function and initialize the evaluator and controller of the underwater robot reinforcement learning algorithm, as well as the model network for fitting dynamics; Step S3, performing multiple rounds of position control experiments in the simulation environment, saving relevant experimental data for subsequent evaluator and controller training, using the experimental data to train the dynamic model network, and using the dynamic model network to expand the data set; Step S4, using the data in step S3 to train the evaluator and controller of the underwater robot, and updating the parameters in the network through the policy gradient algorithm; The specific steps of step S4 are as follows: Step S4-1, randomly extracting some data from the simulation training database; Step S4-2: Update the main evaluator network parameters of the underwater robot using the sampled data. Update the parameter w1 of the main evaluator network Q(s,a|w1) according to the policy gradient method, where a is the propulsion control variable and s is the current system state variable. Step S4-3, update the target evaluator network parameters of the underwater robot, and update the parameter w2 of the target evaluator network Q(s,a|w2) according to the following formula: ； in Indicates the update rate of the parameters; Step S4-4, using the sampled data to update the controller network parameters of the underwater robot, adding an optimization strategy for non-smooth motor drive signals, and introducing the motor utilization rate control parameter P in the controller network update process. U , and the motor drive signal spatial smoothness parameter P S and the temporal smoothness parameter P T , where P U Used to prevent the motor usage from exceeding the limit, P S and P T Used to smooth the motor drive signal, calculated using the following formula: P S =D S ( (s| ), ( | )); P T =D T ( (s| ), (s’| )); Among them D S ,D T represents the root mean square error function, represents the system state after noise is applied, Represents the weight parameters in the network, and the controller network is optimized according to the motor driving force optimization strategy and the policy gradient algorithm. (s| ) Make updates; Step S5, performing a performance test on the controller obtained in step S4 to determine whether it meets the task indicators; Step S6: Use the controller to control the position of the underwater robot in a real environment.

2. The underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis according to claim 1 is characterized by: The specific steps of step S2 are as follows: Step S2-1, construct a reward function, which is set as the current position [x, y, z] of the underwater robot at the target position and the target position [x d ,y d ,z d ], expressed as: ； Step S2-2, construct the controller of the underwater robot, which is represented by a neural network. The input of the network is the current system state s=[ , ,x e ,y e ,z e ], where x e ,y e ,z e Represent the distance between the current position and the target position respectively. The output of the network is the propulsion control amount a of the underwater robot propeller. , the controller network as a whole is expressed as: a= (s| ),in represents the weight parameters in the network, The initial value is a random value; Step S2-3, construct the underwater robot evaluator, which is represented by a neural network. The evaluator network contains two neural networks with exactly the same structure, namely the main evaluator network and the target evaluator network. The inputs of the two networks are the current system state and the control quantity (s, a) = [ , ,x e ,y e ,z e , ], the output is expressed as state-action value, the main evaluator network is expressed as Q(s,a|w1), where w1 represents the weight parameter of the network, and the initial value of w1 is a random value; the target evaluator network is expressed as Q(s,a|w2), where w2 represents the weight parameter of the network, and the initial value of w2 is equal to w1; Step S2-4, construct a model for fitting the underwater robot dynamics, which is represented by a neural network. The input of the network is the current system state and the control quantity (s, a) = [ , ,x e ,y e ,z e , ], the output is expressed as the system state and the corresponding reward value at the next moment, and the model network is expressed as F(s,a|b), where b represents the weight parameter of the network and the initial value of b is a random value.

3. The underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis according to claim 1 is characterized by: The specific steps of step S3 are as follows: Step S3-1, randomly initialize the target position of the underwater robot in the simulation environment [x d ,y d ,z d ]; Step S3-2: input the current system state s to the controller of the underwater robot to obtain the propulsion force control value a of the propeller; Step S3-3, using the simulation environment to perform dynamic calculations on the underwater robot, inputting the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment; Step S3-4: Save the current system state s and control value a, as well as the next system state s' and reward function value r into the simulation training database; Step S3-5: When the simulation time reaches 5 seconds, the current round of experiments ends. After a certain number of experiments, the next step S4 is entered. Step S3-6, using the data obtained from the simulation to update the model network of the underwater robot, and using the gradient descent method to update the parameter b of F(s,a|b) according to the existing system state s' at the next moment; Step S3-7, randomly initialize the current system state s, input the system state s and the robot control quantity a into the underwater robot dynamics model network to obtain the next moment system state s' and reward function value r, and save [s, a, s', r] to the simulation training database.

4. The underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis according to claim 1 is characterized in that: The specific method of step S5 is to set a random target position in the simulation environment, use the trained controller to control the position of the underwater robot, and count the total reward function value during the control process. If the total average reward function value of the controller in the task obtained after multiple rounds of training is greater than the threshold, it is judged that the simulation training has been completed and it can perform the task in the real environment, and step S6 is performed; otherwise, return to step S3 to continue collecting data and training.

5. The underwater robot reinforcement learning control method based on spatiotemporal smoothness analysis according to claim 1 is characterized by: The specific steps of step S6 are as follows: S6-1, randomly initialize the target position of the underwater robot in the real environment [x d ,y d ,z d ]; S6-2, add an integral compensator to eliminate the steady-state error in the position control of the underwater robot, and calculate the error term [x e ,y e ,z e ] are integrated to obtain the compensator output [x I ,y I ,z I ], the system state quantity after integration becomes s I =[ , ,x I ,y I ,z I ], add an integral limiting algorithm to the integral compensator to prevent integral saturation, and stop the integral operation when the integral accumulation of the system is too large; S6-3, the integrated system state quantity s I Input to the controller of the underwater robot to obtain the propeller propulsion control value a; S6-4, use the simulation environment to perform dynamic calculations on the underwater robot, input the robot control variable a into the environment to obtain the system state variable s' and the reward function value r at the next moment; S6-5, repeatedly execute steps S6-1 to S6-5 until the task requirements are completed.

Images