A control method of a gyro stabilizer based on deep reinforcement learning

Abstract

An effective control method, the applicable loss function is constructed and the reward function, training process is optimized and designed, so that the neural network can converge, the robustness and control effect of the control method are improved. Meanwhile, the precession angle of the gyro, the precession angular velocity, the ship roll angle, the roll angular velocity, the last control period of the wave excitation estimation value, five parameters are used as the input quantity of the actor network to improve the system control effect. The control method makes the gyro stabilizer, the gyro stabilizer using the control method has better stabilizing effect, especially when the wave excitation is large, the stabilizing effect can still maintain a good level. The control system using the control method has strong robustness.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control of ship roll reduction equipment, and specifically relates to a control method for a ship gyroscope roll reduction device based on deep reinforcement learning. Background Technology

[0002] Currently, active ship gyro-based roll reduction devices mainly employ traditional control methods, such as composite PID and sliding mode control. However, research on the joint mathematical model of the gyro-based roll reduction device, the ship, and the waves reveals that the model is a nonlinear and complex system. Furthermore, because the excitation of the waves is stationary and random, the system exhibits strong randomness. Consequently, the effectiveness of the gyro-based roll reduction device becomes less than ideal when the ship encounters relatively large waves.

[0003] For example, a composite PID controller calculates the control quantity of the gyroscope using a formula composed of proportional coefficient, derivative coefficient, ship's roll attitude, and gyroscope's precession attitude. However, due to the large randomness of ocean waves, which vary in size, the controller's parameters are often set conservatively to ensure that the gyroscope has a roll reduction effect in both large and small waves. This results in a poor roll reduction effect when the waves are large.

[0004] Qingdao University of Science and Technology has disclosed a method for reducing roll using a ship's torque gyroscope (CN103818524A, publication date: May 28, 2014). The gyroscope is supported on an inner frame by two magnetic bearings. The gyroscope's precession rate directly reflects the ship's roll condition. After a sensor detects the precession rate of the inner frame, it is fed back to the controller in real time. The controller controls the operation of the damper and servo motor. By adjusting the damping force output by the damper or adjusting the speed of the servo motor, the precession rate is adjusted to achieve the required gyroscope torque, thereby reducing roll. As mentioned above, the gyroscope precession rate is set relatively conservatively, resulting in poor roll reduction even in large waves. Summary of the Invention

[0005] To address the problem that existing active gyroscope anti-roll devices have unsatisfactory control effects, with a significant decrease in control performance under large waves, resulting in poor anti-roll performance, a joint mathematical model of the gyroscope anti-roll device, the ship, and the waves, along with a deep reinforcement learning method, is proposed. Based on deep reinforcement learning technology, a control method that can be effectively used for gyroscope anti-roll devices is invented.

[0006] The technical solution of the present invention is as follows:

[0007] A control method for a gyroscope roll reduction device based on deep reinforcement learning, characterized in that the method includes the following steps:

[0008] Step 1: Analyze the motion characteristics of the system when the gyroscope anti-roll device is working, build a virtual simulation environment of the gyroscope, ship and waves in the computer, and interact with the intelligent agent during the method training.

[0009] Step 2: Design a reward function to evaluate the quality of the decisions made by the agent.

[0010] Step 3: Design the agent structure. The agent contains two Actor networks and two Critic networks. Design the structure of four neural networks, set all the parameters that need to be set by the method, such as learning rate, pruning rate, decay rate, discount factor, and initialize the four neural networks.

[0011] Step 4: Allow the intelligent agent to continuously interact with the virtual simulation environment to obtain a series of data.

[0012] Step 5: The reward function in Step 2 scores the agent's behavior based on the data obtained in Step 4. Based on this score and the agent's Critic network, the advantage function value of each interaction between the agent and the environment is calculated.

[0013] Step 6: Based on the agent behavior scores and advantage function values ​​obtained in Step 5, construct the appropriate loss function values ​​for the Actor and Critic networks, respectively. Then, based on the gradient optimization approach, optimize the parameters θ of the Actor network and the Critic network using the loss function values.

[0014] Step 7: Repeat steps 4-6 until the parameters θ of the Actor network in the agent match the parameter values ​​of the Critic network. Until they all converge.

[0015] Step 8: Deploy the Actor network from the converged intelligent agent into the control system of the gyroscope anti-roll device to achieve control of the gyroscope anti-roll device.

[0016] Specifically, the motion state of the gyroscope anti-roll device installed on the ship and encountering waves is analyzed as described in step 1. To reduce the computational load of the simulation process and shorten the training time of the method while ensuring system accuracy, the mathematical model is simplified, and the final joint dynamic model of the three components is established as follows:

[0017]

[0018] Where T is the ship's displacement, h is the ship's transverse metacentric height, and I... s Let N be the rotational moment of inertia of the ship along the roll axis, α(t) be the wave tilt angle function excited by the waves, and N be the moment of inertia of the ship along the roll axis. uLet φ be the water surface damping coefficient, h0 be the angular momentum of the gyroscope rotor, J be the moment of inertia of the gyroscope rotor along the precession axis, φ be the ship's roll angle, β be the gyroscope precession angle, and u be the gyroscope precession control torque, which is the control variable. The development program uses the current system state φ as the basis. k , β k , and control quantity u k As input, the next state φ of the system k+1 , β k+1 , As output.

[0019] The reward function is designed according to step 2, using the ship's roll angle φ and the gyroscope's precession angle β as evaluation indicators. The reward function employs a piecewise function form to improve the method's control effect while ensuring convergence. The reward function is as follows:

[0020]

[0021]

[0022] r = r φ +r β

[0023] Where φ1, φ2, β1, β2, and β3 are the boundary values ​​of roll angle and precession angle selected according to the specifications of the ship and the gyro-based roll damping device, requiring 0 < φ1 < φ2 < 0.55 rad.

[0024] As described in step 3, the intelligent agent comprises four fully connected linear neural networks: Actor, Actor′, Critic, and Critic′. Their corresponding parameter sets (including weights and biases) are θ, θ′, and θ′, respectively. In this system, Actor and Critic are the actual neural networks to be updated, while Actor′ and Critic′ are the neural networks that actually interact with the environment. The parameter values ​​of Actor′ and Critic′ are copied from Actor and Critic once after each training iteration. The Actor neural network is a linear, fully connected, multi-layer neural network, with five input parameters for the input layer, including the ship's roll angle φ. k Ship roll rate gyroscope precession angle β k gyroscope precession angular velocity The estimated value α of the wave excitation in the previous control cycle k-1 ,remember For s kThe intermediate layer consists of several linear neural network layers. The output layer outputs two values: the mean μ and standard deviation σ of a normal distribution. The activation functions are tanh and softplus, respectively. The value output based on the probability of constructing a normal distribution using the mean and standard deviation is the control torque u of the precession axis of the gyroscope anti-roll device.

[0025]

[0026] The input parameters of the Critic network's input layer are the same as those of the Actor network. The intermediate layers are several linear neural networks, and the output layer outputs the current state s. k The estimated value of the function.

[0027] The process of interaction between the agent and the environment as described in step 4 involves initializing the environment and obtaining its initial state. Initial α -1 =0, 5 parameters are input into the Actor' network, and a normal distribution is constructed based on the output value to randomly output the gyroscope precession control torque u0 according to probability. Then, it is input into the environment and a new set of states is returned. Then, the current interaction result is scored according to the reward function described in claim 3 to obtain r0, based on... And u0 estimate to get α0, then Let s1 be the input Actor′, and u1 be obtained. The loop operation will yield a data sequence (this process is called one round of interaction): (s0, u0, r0, s1), (s1, u1, r1, s2)…(s…). n ,u n ,r n ,s n+1 ), where n is the number of interactions in each round, and the number of interactions in each round depends on the set value n or is terminated early when the ship's roll angle φ or gyro precession angle β exceeds the range. Each time the environment is initialized, a different random wave stimulus will be reconstructed to prevent overfitting of the neural network and improve the robustness of the controller. Meanwhile, to ensure accurate calculation of the advantage function value and the Actor network loss function, n > 2000; a larger value is better, but a larger value will require a longer training time.

[0028] According to step 5, based on the data sequence obtained in step 4: (s0,u0,r0,s1), (s1,u1,r1,s2)...(s n ,u n ,r n ,s n+1 Calculate the dominance function value A for each interaction with the environment. k :

[0029]

[0030] A k =adv k +γλA k+1 =adv k +γλ(adv k+1 +γλ(adv k+2 +γλ(adv k+3 +…)))

[0031] Where γ is the discount factor and λ is the attenuation rate. Indicates that s k Substitute The estimated value of the value function obtained from network computation.

[0032] As described in step 6, the obtained (A0, A1, A2, ..., A n The loss function used to construct the Actor network:

[0033]

[0034] Note: u k Indicates based on the current s k The action performed, i.e., the output precession shaft control torque, is therefore the same as u. k .

[0035] Where n≥2000, ε is the clipping rate of the clipping function. The meaning of the loss function is that when the state of the ship and the gyro-based roll reduction device is within s... k Execute action a under the following circumstances k That is, the control torque of the gyroscope's precession axis is u k The dominant function value A at that time k The expected value of the advantage function. Our goal is to maximize the expected value of the advantage function, therefore, based on the idea of ​​gradient ascent, we update θ using the following formula:

[0036]

[0037] μ a The learning rate of the Actor network

[0038] To ensure the accuracy of the advantage function calculation, the Critic network needs to be continuously optimized, and the loss function of the Critic network needs to be constructed:

[0039]

[0040] When the loss function value is minimized, the estimated value of the value function output by the Critic network is closer to the true value of the value function. At this point, the advantage function value A... k The more accurate the calculation, the more effective the gradient optimization of the Actor network parameters θ is guaranteed. Therefore, based on the idea of ​​gradient descent, the parameters of the Critic network... The update is as follows:

[0041]

[0042] μ c The learning rate of the Critic network

[0043] Since the loss function of the Actor network applies the importance-based approach, the data sequence obtained in the first round of interaction described in step 4 is: (s0,u0,r0,s1), (s1,u1,r1,s2)...(s n ,u n ,r n ,s n+1 This process can be used multiple times to optimize the parameters θ of the Actor network, improving training efficiency and method stability; that is, step 6 is repeated multiple times. One round of the process described in step 4, plus multiple rounds of the processes described in steps 5 and 6, is followed by copying all the parameters of the Actor network and Critic network to the Actor' network and Critic' network. The above one round of interaction, multiple optimization processes, and one copying process constitute one round of training.

[0044] 7. Following step 7, repeat steps 4-6, i.e., perform multiple rounds of training until the parameters θ of the Actor network and the parameters of the Critic network in the agent are consistent. Continue until convergence is achieved. During the repetition of steps 4-6, the total score R = r1 + r2 + ... + r2 obtained from each round of agent-environment interaction should be used as the basis. n The value of μ is used to reduce the learning rate μ of the neural network. a and μ c To prevent overfitting.

[0045] 8. Following step 8, deploy the Actor neural network obtained in step 7 into the control system of the gyro-based roll reduction device. The Actor's input layer has 5 parameters, and the gyro-based roll reduction device detects the ship's roll angle φ using an electronic gyroscope. k Ship roll rate Using an encoder to detect the gyroscope precession angle β k The precession angular velocity of the gyroscope is calculated based on the sampling frequency. Using the state φ from the previous control cycle k-1 , β k-1 , The precession control torque u of the previous control cycle k-1 And the estimated value α′ of the wave excitation in the previous control cycle is obtained from the current state estimation. k-1 ,Will Five parameters are used as input parameters to the Actor network. After forward propagation of the Actor network, two parameters are output as the mean μ and standard deviation σ of a normal distribution. Then, a normal distribution is constructed, and the precession control torque u of the gyroscope roll reduction device is obtained by randomly selecting values ​​from this normal distribution. k Then, the control system controls the precession servo motor of the gyroscope anti-roll device to output a value of u in torque output mode. k The torque acts on the precession shaft, completing the control of the current control cycle. Repeating this process continuously constitutes the control process of the gyro anti-roll device.

[0046] The present invention relates to a control method for a ship gyroscope anti-roll device based on deep reinforcement learning. The beneficial effects of adopting the above technical solution are as follows:

[0047] (1) The gyroscope anti-roll device using this control method has a better anti-roll effect, especially when the sea wave excitation is large, the anti-roll effect can still maintain a good level.

[0048] (2) The control system using this control method has strong robustness. Attached Figure Description

[0049] Figure 1 This is a general block diagram of the control method of the present invention;

[0050] Figure 2 This is a flowchart illustrating the deep reinforcement learning process of the control method of the present invention.

[0051] Figure 3 This is a block diagram of the anti-sway control method of the present invention;

[0052] Figure 4 This is a graph of the reward function of the deep reinforcement learning in the control method of the present invention;

[0053] Figure 5 This is a diagram showing the training results of the deep reinforcement learning method for the control method of the present invention;

[0054] Figure 6 This is a diagram illustrating the anti-roll effect of the gyroscope anti-roll device based on deep reinforcement learning, which is the control method of this invention. Detailed Implementation

[0055] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The specific embodiments described below are for illustrative purposes only and are not intended to limit the scope of the invention.

[0056] Example 1

[0057] This invention provides a control method for a gyroscope roll reduction device based on deep reinforcement learning. The framework and specific flow of the method are described below. Figure 1 , Figure 2 The control flow of the method is shown in Figure 3 .

[0058] A control method for a gyroscope roll reduction device based on deep reinforcement learning, characterized by the following steps:

[0059] S1: Analyze the motion characteristics of the system when the gyroscope anti-roll device is working, build a virtual simulation environment of gyroscope-ship-wave in the computer, and interact with the intelligent agent during method training;

[0060] S2: Design a reward function to evaluate the quality of decisions made by the agent;

[0061] S3: Design the agent structure, which includes two Actor networks and two Critic networks. Design the structure of the four networks, set all the parameters that need to be set by the method, such as learning rate, pruning rate, decay rate, discount factor, and initialize the four networks.

[0062] S4: Enables the intelligent agent to continuously interact with the virtual simulation environment to obtain a series of data;

[0063] S5: The reward function in step 2 scores the agent's behavior based on the data obtained in S4, and calculates the advantage function value of each interaction between the agent and the environment based on the score and the agent's Critic network.

[0064] S6: Based on the agent behavior scores and advantage function values ​​obtained in S5, construct appropriate loss functions for the Actor network and Critic network, respectively. Then, based on the gradient optimization approach, optimize the parameters θ of the Actor network and the Critic network according to the loss function values.

[0065] S7: Repeat steps 4-6 until the parameters θ of the Actor network in the agent match the parameter values ​​of the Critic network. Until they converge;

[0066] S8: Deploy the Actor network in the converged intelligent agent into the control system of the gyroscope anti-roll device to realize the control of the gyroscope anti-roll device.

[0067] Specifically, step S1 includes the following: analyzing the motion state of the gyroscope anti-roll device when it is installed on the ship and encounters waves. To reduce the computational load of the simulation process and shorten the training time of the method while ensuring system accuracy, the mathematical model is simplified, and a joint dynamic model of the three components is finally established as shown in formula (1):

[0068]

[0069] Where T is the ship's displacement, h is the ship's transverse metacentric height, and I... s Let N be the rotational moment of inertia of the ship along the roll axis, α(t) be the wave tilt angle function excited by the waves, and N be the moment of inertia of the ship along the roll axis. u Let φ be the water surface damping coefficient, h0 be the angular momentum of the gyroscope rotor, J be the moment of inertia of the gyroscope rotor along the precession axis, φ be the ship's roll angle, β be the gyroscope precession angle, and u be the gyroscope precession control torque, which is the control variable. The development program uses the current system state φ as the basis. k , β k , and control quantity u k As input, the next state φ of the system k+1 , β k+1 , As output.

[0070] Step S2 specifically includes the following:

[0071] The design reward function uses the ship's roll angle φ and the gyroscope's precession angle β as evaluation indicators. The reward function employs a piecewise function form to improve the control effect of the method while ensuring convergence. The reward function is shown in formula (2):

[0072]

[0073]

[0074] r = r φ +r β

[0075] Where φ1, φ2, β1, β2, and β3 are the boundary values ​​of roll angle and precession angle selected according to the specifications of the ship and the gyro-based roll damping device, requiring 0 < φ1 < φ2 < 0.55 rad.

[0076] Step S3 specifically includes the following:

[0077] S3.1 Construct four networks.

[0078] The intelligent agent comprises four fully connected linear neural networks: Actor, Actor′, Critic, and Critic′. Their corresponding parameter sets (including weights and biases) are θ, θ′, and θ′, respectively. In this system, Actor and Critic are the actual neural networks to be updated, while Actor′ and Critic′ are the neural networks that actually interact with the environment. The parameter values ​​of Actor′ and Critic′ are copied from Actor and Critic once after each training round.

[0079] The Actor neural network structure is a linear fully connected multilayer neural network, and the five input parameters of the input layer are the ship's roll angle φ. k Ship roll rate gyroscope precession angle β k gyroscope precession angular velocity The estimated value α of the wave excitation in the previous control cycle k-1 ,remember For s k The intermediate layer has several layers of linear neural networks. The output layer outputs two values, which are the mean μ and standard deviation σ of the normal distribution. The activation functions are tanh and softplus, respectively. The value of the normal distribution output according to the probability is the control torque u of the precession axis of the gyroscope anti-roll device, as shown in formula (3).

[0080]

[0081] The input parameters of the Critic network's input layer are the same as those of the Actor network. The intermediate layers are several linear neural networks, and the output layer outputs the current state s. k The estimated value of the function.

[0082] Step S4 specifically includes: the process of the agent interacting with the environment is to initialize the environment and obtain the initial state of the environment. Initial α -1 =0, 5 parameters are input into the Actor' network, and a normal distribution is constructed based on the output value to randomly output the gyroscope precession control torque u0 according to probability. Then, it is input into the environment and a new set of states is returned. Then, the current interaction result is scored according to the reward function described in claim 3 to obtain r0, based on... And u0 estimate to get α0, then Let s1 be the input Actor′, and u1 be obtained. The loop operation will yield a data sequence (this process is called one round of interaction): (s0, u0, r0, s1), (s1, u1, r1, s2)…(s…). n ,u n ,r n ,s n+1), where n is the number of interactions in each round, and the number of interactions in each round depends on the set value n or is terminated early when the ship's roll angle φ or gyro precession angle β exceeds the range. Each time the environment is initialized, a different random wave stimulus will be reconstructed to prevent overfitting of the neural network and improve the robustness of the controller. Meanwhile, to ensure accurate calculation of the advantage function value and the Actor network loss function, n > 2000; a larger value is better, but a larger value will require a longer training time.

[0083] Step S5 specifically includes: the data sequence obtained from step S4: (s0,u0,r0,s1), (s1,u1,r1,s2)...(s n ,u n ,r n ,s n+1 Calculate the dominance function value A for each interaction with the environment. k For example, in formulas (5.1, 5.2):

[0084]

[0085] A k =adv k +γλA k+1 =adv k +γλ(adv k+1 +γλ(adv k+2 +γλ(adv k+3 +…))) (5.2)

[0086] Where γ is the discount factor and λ is the attenuation rate. Indicates that s k Substitute The estimated value of the value function obtained from network computation.

[0087] Step S6 specifically includes:

[0088] S6.1 Construct the loss function for the Actor network and update the θ parameters.

[0089] The obtained (A0, A1, A2, ..., A n The loss function used to construct the Actor network is shown in equation (6.11):

[0090]

[0091] Note: u k Indicates based on the current s k The action performed, i.e., the output precession shaft control torque, is therefore the same as u. k .

[0092] Where n≥2000, ε is the clipping rate of the clipping function. The meaning of the loss function is that when the state of the ship and the gyro-based roll reduction device is within s... k Execute action a under the following circumstances k That is, the control torque of the gyroscope's precession axis is u k The dominant function value A at that time k The expected value of the advantage function. Our goal is to maximize the expected value of the advantage function, therefore, based on the idea of ​​gradient ascent, we update the parameter θ using the following formula (6.12):

[0093]

[0094] μ a The learning rate of the Actor network

[0095] S6.2 Construct the loss function for the Critic network and update the parameters.

[0096] To ensure the accuracy of the advantage function calculation, the Critic network needs to be continuously optimized. The loss function of the Critic network is constructed as shown in formula (6.21):

[0097]

[0098] When the loss function value is minimized, the estimated value of the value function output by the Critic network is closer to the true value of the value function. At this point, the advantage function value A... k The more accurate the calculation, the more effective the gradient optimization of the Actor network parameters θ is guaranteed. Therefore, based on the idea of ​​gradient descent, the parameters of the Critic network are updated using formula (6.22).

[0099]

[0100] μ c The learning rate is the degree to which the neural network learns from the results of the previous interaction in each iteration. A smaller learning rate results in slower training, while a larger learning rate results in faster training but is less stable.

[0101] Since the loss function of the Actor network applies the importance-based approach, the data sequence obtained in the first round of interaction described in step 4 is: (s0,u0,r0,s1), (s1,u1,r1,s2)...(s n ,u n ,r n ,s n+1This process can be used multiple times to optimize the parameters θ of the Actor network, improving training efficiency and method stability; that is, step 6 is repeated multiple times. One round of the process described in step 4, plus multiple rounds of the processes described in steps 5 and 6, is followed by copying all the parameters of the Actor network and Critic network to the Actor' network and Critic' network. The above one round of interaction, multiple optimization processes, and one copying process constitute one round of training.

[0102] Step S7 specifically includes:

[0103] Repeat steps S4-S6, i.e., perform multiple rounds of training until the parameters θ of the Actor network and the parameters of the Critic network in the agent are equal. Continue until convergence is achieved. During the repetition of steps 4-6, the total score R = r1 + r2 + ... + r2 obtained from each round of agent-environment interaction should be used as the basis. n The value of μ is used to reduce the learning rate μ of the neural network. a and μ c To prevent overfitting.

[0104] Step S8 specifically includes:

[0105] The Actor neural network obtained in step S7 is deployed into the control system of the gyro-based roll stabilization device. The Actor's input layer has 5 parameters, and the gyro-based roll stabilization device detects the ship's roll angle φ using an electronic gyroscope. k Ship roll rate Using an encoder to detect the gyroscope precession angle β k The precession angular velocity of the gyroscope is calculated based on the sampling frequency. Using the state φ from the previous control cycle k-1 , β k-1 , The precession control torque u of the previous control cycle k-1 And the estimated value α′ of the wave excitation in the previous control cycle is obtained from the current state estimation. k-1 ,Will Five parameters are used as input parameters to the Actor network. After forward propagation of the Actor network, two parameters are output as the mean μ and standard deviation σ of a normal distribution. Then, a normal distribution is constructed, and the precession control torque u of the gyroscope roll reduction device is obtained by randomly selecting values ​​from this normal distribution. k Then, the control system controls the precession servo motor of the gyroscope anti-roll device to output a value of u in torque output mode. k The torque acts on the precession shaft, completing the control of the current control cycle. Repeating this process continuously constitutes the control process of the gyro anti-roll device.

[0106] Examples are given below:

[0107] Taking a 14t displacement fishing boat as an example, the specifications of the gyro anti-roll device are 360kg in mass, and the joint dynamics model of the system is as follows:

[0108]

[0109] u is the control variable, i.e., the precession shaft control torque, where ε i The phase angle is 28 random numbers. Because the waves are random, the environment must be initialized at the beginning of each round of agent-environment interaction. Each time the environment is initialized, the computer randomly selects 28 random numbers in the range [0~2π] as ε. i This is to prevent overfitting during neural network training.

[0110] According to formula (2), the reward function is:

[0111]

[0112]

[0113] r = r φ +r β

[0114] Construct four neural networks: Actor, Actor′, Critic, and Critic′. The Actor network structure is as follows:

[0115] The input layer has 5 nodes, meaning there are 5 input parameters. The activation function is ReLU. The second layer of the neural network has 64 nodes with ReLU activation, the third layer has 128 nodes with ReLU activation, and the fourth layer has 256 nodes with ReLU activation. The output layer has two nodes representing the mean μ and standard deviation σ of a normal distribution, with activation functions tanh and softplus, respectively. The value output based on the probability of constructing a normal distribution using the mean and standard deviation is the control torque u of the precession axis of the gyroscope anti-roll device. k .

[0116]

[0117] The Critic network structure is as follows: the input layer has 5 nodes, meaning there are 5 input parameters. The activation function is ReLU. The second layer of the neural network has 64 nodes with ReLU activation, the third layer has 128 nodes with ReLU activation, and the fourth layer has 256 nodes with ReLU activation. The output layer has one node that outputs the current state s. kThe estimated value of the value function. The neural network structures of Actor′ and Critic′ are consistent with those of Actor and Critic, respectively.

[0118] like Figure 2 As shown, the learning rate μ of the Actor and Critic networks is set. a =0.00002 and μ c =0.00002, discount rate γ = 0.99, decay rate λ = 0.95, pruning rate ε = 0.2. Then initialize the neural network and the environment. Then let the agent interact with the environment and score it using the reward function. 4000 interactions are counted as one round of interaction. If |φ|>0.2967 or |β|>1.3962 occurs during the interaction, the round of interaction is terminated early. Therefore, one round of interaction will yield the following empirical data (s0,u0,r0,s1), (s1,u1,r1,s2)...(s n ,u n ,r n ,s n+1 (where n≤4000), the experience data is placed into the experience pool. After one round of interaction, the AI ​​retrieves the experience data from the experience pool and clears the pool. Then, as follows... Figure 1 As shown: The agent uses the aforementioned empirical data and the Critic network to calculate the advantage function value A. θ (s k ,a k The calculation formula is as follows:

[0119]

[0120] A k =adv k +0.94A k+1 =adv k +0.94(adv k+1 +0.94(adv k+2 +0.94(adv k+3 +…)))(Note: A θ (s k ,a k That is, A k )

[0121] Then, P is calculated using empirical data and the Actor network. θ (a k |s k ), empirical data, and Actor's network calculation P θ′ (a k |s k Then, construct the Actor network loss function and perform gradient ascent optimization:

[0122]

[0123]

[0124] Then, using empirical data and the Critic and Critic' networks, the loss function of the Critic network is constructed and optimized using gradient descent:

[0125]

[0126]

[0127] For θ and Each gradient optimization is recorded as one neural network optimization process. Because the Actor network loss function applies the importance sampling principle, the data from the agent's interaction with the environment can be repeatedly used for training. The data obtained from this one round of interaction will be used for eight rounds of neural network optimization. After completing eight rounds of neural network optimization, the latest Actor and Critic network parameters are copied into the Actor' and Critic' networks. The above process of one round of interaction, eight optimizations, and one copy is called one round of training. After 1500 rounds of training, the learning rate μ is... a =0.00002 and μ c =0.00002 adjusted to μ a =0.000007 and μ c =0.000007, then continue training until the neural network converges. The training results are as follows. Figure 5 As shown, the horizontal axis represents the number of training rounds, and the vertical axis represents the total score R obtained in one round of interaction. k It can be seen that after 2000 training rounds, R... k The fact that the value has been reached and stabilized indicates that the neural network has converged.

[0128] The control system of the gyroscope anti-roll device, such as Figure 3 As shown, the trained Actor is deployed to a small computing unit with neural network operation capabilities as the host computer, and an STM32F4 microcontroller is used as the slave computer. The STM32F4 controls the encoder to acquire the gyroscope precession angle β. k The precession angular velocity of the gyroscope is calculated based on the sampling frequency. Controlling the electronic gyroscope to detect the ship's roll angle φ k Ship roll rate Using the state φ from the previous control cycle k-1 , β k-1 , The precession control torque u of the previous control cycle k-1And the estimated value α′ of the wave excitation in the previous control cycle is obtained from the current state estimation. k-1 Then the STM32F4 transmits via serial port... The data is sent to the host computer, which then uses the received data as input to the neural network for forward propagation and constructs a normal distribution to output the precession control torque u of the gyroscope roll reduction device. k The value is then sent to the STM32F4 via serial port. After receiving the value, the STM32F4 uses CANopen to control the precession servo motor to output a value of u in torque mode. k The torque acts on the precession shaft of the gyroscope anti-roll device, and the above process is repeated to complete the control of the gyroscope anti-roll device.

[0129] The control method for the gyroscope roll reduction device based on deep reinforcement learning described above brings the following beneficial technical effects:

[0130] (1) The gyroscope anti-roll device using this control method has a better anti-roll effect, especially when the sea wave excitation is large, the anti-roll effect can still maintain a good level.

[0131] (2) The control system using this control method has strong robustness.

[0132] The good control effect achieved by the above control method is shown as follows: Figure 6 As shown, the horizontal axis represents time, and the vertical axis represents the ship's roll angle. The thin solid line represents the ship's roll angle without a gyro-based roll reduction device, while the thick solid line represents the roll angle of a ship equipped with a gyro-based roll reduction device using this invention as a control method. The control method of this invention enables the ship's roll angle to be controlled between -5° and 5°, which is much smaller than the range of -15° to 15° for a ship without a gyro-based roll reduction device. It is evident that this control method has a good roll reduction control effect.

Claims

1. A control method for a gyroscope roll reduction device based on deep reinforcement learning, characterized in that: The method includes the following steps: Step 1: Analyze the motion characteristics of the system when the gyroscope anti-roll device is working, and build a virtual simulation environment in the computer that combines the gyroscope, ship and waves. This virtual simulation environment interacts with the agent during the method training. Step 2: Design a reward function to evaluate the quality of the agent's decisions; the reward function is based on the ship's roll angle. The precession angle of the gyroscope The evaluation metrics and reward function are as follows: ； in , , , , The roll angle and precession angle are selected based on the specifications of the ship and the gyro-based roll damping device, requiring 0°. 0 ; Step 3: Design the agent structure. The agent contains two Actor networks, two Critic networks, and a structure of four neural networks. Set all the necessary parameters for the method and initialize the four neural networks. The agent contains a total of four neural networks, namely... , , , Their corresponding weights and biases are as follows: , , , ,in and It is the actual neural network that needs to be updated. and It is a neural network that actually interacts with the environment; and The parameter values ​​change after each training round. and It was copied over once, among which The neural network structure is a linear fully connected multilayer neural network, and the five input parameters of the input layer are the ship's roll angle. Ship roll rate gyroscope precession angle gyroscope precession angular velocity Estimate of wave excitation in the previous control period ,remember( , , , , )for ; Step 4: Allow the intelligent agent to continuously interact with the virtual simulation environment to obtain a series of data; Step 5: The reward function in Step 2 scores the agent's behavior based on the data obtained in Step 4, and calculates the advantage function value of each interaction between the agent and the environment based on the score and the agent's Critic network. Step 6: Based on the agent behavior scores and advantage function values ​​obtained in Step 5, construct the appropriate loss functions for the Actor and Critic networks, respectively. Then, optimize the parameters of the Actor network based on the loss function values ​​using the gradient optimization approach. Parameter values ​​of the Critic network θ and v are the weights of the Actor network and the Critic network, respectively; Step 7: Repeat steps 4-6 until the parameters of the Actor network in the agent are... Parameter values ​​of the Critic network Until they converge; Step 8: Deploy the Actor network from the converged intelligent agent into the control system of the gyroscope anti-roll device to achieve control of the gyroscope anti-roll device.

2. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 1, characterized in that, Step 1 analyzes the motion state of the gyro anti-roll device when it is installed on the ship and encounters waves. To reduce the computational load of the simulation process and shorten the training time of the method while ensuring the accuracy of the system, the mathematical model is simplified, and the joint dynamic model of the three is finally established as follows: ; in For ship displacement, For the ship's transverse center of gravity to be high, Let be the moment of inertia of the ship along its roll axis. Let be the wave dip angle function excited by the ocean waves. The water surface damping coefficient, Let be the angular momentum of the gyroscope rotor. Let be the moment of inertia of the gyroscope rotor along the precession axis. For the ship's roll angle, This represents the precession angle of the gyroscope. The control torque for gyroscope precession is the control quantity; the program is developed based on the current state of the system. , , , and control quantity As input, the next state of the system , , , As output.

3. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 1, characterized in that, The intermediate layers consist of several linear neural network layers, and the output layer outputs two values, which are the mean of a normal distribution. and standard deviation The activation functions are tanh and softplus, respectively. A normal distribution is constructed using the mean and standard deviation. The output value based on the probability is the control torque of the precession axis of the gyroscope anti-roll device. , ; The input parameters of the network input layer and The network is consistent, with intermediate layers consisting of several linear neural network layers, and the output layer outputting the current state. The estimated value of the function; Indicates based on the current The control torque applied.

4. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 3, characterized in that, Step 4 describes the process of the intelligent agent interacting with the virtual simulation environment, which involves initializing the environment and obtaining its initial state. , , , ),initial 5 parameters input The network constructs a normal distribution based on the output values ​​and outputs the gyroscope precession control torque randomly according to probability. Then input it into the environment and it returns a new set of states. , , , Then, score the current interaction result according to the reward function. ,according to( , , , ), ( , , , )and Estimated Then ( , , , , ) is denoted as enter get The loop operation will result in a data sequence; this process is called a round of interaction. , , , ), ( , , , )...( , , , ), The number of interactions in this round, where the number of interactions in each round depends on the set value n or the ship's roll angle. gyroscope precession angle If the range is exceeded, the interaction will be terminated early. A random wave stimulus will be reconstructed each time the environment is initialized to prevent the neural network from overfitting and to improve the robustness of the controller. At the same time, in order to ensure the accurate calculation of the advantage function value and the accurate calculation of the Actor network loss function, the larger the value of n>2000, the better. However, the larger the value, the longer the training time will be.

5. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 4, characterized in that, Based on the data sequence described in step 3: ( , , , ), ( , , , )...( , , , Calculate the advantage function value for each interaction with the virtual simulation environment. : ； in As a discount factor, The attenuation rate, Indicates will Substitute The estimated value of the value function obtained from network computation; According to step 6, the obtained ( , , , ..., ) used for construction The network's loss function: ; Note: Indicates based on the current The action performed, i.e., the output precession shaft control torque, is therefore the same as... ; for Network calculation of advantage function value; in, The clipping rate is the clipping function; the loss function represents the state of the ship and the gyro-based roll reduction device when... Perform actions under the circumstances That is, the control torque of the gyroscope precession axis is Advantage function value at time The expected value of the advantage function is as high as possible; therefore, based on the idea of ​​gradient ascent, it is updated using the following formula. : ； is the learning rate of the Actor network; To ensure the accuracy of the dominant function calculation, continuous optimization is still needed. Network, Construction The network's loss function: ； When the loss function value is minimized The closer the estimated value of the value function output by the network is to the true value of the value function, the better the dominant function value is at that time. The more accurate the calculation, the better. Network parameters The effectiveness of gradient optimization; therefore, based on the idea of ​​gradient descent... Network parameters The update is as follows: ； is the learning rate of the Critic network; because The network's loss function employs an importance-based approach; therefore, the data sequence obtained in the first round of interaction is as follows: ( , , , ), ( , , , )...( , , , It can be used for optimization multiple times. Network parameters To improve training efficiency and method stability, the above process is repeated multiple times, and then... Network and Copy all network parameters to Network and For a network, the above-mentioned one round of interaction, multiple optimization processes, and one copying process constitute one round of training.

6. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 4, characterized in that, As described in step 7, repeat steps 4-6, i.e., perform multiple rounds of training until the parameters of the Actor network in the agent are determined. Parameter values ​​of the Critic network Repeat steps 4-6 until all convergences are achieved, based on the total score obtained from each round of agent-environment interaction. The value of is used to reduce the learning rate of the neural network. and To prevent overfitting.

7. The control method for the gyroscope roll reduction device based on deep reinforcement learning as described in claim 5, characterized in that, Step 8 involves deploying the Actor neural network into the control system of the gyro-based roll stabilization device. Specifically, the Actor's input layer has five parameters, and the gyro-based roll stabilization device detects the ship's roll angle using an electronic gyroscope. Ship roll rate The encoder is used to detect the precession angle of the gyroscope. The precession angular velocity of the gyroscope is calculated based on the sampling frequency. Utilize the state of the previous control cycle , , , Precession control torque of the previous control cycle And the estimated value of wave excitation from the previous control cycle is obtained from the current state estimation. ,Will( , , , , Five parameters are used as input parameters to the Actor network. After forward propagation of the Actor network, the output parameters are the mean of a normal distribution. and standard deviation Then, a normal distribution is constructed, and the precession control torque of the gyroscope anti-roll device is obtained by randomly selecting values ​​based on this normal distribution. Then, the control system controls the precession servo motor of the gyroscope anti-roll device to output a magnitude of [value missing] in torque output mode. The torque acts on the precession shaft to complete the control of the current control cycle; the continuous repetition of the above process is the control process of the gyroscope anti-roll device.

Images