Training system, training method, control system, and control method

Abstract

A training system, training method, control system, and control method are disclosed. The training method includes, performed by the processing module, (a) initializing the training module and (b) training the training module based on a simulation unit using a dual-delay deep deterministic policy gradient algorithm.

Claims

1. A training system comprising: a simulation unit configured to, in response to receiving an action, return a reward corresponding to the action and a next state of a simulated unmanned surface vehicle, wherein, The reward is the sum of a roll reward value, a traverse speed reward value, an extreme action selection reward value, and a traverse speed maintenance reward value, wherein the roll reward value is used to reward when the roll angle is small, the traverse speed reward value is used to penalize when the traverse speed exceeds a target value, the extreme action selection reward value is used to penalize when the action value of the action is in an extreme range, and the traverse speed maintenance reward value is used to reward when the absolute value of the speed difference between the traverse speed and a target speed is small; and a training unit, comprising: A training module includes an experience replay buffer, a first value network, a second value network, a policy network, a first target value network, a second target value network, and a target policy network, wherein the first target value network, the second target value network, and the target policy network have the same architecture as the first value network, the second value network, and the policy network, respectively; and a processing module configured to initialize the training module and train the training module based on the simulation unit using a dual-delay deep deterministic policy gradient algorithm.

2. The training system as described in claim 1, wherein the policy network is a first fully connected network, the first value network is a second fully connected network, the second value network is a third fully connected network, a startup function of an output layer of the first fully connected network is a hyperbolic tangent function, a startup function of an output layer of the second fully connected network is a linear function, and a startup function of an output layer of the third fully connected network is a linear function.

3. The training system as claimed in claim 1, wherein the simulation unit generates the roll reward value based on the following steps: (a) in response to an absolute value of the roll angle being less than or equal to a predetermined angle, setting the traverse speed reward value to a first value multiplied by a difference between the absolute value of the roll angle and a second value, wherein the second value is a positive real number; and (b) in response to the absolute value of the roll angle being greater than the predetermined angle, setting the traverse speed reward value to 0.

4. The training system as claimed in claim 1, wherein the simulation unit generates the longitudinal velocity reward value based on the following steps: (a) setting the longitudinal velocity reward value to a first negative value in response to the longitudinal velocity being greater than the target velocity and the action value of the action being greater than or equal to 0; and (b) setting the longitudinal velocity reward value to 0 in response to the longitudinal velocity being less than or equal to the target velocity or the action value of the action being less than 0.

5. The training system as claimed in claim 1, wherein the simulation unit generates the extreme action selection reward value based on the following steps: (a) setting the extreme action selection reward value to a second negative value in response to the action value being less than a third value or the action value being greater than a fourth value, wherein the third value and the fourth value are both positive real numbers; and (b) setting the extreme action selection reward value to 0 in response to the action value being greater than or equal to the third value and the action value being less than or equal to the fourth value.

6. The training system as claimed in claim 1, wherein the simulation unit generates the longitudinal speed maintenance reward value based on the following steps: (a) in response to the absolute value of the speed difference being less than or equal to a preset speed, setting the longitudinal speed maintenance reward value to the difference between the preset speed and the absolute value of the speed difference multiplied by a fifth value, wherein the fifth value is a positive real number; and (b) in response to the absolute value of the speed difference being greater than the preset speed, calculating a product of the difference between the absolute value of the speed difference and the preset speed and a negative value of the fifth value, and setting the longitudinal speed maintenance reward value to the larger of the negative value of the fifth value and the product.

7. A control system for a trained target policy network obtained using the training system described in any one of claims 1 to 6, comprising: a neural network module including the trained target policy network; and a control processing module configured to perform: receiving a current roll angle and a current pitch speed of an unmanned surface vehicle as a current state of the unmanned surface vehicle; inputting the current state into the trained target policy network to obtain an output; and controlling a propeller speed of the unmanned surface vehicle based on the output.

8. A training method applicable to a training system, wherein the training system comprises: a simulation unit configured to, in response to receiving an action, return a reward corresponding to the action and a next state of a simulated unmanned surface vehicle, wherein, The reward is the sum of a roll reward value, a traverse speed reward value, an extreme action selection reward value, and a traverse speed maintenance reward value, wherein the roll reward value is used to reward when the roll angle is small, the traverse speed reward value is used to penalize when the traverse speed exceeds a target value, the extreme action selection reward value is used to penalize when the action value of the action is in an extreme range, and the traverse speed maintenance reward value is used to reward when the absolute value of the speed difference between the traverse speed and a target speed is small; and a training unit, comprising: a training module, comprising an experience replay buffer, a first value network, a second value network, a policy network, a first target value network, a second target value network, and a target policy network, wherein the first target value network, the second target value network, and the target policy network have the same architecture as the first value network, the second value network, and the policy network, respectively; and a processing module; the training method comprises being executed by the processing module: (a) Initialize the training module; and (b) Train the training module based on the simulation unit using a dual-delay depth deterministic policy gradient algorithm.

9. The training method as described in claim 8, wherein the policy network is a first fully connected network, the first value network is a second fully connected network, the second value network is a third fully connected network, a startup function of an output layer of the first fully connected network is a hyperbolic tangent function, a startup function of an output layer of the second fully connected network is a linear function, and a startup function of an output layer of the third fully connected network is a linear function.

10. The training method as described in claim 8, wherein the simulation unit generates the roll reward value based on the following steps: (a) in response to an absolute value of the roll angle being less than or equal to a predetermined angle, setting the traverse speed reward value to a first value multiplied by a difference between the absolute value of the roll angle and a second value, wherein the second value is a positive real number; and (b) in response to the absolute value of the roll angle being greater than the predetermined angle, setting the traverse speed reward value to 0.

11. The training method as described in claim 8, wherein the simulation unit generates the longitudinal velocity reward value based on the following steps: (a) setting the longitudinal velocity reward value to a first negative value in response to the longitudinal velocity being greater than the target velocity and the action value of the action being greater than or equal to 0; and (b) setting the longitudinal velocity reward value to 0 in response to the longitudinal velocity being less than or equal to the target velocity or the action value of the action being less than 0.

12. The training method as described in claim 8, wherein the simulation unit generates the extreme action selection reward value based on the following steps: (a) in response to the action value of the action being less than a third value or the action value of the action being greater than a fourth value, setting the extreme action selection reward value to a second negative value, wherein the third value and the fourth value are both positive real numbers; and (b) in response to the action value of the action being greater than or equal to the third value and the action value of the action being less than or equal to the fourth value, setting the extreme action selection reward value to 0.

13. The training method as described in claim 8, wherein the simulation unit generates the longitudinal velocity maintenance reward value based on the following steps: (a) in response to the absolute value of the velocity difference being less than or equal to a preset velocity, setting the longitudinal velocity maintenance reward value to the difference between the preset velocity and the absolute value of the velocity difference multiplied by a fifth value, wherein the fifth value is a positive real number; and (b) in response to the absolute value of the velocity difference being greater than the preset velocity, calculating a product of the difference between the absolute value of the velocity difference and the preset velocity and a negative value of the fifth value, and setting the longitudinal velocity maintenance reward value to the larger of the negative value of the fifth value and the product.

14. A control method for a trained target policy network obtained using the training method described in any one of claims 8 to 13, executed by a control processing module, the control method comprising: (a) receiving a current roll angle and a current pitch speed of an unmanned surface vehicle as a current state of the unmanned surface vehicle; (b) inputting the current state into the trained target policy network to obtain an output, wherein the trained target policy network is included in a neural network module; and (c) controlling a propeller speed of the unmanned surface vehicle based on the output.

Images