A full-revolution pile-driving ship control method and system based on deep reinforcement learning

Abstract

The present invention discloses a control method and system for a full-revolution pile-driving ship based on deep reinforcement learning, including: constructing a pile-driving strategy model, and performing pile-driving simulation training on the pile-driving strategy model; installing a sensor on the pile-driving ship, responding to a pile-driving instruction, wherein the pile-driving instruction includes a target pile body and a target state of the target pile body, and controlling the pile-driving ship to go to the target pile body; using the sensor to obtain the state parameters of the pile-driving ship, inputting the state parameters into the pile-driving strategy model, the pile-driving strategy model outputs a pile-driving action instruction, and the pile-driving ship operates the target pile body according to the pile-driving action instruction until the pile-driving instruction is completed, and the pile-driving is terminated. The present invention constructs a strategy model using DQN, trains the strategy model using experience replay and exploration-learning mechanisms, inputs real-time state parameters into the strategy model to obtain pile-driving action instructions, and thus automatically operates the pile driver according to sea conditions, ship conditions, etc. in real time, reduces human intervention, and improves pile driving accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of piling by a piling ship, and in particular to a control method and system for a fully rotating piling ship based on deep reinforcement learning. Background Art

[0002] A pile-driving ship is an engineering ship used for driving and pulling foundation piles for hydraulic structures. A sturdy triangular truss pile frame and a pile hammer are installed in the center of the bow. During operation, the pile lifting, ship moving and positioning are completed through linkage mechanisms such as steel cables, pulleys and winches, and the piles are driven underwater with the explosive force of the pile hammer.

[0003] A fully revolving pile-driving vessel plays an important role in marine engineering construction, especially in the construction of large offshore wind farms and offshore platforms. Traditional pile-driving vessels usually rely on manual operation and preset procedures, which often makes it difficult to achieve optimal pile driving efficiency and accuracy in complex and ever-changing marine environments.

[0004] Therefore, there is a need for a method that can adjust the piling action according to environmental changes in real time to improve the piling efficiency. Summary of the invention

[0005] To solve the above problems, the present invention provides a control method and system for a full-revolution pile-driving vessel based on deep reinforcement learning. The piling is simulated and trained by constructing a DQN piling strategy model. Sensors are set on the piling vessel to input parameters into the piling strategy model to generate a piling strategy, so that piling is performed in real time according to the current state. This solves the problem that the prior art relies on manual operation and preset programs, resulting in low piling efficiency and accuracy.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A control method for a full-revolution pile-driving ship based on deep reinforcement learning comprises the following steps:

[0008] S1. Construct a piling strategy model, wherein the piling strategy model is a DQN model, and perform piling simulation training on the piling strategy model to obtain a trained piling strategy model;

[0009] S2. Install sensors on the piling ship, the sensors include an attitude meter, GPS, anemometer, Doppler current meter, in response to the piling instruction, the piling instruction includes the target state of the target pile and the target pile, control the piling ship to the target pile;

[0010] S3. Use sensors to obtain the current state parameters of the piling ship, input the state parameters to the trained piling strategy model, the piling strategy model outputs piling action instructions, and the piling ship operates the target pile according to the piling action instructions;

[0011] S4. Repeat step S3 until the target pile body reaches the target state, and then end the piling.

[0012] Furthermore, the state parameters include ship position, ship orientation, ship speed, pile state and sea conditions.

[0013] Furthermore, the piling strategy model includes a Q-value network and a target Q-value network. The Q-value network and the target Q-value network are two convolutional neural networks with the same structure. The Q-value network and the target Q-value network both include 3 convolutional layers and 2 fully connected layers.

[0014] Furthermore, the piling strategy model is subjected to piling simulation training, and its specific implementation method is as follows:

[0015] S211. Initialize the parameters of the Q value network and copy the parameters of the Q value network to the target Q value network;

[0016] S212. Randomly generate state parameter s, input state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the pile driving action a to execute according to the output Q value function, and obtains the next state parameter after executing the pile driving action a , input to the target Q value network , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function;

[0017] S213. Repeat step S212, and copy the parameters of the Q value network to the target Q value network every time step S212 is repeated a fixed number of times, until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than the preset loss, and then the training ends.

[0018] Furthermore, the piling simulation training of the piling strategy model further includes:

[0019] S221. Initialize the parameters of the Q value network, copy the parameters of the Q value network to the target Q value network, randomly generate state parameters s, construct a replay memory unit D, the capacity of the replay memory unit D has a preset upper limit, and initialize the replay memory unit;

[0020] S222. Input the state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the piling action a to execute according to the output Q value function, and obtains the next state parameter after executing the piling action a And the environmental reward r after executing the pile driving action a, input to the target Q value network , get the target Q value function, calculate the loss between the Q value function and the target Q value function, update the parameters of the Q value network according to the loss between the Q value function and the target Q value function, and update the state parameter s, action a, the environment reward r after executing the pile driving action a, and the next state parameter As a set of transfer samples , , the sample will be transferred Store in playback memory unit D, and set the state parameter s equal to the next state parameter ;

[0021] S223. Repeat step S222, and copy the parameters of the Q value network to the target Q value network every time step S222 is repeated a fixed number of times, until the playback memory unit D reaches the upper capacity limit. ;

[0022] S224. Randomly select a set of transfer samples from the replay memory unit D , transfer samples to the Q value network input The state parameter s in the Q value network outputs the corresponding Q value function, and the transfer sample is input to the target Q value network. In , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function;

[0023] S225. Repeat step S224 until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than a preset loss, and then end the training.

[0024] Furthermore, the loss between the calculated Q value function and the target Q value function is specifically implemented as follows: the error calculation formula is:

[0025]

[0026] in, , represents the output of the Q-value network at the i-th iteration, represents the parameters of the Q-value network at the i-th iteration, s is the state parameter, a is the pile driving action, is the next state parameter after executing pile driving action a, r is the environmental reward after executing pile driving action a, is the discount parameter, is the output of the target Q value network at the i-th iteration, is the parameter of the target Q value network at the i-th iteration, and E represents the mean square error.

[0027] Furthermore, it also includes a preset probability ε, and the piling action a is selected for execution according to the output Q value function. The specific implementation method is: preset probability ε, generate a random number R between 0 and 1, when the random number R is less than the probability ε, randomly select an action a to execute, when the random number R is greater than the probability ε, the Q value function Indicates execution of an action The Q value obtained after that is calculated, the action a when the Q value is the largest is calculated, and the action a when the Q value is the largest is selected for execution; each time an iteration is performed, the probability ε decreases once.

[0028] Furthermore, the method of updating the parameters of the Q-value network according to the loss between the Q-value function and the target Q-value function includes: updating the parameters of the Q-value network according to the stochastic gradient descent method.

[0029] Further, in step S3, the state parameter is input to the trained piling strategy model, and the piling strategy model outputs the piling action instruction. The specific implementation method is: the state parameter s is input to the Q value network, and the Q value network outputs the Q value function corresponding to the state parameter s. , please Maximum pile driving action , That is, the output pile driving action instruction, which includes the pile hammer position, driving force and driving frequency.

[0030] Through the above technical scheme, the present invention has the following beneficial effects: using DQN to build a strategy model, using experience replay and exploration-learning mechanism to train the strategy model, and setting sensors on the pile-driving ship to obtain state parameters in real time according to the sensors, and inputting the real-time state parameters into the strategy model to obtain piling action instructions, so as to automatically operate the pile driver according to sea conditions, ship conditions, etc. in real time, reduce human intervention, and improve piling accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the overall process of a full-revolution pile-driving ship control method based on deep reinforcement learning of the present invention.

[0032] Figure 2 It is a structural schematic diagram of a full-revolution pile-driving vessel control system based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Example 1

[0036] See also Figure 1 , a full-turn pile-driving ship control method based on deep reinforcement learning, comprising the following steps:

[0037] S1. Construct a piling strategy model, wherein the piling strategy model is a DQN model, and perform piling simulation training on the piling strategy model to obtain a trained piling strategy model;

[0038] S2. Install sensors on the piling ship, the sensors include an attitude meter, GPS, anemometer, Doppler current meter, in response to the piling instruction, the piling instruction includes the target state of the target pile and the target pile, control the piling ship to the target pile;

[0039] S3. Use sensors to obtain the current state parameters of the piling ship, input the state parameters to the trained piling strategy model, the piling strategy model outputs piling action instructions, and the piling ship operates the target pile according to the piling action instructions;

[0040] S4. Repeat step S3 until the target pile body reaches the target state, and then end the piling.

[0041] Install attitude indicators, GPS, anemometers, Doppler current meters and other sensors on the full-revolving pile-driving vessel to monitor the movement status and environmental parameters of the pile-driving vessel. Ensure that all sensors are installed and debugged, and data can be transmitted to the control system in real time.

[0042] The DQN algorithm, also known as the Deep Q Network algorithm, pre-trains the control strategy through environmental simulation, enabling the system to automatically learn the optimal piling control strategy in a continuous trial and error process. The piling process involves multiple variables and uncertain factors, such as geological conditions, equipment status, and operating parameters. Using deep neural networks, DQN can effectively process and learn complex relationships in high-dimensional state spaces, which makes it particularly good at dealing with dynamically changing piling environments. DQN continuously optimizes its strategy through continuous trial and error and learning, thereby finding the optimal operation path under different piling conditions. This self-learning ability enables it to continuously improve the accuracy and efficiency of piling control in actual operations. In addition, because DQN relies on neural networks for strategy learning, its model has strong generalization capabilities and can adapt to different piling environments and construction conditions. This means that the same DQN model can work effectively under a variety of piling conditions, reducing the need for model adjustment and retraining.

[0043] The learning process of the piling strategy model includes: interacting with the environment at each moment to obtain a high-dimensional observation, and using deep learning methods to perceive the observation to obtain a specific state feature representation. At each moment, the piling strategy model observes the environment and obtains the state. The piling strategy model generates a Q function based on the current state. The Q function is a value function for evaluating the current state-action pair, and maps the current state to the corresponding action through a certain strategy. The environment responds to this action and obtains the next observation. By continuously looping the above process, the optimal strategy to achieve the goal can be finally obtained.

[0044] In an optional embodiment, the state parameters include ship position, ship heading, ship speed, pile state and sea conditions.

[0045] In an optional embodiment, the piling strategy model includes a Q-value network and a target Q-value network, the Q-value network and the target Q-value network are two convolutional neural networks with the same structure, and the Q-value network and the target Q-value network both include 3 convolutional layers and 2 fully connected layers.

[0046] DQN uses a target network to stabilize the training process. The parameters of the target Q-value network are not updated in every time step, but the parameters of the Q-value network are copied after a certain interval. This can make the learning process more stable and avoid the oscillation problem caused by the rapid update of the online network.

[0047] In an optional embodiment, the piling strategy model is subjected to piling simulation training, and its specific implementation is as follows:

[0048] S211. Initialize the parameters of the Q value network and copy the parameters of the Q value network to the target Q value network;

[0049] S212. Randomly generate state parameter s, input state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the pile driving action a to execute according to the output Q value function, and obtains the next state parameter after executing the pile driving action a , input to the target Q value network , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function;

[0050] S213. Repeat step S212, and copy the parameters of the Q value network to the target Q value network every time step S212 is repeated a fixed number of times, until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than the preset loss, and then the training ends.

[0051] In an optional embodiment, the performing piling simulation training on the piling strategy model further includes:

[0052] S221. Initialize the parameters of the Q value network, copy the parameters of the Q value network to the target Q value network, randomly generate state parameters s, construct a replay memory unit D, the capacity of the replay memory unit D has a preset upper limit, and initialize the replay memory unit;

[0053] S222. Input the state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the piling action a to execute according to the output Q value function, and obtains the next state parameter after executing the piling action a And the environmental reward r after executing the pile driving action a, input to the target Q value network , get the target Q value function, calculate the loss between the Q value function and the target Q value function, update the parameters of the Q value network according to the loss between the Q value function and the target Q value function, and update the state parameter s, action a, the environment reward r after executing the pile driving action a, and the next state parameter As a set of transfer samples , , the sample will be transferred Store in playback memory unit D, and set the state parameter s equal to the next state parameter ;

[0054] S223. Repeat step S222, and copy the parameters of the Q value network to the target Q value network every time step S222 is repeated a fixed number of times, until the playback memory unit D reaches the upper capacity limit. ;

[0055] S224. Randomly select a set of transfer samples from the replay memory unit D , transfer samples to the Q value network input The state parameter s in the Q value network outputs the corresponding Q value function, and the transfer sample is input to the target Q value network. In , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function;

[0056] S225. Repeat step S224 until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than a preset loss, and then end the training.

[0057] Through the experience replay mechanism, the obtained transfer samples Insert playback memory unit Among them is the set of all environmental states. Indicates that the agent is The current working status of the ship and pile driver at all times, including environmental conditions such as waves, currents, wind speed, and the ship's position, speed, heading, and other status parameters. is a set of actions that the agent can perform. represents the actions taken by the agent at time t, including the adjustment of the ship's heading, the change of speed, the control of the pile hammer force and the frequency of impact; It is the reward from environmental feedback. If the accuracy and depth of pile driving meet expectations, the reward increases. If the ship deviates from its course or the pile driving force is insufficient, the reward decreases. During training, a small batch of transfer samples are randomly selected from D each time. In the training deep network, the samples are independent of each other, which reduces the correlation between samples and improves the stability of the algorithm. For the next state, through continuous state feedback, action adjustment and reward optimization, the model is able to gradually learn and adjust its control strategy, thereby improving the piling efficiency and ship stability, and ultimately achieving more precise and efficient piling operations.

[0058] In an optional embodiment, the loss between the calculated Q value function and the target Q value function is specifically implemented as follows: the error calculation formula is:

[0059]

[0060] in, , represents the output of the Q-value network at the i-th iteration, represents the parameters of the Q-value network at the i-th iteration, s is the state parameter, a is the pile driving action, is the next state parameter after executing pile driving action a, r is the environmental reward after executing pile driving action a, is the discount parameter, is the output of the target Q value network at the i-th iteration, is the parameter of the target Q value network at the i-th iteration, and E represents the mean square error.

[0061] After taking the derivative of the error function, we get the following gradient formula:

[0062]

[0063] The parameters of the Q-value network are updated according to the gradient formula using the stochastic gradient descent method.

[0064] Is a discount adjustment parameter used to reduce the impact of new values, ranging from 0 to 1.

[0065] In an optional embodiment, a preset probability ε is further included, and the piling action a is selected for execution according to the output Q value function. The specific implementation method is: the preset probability ε is generated, a random number R between 0 and 1 is generated, and when the random number R is less than the probability ε, an action a is randomly selected for execution; when the random number R is greater than the probability ε, the Q value function Indicates execution of an action The Q value obtained after that is calculated, the action a when the Q value is the largest is calculated, and the action a when the Q value is the largest is selected for execution; each time an iteration is performed, the probability ε decreases once.

[0066] In an optional embodiment, the updating of the parameters of the Q-value network according to the loss between the Q-value function and the target Q-value function is specifically implemented by: updating the parameters of the Q-value network according to a stochastic gradient descent method.

[0067] In an optional embodiment, in step S3, the state parameter is input to the trained piling strategy model, and the piling strategy model outputs the piling action instruction, which is specifically implemented as follows: the state parameter s is input to the Q value network, and the Q value network outputs the Q value function corresponding to the state parameter s. , please Maximum pile driving action , That is, the output pile driving action instruction, which includes the pile hammer position, driving force and driving frequency.

[0068] Specifically, the piling instruction also includes: adjusting the posture of the ship.

[0069] Example 2

[0070] See also Figure 2 , a full-revolution pile-driving ship control system based on deep reinforcement learning, including:

[0071] A model building module is used to build a piling strategy model, where the piling strategy model is a DQN model, and piling simulation training is performed on the piling strategy model to obtain a trained piling strategy model;

[0072] The command response module is equipped with sensors on the piling vessel, wherein the sensors include an attitude meter, a GPS, an anemometer, and a Doppler current meter, and controls the piling vessel to go to the target pile body in response to a piling command including a target pile body and a target state of the target pile body;

[0073] The piling execution module uses sensors to obtain the current state parameters of the piling ship, inputs the state parameters into the trained piling strategy model, and the piling strategy model outputs piling action instructions. The piling ship operates the target pile according to the piling action instructions until the piling instructions are completed and the piling is finished.

[0074] The embodiment disclosed in this specification is only an example of a unilateral feature of the present invention, and the protection scope of the present invention is not limited to this embodiment, and any other functionally equivalent embodiments fall within the protection scope of the present invention. For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A control method for a full-revolution pile-driving ship based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Construct a piling strategy model, which is a deep Q network DQN model. The piling strategy model includes a Q value network and a target Q value network. The Q value network and the target Q value network are two convolutional neural networks with the same structure. The Q value network and the target Q value network both include 3 convolutional layers and 2 fully connected layers. Perform piling simulation training on the piling strategy model. The piling simulation training on the piling strategy model is specifically implemented as follows: S211. Initialize the parameters of the Q value network and copy the parameters of the Q value network to the target Q value network; S212. Randomly generate state parameter s, input state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the pile driving action a to execute according to the output Q value function, and obtains the next state parameter after executing the pile driving action a , input to the target Q value network , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function; S213. Repeat step S212, and copy the parameters of the Q value network to the target Q value network every time step S212 is repeated for a fixed number of times, until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than the preset loss, then the training is terminated to obtain a trained piling strategy model; S2. Install sensors on the piling ship, the sensors include an attitude meter, GPS, anemometer, Doppler current meter, in response to the piling instruction, the piling instruction includes the target state of the target pile and the target pile, control the piling ship to the target pile; S3. Use sensors to obtain the current state parameters of the piling ship, input the state parameters to the trained piling strategy model, the piling strategy model outputs piling action instructions, and the piling ship operates the target pile according to the piling action instructions; S4. Repeat step S3 until the target pile body reaches the target state, and then end the piling.

2. The method for controlling a full-revolution pile-driving vessel based on deep reinforcement learning according to claim 1, characterized in that: The state parameters include ship position, ship orientation, ship speed, pile state and sea conditions.

3. The method for controlling a full-revolution pile-driving vessel based on deep reinforcement learning according to claim 1, characterized in that: The piling simulation training of the piling strategy model further includes: S221. Initialize the parameters of the Q value network, copy the parameters of the Q value network to the target Q value network, randomly generate state parameters s, construct a replay memory unit D, the capacity of the replay memory unit D has a preset upper limit, and initialize the replay memory unit; S222. Input the state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the piling action a to execute according to the output Q value function, and obtains the next state parameter after executing the piling action a And the environmental reward r after executing the pile driving action a, input to the target Q value network , get the target Q value function, calculate the loss between the Q value function and the target Q value function, update the parameters of the Q value network according to the loss between the Q value function and the target Q value function, and update the state parameter s, action a, the environment reward r after executing the pile driving action a, and the next state parameter As a set of transfer samples , , the sample will be transferred Store in playback memory unit D, and set the state parameter s equal to the next state parameter ; S223. Repeat step S222, and copy the parameters of the Q value network to the target Q value network every time step S222 is repeated a fixed number of times, until the playback memory unit D reaches the upper capacity limit. ; S224. Randomly select a set of transfer samples from the replay memory unit D , transfer samples to the Q value network input The state parameter s in the Q value network outputs the corresponding Q value function, and the transfer sample is input to the target Q value network. In , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function; S225. Repeat step S224 until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than a preset loss, and then end the training.

4. The method for controlling a fully rotating pile-driving ship based on deep reinforcement learning according to claim 1 or 3, characterized in that: The loss between the Q value function and the target Q value function is calculated as follows: the loss calculation formula between the Q value function and the target Q value function is: in, , represents the output of the Q-value network at the i-th iteration, represents the parameters of the Q-value network at the i-th iteration, s is the state parameter, a is the pile driving action, is the next state parameter after executing pile driving action a, r is the environmental reward after executing pile driving action a, is the discount parameter, is the output of the target Q value network at the i-th iteration, is the parameter of the target Q value network at the i-th iteration, and E represents the mean square error.

5. The method for controlling a fully rotating pile-driving ship based on deep reinforcement learning according to claim 1 or 3, characterized in that: It also includes a preset probability ε, and the piling action a is selected for execution according to the output Q value function. The specific implementation method is: preset probability ε, generate a random number R between 0 and 1, when the random number R is less than the probability ε, randomly select an action a to execute, when the random number R is greater than the probability ε, the Q value function Indicates execution of an action The Q value obtained after that is calculated, the action a when the Q value is the largest is calculated, and the action a when the Q value is the largest is selected for execution; each time an iteration is performed, the probability ε decreases once.

6. The method for controlling a fully rotating pile-driving vessel based on deep reinforcement learning according to claim 1 or 3, characterized in that: The method of updating the parameters of the Q value network according to the loss between the Q value function and the target Q value function includes: updating the parameters of the Q value network according to the stochastic gradient descent method.

7. The method for controlling a fully rotating pile-driving vessel based on deep reinforcement learning according to claim 1 or 3, characterized in that: In step S3, the state parameter is input to the trained piling strategy model, and the piling strategy model outputs the piling action instruction. The specific implementation method is: the state parameter s is input to the Q value network, and the Q value network outputs the Q value function corresponding to the state parameter s , wherein the Indicates action , please Maximum pile driving action , That is, the output pile driving action instruction, which includes the pile hammer position, driving force and driving frequency.

8. A full-revolution pile-driving ship control system based on deep reinforcement learning, characterized in that: include: A model building module is used to build a piling strategy model, wherein the piling strategy model is a deep Q network DQN model, and the piling strategy model includes a Q value network and a target Q value network, wherein the Q value network and the target Q value network are two convolutional neural networks with the same structure, and each of the Q value network and the target Q value network includes 3 convolutional layers and 2 fully connected layers; Performing piling simulation training on the piling strategy model to obtain a trained piling strategy model; The piling simulation training of the piling strategy model is specifically implemented as follows: S211. Initialize the parameters of the Q value network and copy the parameters of the Q value network to the target Q value network; S212. Randomly generate state parameter s, input state parameter s to the Q value network, the Q value network outputs the corresponding Q value function, selects the pile driving action a to execute according to the output Q value function, and obtains the next state parameter after executing the pile driving action a , input to the target Q value network , obtain the target Q value function, calculate the loss between the Q value function and the target Q value function, and update the parameters of the Q value network according to the loss between the Q value function and the target Q value function; S213. Repeat step S212, and copy the parameters of the Q value network to the target Q value network every time step S212 is repeated for a fixed number of times, until the number of repetitions reaches a preset number of times or the loss between the Q value function and the target Q value function is less than a preset loss, and then end the training; The command response module is equipped with sensors on the piling vessel, wherein the sensors include an attitude meter, a GPS, an anemometer, and a Doppler current meter, and controls the piling vessel to go to the target pile body in response to a piling command including a target pile body and a target state of the target pile body; The piling execution module uses sensors to obtain the current state parameters of the piling ship, inputs the state parameters into the trained piling strategy model, and the piling strategy model outputs piling action instructions. The piling ship operates the target pile according to the piling action instructions until the piling instructions are completed and the piling is finished.

Images