Unmanned ship supporting multiple underwater execution mechanisms and cooperative control method thereof

Abstract

The invention relates to an unmanned ship supporting multiple underwater execution mechanisms and a cooperative control method of the unmanned ship, and belongs to the field of artificial intelligence and unmanned ships. The system is composed of an unmanned ship, a plurality of underwater execution mechanisms and a cable. The method comprises the following steps: S1, initializing a deep reinforcement learning network; s2, the unmanned ship reaches the operation area to release the underwater execution mechanism; s3, the unmanned ship receives the operation task; s4, the underwater executing mechanism obtains the current state and feeds back the current state to the unmanned ship; s5, rewards of the constraint conditions are constructed; s6, generating an operation task distribution action and issuing the operation task distribution action to an underwater execution mechanism; s7, the underwater execution mechanism obtains a real-time state; s8, constructing a reward of the physical information; s9, generating an execution action and executing the execution action through an execution device; and S10, repeating the steps S3-S9 until all the operation tasks are completed. According to the method, organic fusion of constraint conditions and physical information can be realized, and automatic, optimized and cooperative control of multiple underwater execution mechanisms in an unsupervised complex environment is realized.

Description

Technical Field

[0001] The present invention relates to an unmanned ship supporting multiple underwater actuators and a coordinated control method thereof, belonging to the field of artificial intelligence and unmanned ships, and in particular to an unmanned ship supporting multiple underwater actuators and a coordinated control method thereof. Background Art

[0002] With the demand for river and lake ecological monitoring and improvement, unmanned operation boats are increasingly used in river and lake ecological maintenance, such as underwater terrain survey, underwater sediment disposal, and the use of chemicals to control blue algae blooms.

[0003] However, current unmanned vessels are mainly single actuators on the water surface, with a narrow operating range and low efficiency. However, when performing complex underwater tasks, existing unmanned vessels often require multiple underwater actuators that are independently controlled. These actuators lack an effective collaborative operation mechanism, and complex and changeable water scenes will lead to low operating efficiency and increased costs.

[0004] With the development of artificial intelligence technology, reinforcement learning-based methods can learn directly from experience without prior knowledge of the work scenario, and have good generalization and adaptability, and are expected to be used to achieve coordinated control of unmanned ships with multiple underwater actuators. At the same time, during the entire operation process, it is also necessary to consider constraints such as underwater dynamic obstacles and power consumption, as well as the efficiency and cost of the operation. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and to provide an unmanned ship supporting multiple underwater actuators and a collaborative control method thereof. The invention intends to construct a reinforcement learning network with physical information through the variational principle, and at the same time introduce environmental constraints and self-constraints, so as to transform the collaborative control problem of the unmanned ship with multiple underwater actuators into a problem of solving an optimization model, and finally realize the automatic, optimized and collaborative control of multiple underwater actuators in an unsupervised complex environment.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An unmanned boat supporting multiple underwater actuators, characterized in that it is composed of an unmanned boat, multiple underwater actuators, and cables; the unmanned boat and a single underwater actuator are connected by cables to achieve communication and power supply; the unmanned boat is composed of a hull, and a power system, a positioning module, an image sensor, a power supply, a communication module 1, and a distributed central processing unit 1 installed on the hull; the power system, the positioning module, the image sensor, and the communication module 1 are respectively connected to the distributed central processing unit 1; the power supply is respectively connected to the power system, the positioning module, the image sensor, the communication module 1, the distributed central processing unit 1, and the cable for power supply; the power system is an engine; the positioning system includes a GPS positioning system and a Beidou positioning system; the communication module 1 It comprises a wired communication module and a wireless communication module, wherein the wired communication module is connected to the cable, and the wireless communication module is connected to the client; the distributed central processing unit 1 is a microprocessor; any one of the underwater actuators is composed of a state perception sensor, an actuator, a communication module 2 and a distributed central processing unit 2; the state perception sensor, the actuator, and the communication module 2 are respectively connected to the distributed central processing unit 2; the state perception sensor is composed of an inertial sensor, a positioning sensor, a depth sensor, a sonar, and an infrared camera; the inertial sensor is a gyroscope; the communication module 2 is a wired communication module, connected to the cable; the actuator is a motor-driven mechanism for traveling and working; the distributed central processing unit 2 is a microprocessor.

[0008] A collaborative control method for an unmanned ship supporting multiple underwater actuators comprises the following steps:

[0009] S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer (Replay Buffer) in the distributed central processing unit 1 of the unmanned ship and the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer in the distributed central processing unit 2 of the underwater actuator;

[0010] S2: The unmanned ship receives the client's command and arrives at the operation area, and releases the underwater actuator;

[0011] S3: The unmanned ship receives the operation task from the client and issues the task to all underwater actuators according to the operation cycle T;

[0012] S4: At each operation cycle T, all underwater actuators obtain the current status through the state perception sensor and feed it back to the distributed central processing unit 1 on the unmanned ship;

[0013] S5: Construct a reward based on the constraints of the current state, power consumption, and task waiting time;

[0014] S6: The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 of the unmanned ship generates the task allocation action and issues the instruction to the corresponding underwater actuator;

[0015] S7: In an operation cycle T, the corresponding underwater actuator obtains the real-time status through the status perception sensor in real time;

[0016] S8: Based on the variational principle, the fastest path of the underwater actuator and the obstacle distance are constructed to construct the reward of physical information;

[0017] S9: Using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 of the corresponding underwater actuator generates an execution action and executes it through the execution device;

[0018] S10: Repeat steps S3 to S9 until all the tasks are completed.

[0019] The DDPG deep reinforcement learning network based on the Actor-Critic framework adopts the Markov Decision Process (MDP) as the gradient strategy, which includes a triple (S, A, R), where S is the perception state space of the virtual machine, A is the task scheduling action space of the virtual machine, and R is the reward.

[0020] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 of the unmanned ship is composed of an actor network μ(s|θ μ ), critic network Q(s,a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ ); where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s is the current state of all underwater actuators; a is the task scheduling action.

[0021] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 of the underwater actuator end is composed of an actor network Critics Network Actor target network Critics target network constitute; of which: are the weight coefficients of the four networks respectively; is the real-time status of the corresponding underwater actuator; To perform an action.

[0022] The operation task includes the three-dimensional coordinates of the operation position and the operation content.

[0023] The parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 at the unmanned ship end and the distributed central processing unit 2 at the underwater actuator end described in step S1 are independent of each other; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 at each underwater actuator end are shared.

[0024] The current state described in step S3 is the coordinate position of all underwater actuators, and is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 of the unmanned ship.

[0025] Furthermore, the reward for the constraint condition described in step S5 is:

[0026] Where J is the total number of underwater actuators; I is the total number of operation tasks in the previous operation cycle T, T j is the average waiting time of the jth underwater actuator’s historical execution task, D j is the average power consumption per unit time of the jth underwater actuator in the history of executing the operation task, α1, α2, α3 are artificially set hyper parameters, L ij The distance traveled by the j-th underwater actuator during its operation cycle to perform task i.

[0027] Further, the step S6 is specifically as follows:

[0028] S601: Selecting a number of state-action transition pairs (transition) from the experience buffer according to the current states of all underwater actuators;

[0029] S602: Calculate the reward of the constraint condition according to the task allocation action of the state-action transfer pair and the current state of all underwater actuators;

[0030] S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 of the unmanned ship are updated and iterated.

[0031] Furthermore, the update iteration described in step S603 is specifically as follows:

[0032] S6031: Using loss function Update the critic network Q(s,a|θ Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transition pairs, λ is a hyperparameter, and t is the corresponding current time s of the experience buffer t The subscript of the state-action transition pair;

[0033] S6032: Update the actor network μ(s|θ) using the gradient strategy μ )’s network parameters θ μ ;

[0034] S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0035] Furthermore, the real-time state of an underwater actuator in step S7 obtained by the state perception sensor in real time includes: the real-time position [x1, x2, x3] of the underwater actuator obtained by the positioning sensor, the real-time speed [x1′, x2′, x3′] and real-time acceleration [x1″, x2″, x3″] of the underwater actuator obtained by the inertial sensor, and the coordinates of obstacles detected by the sonar and infrared camera. Where j is the number of obstacles.

[0036] In particular, the coordinates of obstacles detected by sonar and infrared cameras are existing mature technologies and can be directly implemented by calling existing algorithms.

[0037] Preferably, the depth sensor can assist the positioning sensor in correcting the coordinate position of the underwater actuator.

[0038] Furthermore, the brachistocentre lines in three directions as well as the gravitational field and the repulsive field are established according to the variational principle, and the reward for constructing the physical information described in step S8 is:

[0039]

[0040] Among them, β1, β2, and β3 are artificially set hyperparameters; is the brachistocentric formula, is the gravitational field formula, is the repulsive field formula, is the three-dimensional coordinate of the working position.

[0041] Further, the step S6 is specifically as follows:

[0042] S901: Selecting a number of state-action transfer pairs from the experience buffer according to the current state of the underwater actuator corresponding to the operation task;

[0043] S902: Calculate the reward of physical information according to the execution action of the state-action transfer pair and the current state of the corresponding underwater actuator;

[0044] S903: Using the reward of physical information, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 of the corresponding underwater actuator are updated and iterated.

[0045] In particular, the updating process of step S903 is the same as that of step S603.

[0046] The beneficial effects of the present invention are: providing an unmanned ship supporting multiple underwater actuators, realizing large-area operations and multi-target operations of the linkage of multiple underwater actuators; at the same time, providing a collaborative control method for an unmanned ship supporting multiple underwater actuators, by converting multi-level and multi-target optimization problems into intelligent agents for solving in a two-layer reinforcement learning network, while realizing the organic integration of constraints and physical information, and solving the problem of automatic, optimized and collaborative control of multiple underwater actuators in an unsupervised complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0048] Figure 1 This is a structural diagram of an unmanned ship supporting multiple underwater actuators in Example 1 of the present invention; wherein 1 is an unmanned ship, 2 is an underwater actuator, and 3 is a cable;

[0049] Figure 21 is a schematic diagram of the unmanned boat in Embodiment 1 of the present invention; 11 is the boat body, and the arrow direction is the signal flow direction;

[0050] Figure 3 : is a structural diagram of the underwater actuator in Example 1 of the present invention; wherein the direction of the arrow is the direction of the signal flow;

[0051] Figure 4 This is a flow chart of a collaborative control method for an unmanned ship supporting multiple underwater actuators in Embodiment 2 of the present invention;

[0052] Figure 5 This is a diagram of the DDPG deep reinforcement learning network architecture based on the Actor-Critic framework in Example 2 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose and technical solution of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] Example 1: Based on the regional water quality images taken by satellite, it is found that a large number of cyanobacteria cells in the water body suddenly grow and reproduce to form algae blooms. After analysis, it is now necessary to place copper sulfate at multiple points in the region to control the cyanobacteria blooms. In view of this scenario, the present invention provides "an unmanned boat supporting multiple underwater actuators".

[0055] Combination Figure 1 , consisting of an unmanned ship (1), N underwater actuators (2), and cables (3); the unmanned ship (1) and the single underwater actuator (2) are connected via the cables (3) to achieve communication and power supply.

[0056] Combination Figure 2The unmanned boat (1) is composed of a hull (11), and a power system (12), a positioning module (13), an image sensor (14), a power supply (15), a communication module 1 (16) and a distributed central processing unit 1 (17) installed on the hull (11); the power system (12), the positioning module (13), the image sensor (14), and the communication module 1 (16) are respectively connected to the distributed central processing unit 1 (17); the power supply (15) is respectively connected to the power system (12), the positioning module (13), the image sensor (14), the communication module 1 (16), the distributed central processing unit 1 (17), and the cable (3) for power supply; the power system (12) is an engine; the positioning system (13) includes a GPS positioning system and a Beidou positioning system; the image sensor (14) is a camera; the communication module 1 (16) includes a wired communication module and a wireless communication module, wherein the wired communication module is connected to the cable (3), and the wireless communication module is connected to the client; the distributed central processing unit 1 (17) is a microprocessor.

[0057] Combination Figure 3 Any one of the underwater actuators (2) is composed of a state perception sensor (21), an actuator (22), a communication module 2 (23) and a distributed central processing unit 2 (24); the state perception sensor (21), the actuator (22) and the communication module 2 (23) are respectively connected to the distributed central processing unit 2 (24); the state perception sensor (21) is composed of an inertial sensor, a positioning sensor, a depth sensor, a sonar and an infrared camera; the sonar module is a multi-beam centimeter-level ultra-high resolution sonar, which can be used for high-definition underwater terrain detection collected by unmanned ships; the inertial sensor is a gyroscope; the communication module 2 (23) is a wired communication module connected to the cable (3); the actuator (22) is a motor-driven mechanism for driving and copper sulfate delivery operations; the distributed central processing unit 2 (24) is a microprocessor.

[0058] When working, the unmanned boat (1) is driven by the power system (12), assisted by the positioning module (13) and the image sensor (14), and travels to the designated area. Then, all the underwater actuators (2) are deployed, and the underwater actuators (2) are connected through cables (3) to achieve coordinated control and power supply. The underwater actuators (2) are driven by the actuator device (22), assisted by the state perception sensor (21), and after traveling to the designated location, copper sulfate is deployed, and the unmanned boat (1) is returned after completion.

[0059] Example 2: Based on the scenario and system of Example 1, in order to achieve automatic, optimized and collaborative control of multiple underwater actuators in an unsupervised complex environment, the present invention provides a "collaborative control method for an unmanned vessel supporting multiple underwater actuators".

[0060] Combination Figure 4 , including the following steps:

[0061] S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer in the distributed central processing unit 1 (17) at the unmanned ship (1) and the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer in the distributed central processing unit 2 (24) at the underwater actuator (2).

[0062] The DDPG deep reinforcement learning network based on the Actor-Critic framework adopts the Markov decision process as the gradient strategy, which includes a triple (S, A, R), where S is the perception state space of the virtual machine, A is the task scheduling action space of the virtual machine, and R is the reward.

[0063] Combination Figure 5 The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) is composed of an actor network μ(s|θ μ ), critic network Q(s,a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ ); where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s is the current state of all underwater actuators; a is the task scheduling action.

[0064] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator (2) end is composed of an actor network Critics Network Actor target network Critics target network constitute; of which: are the weight coefficients of the four networks respectively; is the real-time status of the corresponding underwater actuator; To perform an action.

[0065] The parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) end and the distributed central processing unit 2 (24) at the underwater actuator end (2) described in step S1 are independent of each other; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator end (2) are shared.

[0066] S2: The unmanned vessel (1) receives the client command, arrives at the operation area, and releases the underwater actuator; the operation task includes the three-dimensional coordinates of the operation location and the operation content.

[0067] S3: The unmanned ship (1) receives the operation task from the client, and issues the task to all underwater actuators (2) according to the high-definition underwater terrain and operation cycle T collected by the unmanned ship.

[0068] The current state is the coordinate position of all underwater actuators (2), and is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1).

[0069] S4: At each operation cycle T, all underwater actuators (2) obtain the current status through the state perception sensor (21) and feed it back to the distributed central processing unit 1 (17) at the unmanned ship (1).

[0070] S5: Based on the constraints of the current state, power consumption, and task waiting time, construct a constraint reward. The constraint reward is:

[0071] Where J is the total number of underwater actuators; I is the total number of operation tasks in the previous operation cycle T, T j is the average waiting time of the jth underwater actuator’s historical execution task, D j is the average power consumption per unit time of the jth underwater actuator in the history of executing the operation task, α1, α2, α3 are artificially set hyper parameters, L ij The distance traveled by the j-th underwater actuator during its operation cycle to perform task i.

[0072] S6: The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) generates the task assignment action and issues the instruction to the corresponding underwater actuator (2). Specifically:

[0073] S601: Selecting a number of state-action transition pairs (transition) from the experience buffer according to the current states of all underwater actuators (2);

[0074] S602: Calculate the constraint conditions according to the task allocation action of the state-action transition pair and the current state of all underwater actuators (2)

[0075] Rewards;

[0076] S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) are updated and iterated.

[0077] Furthermore, the update iteration described in step S603 is specifically as follows:

[0078] S6031: Using loss function Update the critic network Q(s,a|θ Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transition pairs, λ is a hyperparameter, and t is the corresponding current time s of the experience buffer t The subscript of the state-action transition pair;

[0079] S6032: Update the actor network μ(s|θ) using the gradient strategy μ )’s network parameters θ μ ;

[0080] S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0081] S7: In an operation cycle T, the corresponding underwater actuator (2) obtains the real-time status through the status perception sensor (21) in real time.

[0082] The real-time status of an underwater actuator obtained by the state perception sensor includes: the real-time position of the underwater actuator [x1, x2, x3] obtained by the positioning sensor, the real-time speed [x1′, x2′, x3′] and real-time acceleration [x1″, x2″, x3″] of the underwater actuator obtained by the inertial sensor, and the coordinates of obstacles detected by the sonar and infrared camera. Where j is the number of obstacles.

[0083] The coordinates of obstacles detected by sonar and infrared cameras are existing mature technologies, such as the YOLO network or the invention patent "A fixed-wing UAV trajectory planning method" (ZL201910169620), which can be directly implemented by calling existing algorithms.

[0084] The depth sensor can assist the positioning sensor in correcting the coordinate position of the underwater actuator (2), specifically by adding redundant equations and solving them using the least squares method.

[0085] S8: Based on the fastest path of the underwater actuator (2) constructed according to the variational principle and the obstacle distance, a reward for constructing physical information is constructed.

[0086] Specifically, the brachistocentre lines in three directions are established according to the variational principle, and then the reward of constructing the physical information described in step S8 by using the gravitational field and the repulsive field is:

[0087]

[0088] Among them, β1, β2, and β3 are artificially set hyperparameters; is the brachistocentric formula, is the gravitational field formula, is the repulsive field formula, is the three-dimensional coordinate of the working position.

[0089] S9: Using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the corresponding underwater actuator (2) generates an execution action and executes it through the execution device (22).

[0090] Specifically:

[0091] S901: Selecting a number of state-action transfer pairs from the experience buffer according to the current state of the underwater actuator (2) corresponding to the operation task;

[0092] S902: Calculate the reward of physical information according to the execution action of the state-action transfer pair and the current state of the corresponding underwater actuator (2);

[0094] S903: Using the reward of physical information, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the corresponding underwater actuator (2) are updated and iterated.

[0095] The update iteration described in step S903 is specifically as follows:

[0096] S9031: Using loss function Update the critic network Parameter θ Q ;in, N is the number of state-action transition pairs, is a hyperparameter, and t is the corresponding current time s of the experience buffer t The subscript of the state-action transition pair;

[0097] S9032: Updating actor networks using gradient strategies Network parameters

[0098] S9033: Update actor target network Critics target network Network parameters in, is a hyperparameter.

[0099] S10: Repeat steps S3 to S9 until all the tasks are completed.

[0100] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. An unmanned ship supporting multiple underwater actuators, characterized in that: The unmanned boat (1) is composed of an unmanned boat (1), a plurality of underwater actuators (2), and a cable (3); the unmanned boat (1) and a single underwater actuator (2) are connected via the cable (3) to realize communication and power supply; the unmanned boat (1) is composed of a hull (11), and a power system (12), a positioning module (13), an image sensor (14), a power supply (15), a communication module 1 (16), and a distributed central processing unit 1 (17) installed on the hull (11); the power system (12), the positioning module (13), the image sensor (14), and the communication module 1 (16) are respectively connected to the distributed central processing unit 1 (17); the power supply (15) is respectively connected to the power system (12), the positioning module (13), the image sensor (14), the communication module 1 (16), the distributed central processing unit 1 (17), and the cable (3) to provide power; the power system (12) is an engine; the positioning system includes a GPS positioning system and a Beidou positioning system; The communication module 1 (16) comprises a wired communication module and a wireless communication module, wherein the wired communication module is connected to the cable (3), and the wireless communication module is connected to the client; the distributed central processing unit 1 (17) is a microprocessor; any one of the underwater actuators (2) is composed of a state perception sensor (21), an actuator (22), a communication module 2 (23) and a distributed central processing unit 2 (24); the state perception sensor (21), the actuator (22) and the communication module 2 (23) are respectively connected to the distributed central processing unit 2 (24); the state perception sensor (21) is composed of an inertial sensor, a positioning sensor, a depth sensor, a sonar and an infrared camera; the inertial sensor is a gyroscope; the communication module 2 (23) is a wired communication module connected to the cable (3); the actuator (22) is a motor-driven mechanism for traveling and working; the distributed central processing unit 2 (24) is a microprocessor.

2. A collaborative control method for an unmanned ship supporting multiple underwater actuators as claimed in claim 1, characterized in that: The steps include: S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer (Replay Buffer) in the distributed central processing unit 1 (17) at the unmanned ship (1) and the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer in the distributed central processing unit 2 (24) at the underwater actuator (2); S2: The unmanned ship (1) receives the client command, arrives at the operation area, and releases the underwater actuator (2); S3: The unmanned ship (1) receives the operation task from the client and issues the task to all underwater actuators (2) according to the operation cycle T; S4: At each operation cycle T, all underwater actuators (2) obtain the current state through the state perception sensor (21) and feed it back to the distributed central processing unit 1 (17) at the unmanned ship (1); S5: construct a reward for the constraint conditions according to the constraints of the current state, power consumption of the power source (15), and task waiting time; S6: The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) generates the task assignment action and issues the instruction to the corresponding underwater actuator (2); S7: In an operation cycle T, the corresponding underwater actuator (2) obtains the real-time status through the status sensing sensor (21) in real time; S8: Based on the fastest path of the underwater actuator (2) constructed according to the variational principle and the distance from obstacles, construct the reward of physical information; S9: Using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the corresponding underwater actuator (2) generates an execution action and executes it through the execution device (22); S10: Repeat steps S3 to S9 until all the tasks are completed; The DDPG deep reinforcement learning network based on the Actor-Critic framework adopts the Markov Decision Process (MDP) as the gradient strategy, which includes a triple (S, A, R), where S is the perception state space of the virtual machine, A is the task scheduling action space of the virtual machine, and r is the reward; The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) is composed of an actor network μ(s|θ μ ), critic network Q(s, a|θ Q ), actor target network μ'(s|θ μ’ ), critic target network Q'(s, a|θ Q’ ); where: θ μ ,θ Q ,θμ ’ ,θ Q’ are the weight coefficients of the four networks respectively; s is the current state of all underwater actuators (2); a is the task scheduling action; The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator (2) end is composed of an actor network Critics Network Actor target network Critics target network constitute; of which: are the weight coefficients of the four networks respectively; is the real-time status of the corresponding underwater actuator (2); To perform an action; The operation task includes the three-dimensional coordinates of the operation position and the operation content; The current state described in step S3 is the coordinate position of all underwater actuators (2), and is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1); The reward for the constraint condition described in step S5 is: Where J is the total number of underwater actuators (2); I is the total number of operation tasks in the previous operation cycle T, T j is the average waiting time of the jth underwater actuator (2) in the history of executing the task, D j is the average power consumption per unit time of the jth underwater actuator (2) in the history of executing the operation task, α1, α2, α3 are artificially set hyper parameters, L ij is the travel distance of the jth underwater actuator (2) during the operation cycle to perform task i; The real-time state of an underwater actuator (2) in step S7 obtained in real time by the state perception sensor (21) includes: the real-time position [x1, x2, x3] of the underwater actuator (2) obtained by the positioning sensor, the real-time speed [x1′, x2′, x3′] and real-time acceleration [x1″, x2″, x3″] of the underwater actuator (2) obtained by the inertial sensor, and the coordinates of obstacles detected by the sonar and infrared camera. Where j is the number of obstacles; The reward of the physical information described in step S8 is: Among them, β1, β2, and β3 are artificially set hyperparameters; is the brachistocentric formula, is the gravitational field formula, is the repulsive field formula, is the three-dimensional coordinate of the working position.

3. The method for cooperative control of an unmanned ship supporting multiple underwater actuators according to claim 2, characterized in that: The parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) end and the distributed central processing unit 2 (24) at the underwater actuator (2) end described in step S1 are independent of each other; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator (2) end are shared.

4. The method for cooperative control of an unmanned ship supporting multiple underwater actuators according to claim 2, characterized in that: The step S6 is specifically as follows: S601: Selecting a number of state-action transition pairs (transition) from the experience buffer according to the current states of all underwater actuators (2); S602: Calculate the reward of the constraint condition according to the task allocation action of the state-action transition pair and the current state of all underwater actuators (2); S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) are updated and iterated.

5. The method for cooperative control of an unmanned ship supporting multiple underwater actuators according to claim 4, characterized in that: The update iteration described in step S603 is specifically as follows: S6031: Using loss function Update the critic network Q(s,a|θ Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transfer pairs, λ is a hyperparameter, and t is the corresponding current time s of the experience buffer t The subscript of the state-action transition pair; S6032: Update the actor network μ(s|θ) using the gradient strategy μ )’s network parameters θ μ ; S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

6. The method for cooperative control of an unmanned ship supporting multiple underwater actuators according to claim 2, characterized in that: The depth sensor can assist the positioning sensor in correcting the coordinate position of the underwater actuator (2).

Images