An unmanned ship supporting multiple underwater actuators and a cooperative control method thereof

Abstract

The application relates to an unmanned ship supporting multiple underwater actuators and a cooperative control method thereof, and belongs to the fields of artificial intelligence and unmanned ships. The system is composed of an unmanned ship, multiple underwater actuators and a cable. The method comprises the following steps: S1, initializing a deep reinforcement learning network; S2, the unmanned ship reaches a work area to release the underwater actuators; S3, the unmanned ship receives a work task; S4, the underwater actuators acquire a current state and feed back to the unmanned ship; S5, a reward of a constraint condition is constructed; S6, a work task distribution action is generated and is issued to the underwater actuators; S7, the underwater actuators acquire a real-time state; S8, a reward of physical information is constructed; S9, an execution action is generated and is executed through an execution device; and S10, steps S3-S9 are repeated until all work tasks are completed. The method can realize organic fusion of constraint conditions and physical information, and solve automatic, optimized and cooperative control of the multiple underwater actuators in an unsupervised complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to an unmanned ship supporting multiple underwater actuators and a cooperative control method thereof, and belongs to the field of artificial intelligence and unmanned ships, in particular to an unmanned ship supporting multiple underwater actuators and a cooperative control method thereof. BACKGROUND

[0002] With the demand for ecological monitoring and improvement of rivers and lakes, unmanned operation ships are increasingly widely used in the ecological maintenance of rivers and lakes. For example, surveying underwater topography, disposing underwater sediments, and throwing chemicals to control cyanobacterial blooms.

[0003] However, the current unmanned ship mainly has a single actuator on the water surface, and its operation range is small and its efficiency is low. However, when the existing unmanned operation ship performs complex underwater tasks, it often needs multiple underwater actuators controlled independently. These actuators lack effective cooperative operation mechanisms, and complex and variable water scenarios can lead to low operation efficiency and increased costs.

[0004] With the development of artificial intelligence technology, methods based on reinforcement learning can learn directly from experience without prior knowledge of the work scene, have good generalization and adaptability, and are expected to be used to realize the coordinated control of unmanned ships with multiple underwater actuators. At the same time, during the entire operation process, the constraints related to underwater dynamic obstacles and power consumption also need to be considered, and the efficiency and cost of the operation also need to be taken into account. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide an unmanned ship supporting multiple underwater actuators and a cooperative control method thereof. The intention is to construct a reinforcement learning network with physical information based on the principle of variation, introduce environmental constraints and self-constraints, convert the cooperative control problem of the unmanned ship with multiple underwater actuators into a solution problem of an optimization model, and ultimately realize the automatic, optimized, and cooperative control of multiple underwater actuators in an unsupervised complex environment.

[0006] To achieve the above purpose, the present application provides the following technical solutions:

[0007] The application discloses an unmanned ship supporting multiple underwater actuators, which is characterized by comprising an unmanned ship, multiple underwater actuators and a cable; the unmanned ship and the single underwater actuator are connected through the cable to realize communication and power supply; the unmanned ship comprises a ship body, a power system, a positioning module, an image sensor, a power supply, a communication module 1 and a distributed central processing unit 1 which are installed on the ship body; the power system, the positioning module, the image sensor and the communication module 1 are connected with the distributed central processing unit 1 respectively; the power supply is connected with the power system, the positioning module, the image sensor, the communication module 1, the distributed central processing unit 1 and the cable to supply power; the power system is a motor; the positioning system comprises a GPS positioning system and a Beidou positioning system; the communication module 1 comprises a wired communication module and a wireless communication module, wherein the wired communication module is connected with the cable, and the wireless communication module is connected with a client; the distributed central processing unit 1 is a microprocessor; any underwater actuator comprises a state sensing sensor, an execution device, a communication module 2 and a distributed central processing unit 2; the state sensing sensor, the execution device and the communication module 2 are connected with the distributed central processing unit 2 respectively; the state sensing sensor comprises an inertial sensor, a positioning sensor, a depth sensor, a sonar and an infrared camera; the inertial sensor is a gyroscope; the communication module 2 is a wired communication module and is connected with the cable; the execution device is a motor-driven mechanism for driving and operation; the distributed central processing unit 2 is a microprocessor.

[0008] A cooperative control method of an unmanned ship supporting multiple underwater actuators, comprising the following steps:

[0009] S1: initializing an DDPG deep reinforcement learning network based on an Actor-Critic framework and an experience buffer (Replay Buffer) in the distributed central processing unit 1 of the unmanned ship end and an DDPG deep reinforcement learning network based on an Actor-Critic framework and an experience buffer in the distributed central processing unit 2 of the underwater actuator end;

[0010] S2: the unmanned ship receives a client instruction to reach a work area and releases the underwater actuators;

[0011] S3: the unmanned ship receives a work task of the client and issues the task to all underwater actuators according to a work cycle T;

[0012] S4: at each work cycle T moment, all underwater actuators acquire a current state through a state sensing sensor and feed back to the distributed central processing unit 1 of the unmanned ship end;

[0013] S5: constructing a reward of the constraint condition according to the current state, power consumption, and constraint condition of task waiting time length;

[0014] S6: the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 at the unmanned ship end generates a task assignment action and issues an instruction to the corresponding underwater execution mechanism;

[0015] S7: in one work cycle T, the corresponding underwater execution mechanism acquires real-time state through a state perception sensor in real time;

[0016] S8: constructing a reward of physical information based on the steepest path of the underwater execution mechanism and the obstacle distance constructed based on the variational principle;

[0017] S9: using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 at the corresponding underwater execution mechanism end generates an execution action and executes through an execution device;

[0018] S10: repeating steps S3-S9 until all work tasks are completed.

[0019] The DDPG deep reinforcement learning network based on the Actor-Critic framework adopts Markov Decision Process (MDP) as a gradient strategy and contains a triple (S, A, R), wherein S is a perception state space of a virtual machine, A is a virtual machine task scheduling action space, and R is a reward.

[0020] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 at the unmanned ship end is composed of an actor network μ(s|θ μ ), a critic network Q(s, a|θ Q ), an actor target network μ'(s|θ μ′ ), and a critic target network Q'(s, a|θ Q′ ); wherein θ μ , θ Q , θ μ′ , and θ Q′ are weight coefficients of the four networks, respectively; s is a current state of all underwater execution mechanisms; and a is a task scheduling action.

[0021] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 at the underwater execution mechanism end is composed of an actor network critic network actor target network critic target network constituted; wherein: are weight coefficients of the four networks respectively; is a real-time state of a corresponding underwater actuator; is an executed action.

[0022] The operation task contains an operation position three-dimensional coordinate and operation content.

[0023] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 at the unmanned ship end and the distributed central processing unit 2 at the underwater actuator end in step S1 are mutually parameter-independent; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in each distributed central processing unit 2 at the underwater actuator end are shared.

[0024] The current state in step S3 is the coordinate position of all underwater actuators, which is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 at the unmanned ship end.

[0025] Further, the reward of the constraint condition in step S5 is:

[0026] wherein J is the total number of underwater actuators; I is the total number of operation tasks in the last operation period T, T j is the average waiting time length of the historical operation task of the jth underwater actuator, D j is the average power consumption per unit time of the historical operation task of the jth underwater actuator, α1, α2, α3 are artificial set hyperparameters, L ij is the driving distance of the jth underwater actuator in the operation period T to execute the operation task i.

[0027] Further, the step S6 is specifically:

[0028] S601: selecting a plurality of state-action transition pairs from the experience buffer according to the current state of all underwater actuators;

[0029] S602: calculating the reward of the constraint condition according to the task assignment action of the state-action transition pair and the current state of all underwater actuators;

[0030] S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 of the unmanned ship are updated and iterated.

[0031] Furthermore, the update iteration described in step S603 is specifically as follows:

[0032] S6031: Using loss function Update the critic network Q(s,a|θ) Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transfer pairs, λ is a hyperparameter, and t is the corresponding current moment s in the experience buffer t The subscript of the state-action transition pair;

[0033] S6032: Update the actor network μ(s|θ using the gradient strategy μ )’s network parameters θ μ ;

[0034] S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0035] Furthermore, the real-time status of an underwater actuator obtained by the state perception sensor in step S7 includes: the real-time position of the underwater actuator [x1, x2, x3] obtained by the positioning sensor, the real-time speed [x1′, x2′, x3′] and real-time acceleration [x1″, x2″, x3″] of the underwater actuator obtained by the inertial sensor, and the coordinates of the obstacle detected by the sonar and infrared camera. Where j is the number of obstacles.

[0036] In particular, the obstacle coordinates detected by sonar and infrared cameras are existing mature technologies and can be directly implemented by calling existing algorithms.

[0037] Preferably, the depth sensor can assist the positioning sensor in correcting the coordinate position of the underwater actuator.

[0038] Further, according to the variational principle, the steepest descent lines in three directions are established, and the gravity field and the repulsive field are established, and the reward of the physical information in step S8 is:

[0039]

[0040] Wherein, β1, β2, β3 are artificial super parameters; is the steepest descent line formula, is the gravity field formula, is the repulsive field formula, is the three-dimensional coordinate of the work position.

[0041] Further, the step S6 is specifically:

[0042] S901: selecting a plurality of state-action transition pairs from the experience buffer according to the current state of the underwater execution mechanism corresponding to the work task;

[0043] S902: calculating the reward of the physical information according to the execution action of the state-action transition pair and the current state of the corresponding underwater execution mechanism;

[0044] S903: updating and iterating the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 at the end of the corresponding underwater execution mechanism by using the reward of the physical information.

[0045] In particular, the updating process of step S903 is the same as step S603.

[0046] The present application has the beneficial effects that: a kind of unmanned ship of supporting multiple underwater execution mechanisms is provided, multiple underwater execution mechanisms linkage is realized Large-scale operation, multi-target operation;While providing a kind of collaborative control method of unmanned ship of supporting multiple underwater execution mechanisms, by converting multi-level multi-objective optimization problem into the agent of two-layer reinforcement learning network Intelligent solution, simultaneously realize the organic integration of constraint condition and physical information, solve the automatic, optimization, collaborative control of multiple underwater execution mechanisms in unattended complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to make the purpose, technical scheme of the present application more clear, the present application provides the following drawings for illustration:

[0048] Figure 1 It is the architecture diagram of a kind of unmanned ship of supporting multiple underwater execution mechanisms in embodiment 1 of the present application;Wherein, 1 is unmanned ship, 2 is underwater execution mechanism, 3 is cable;

[0049] Figure 2It is the framework diagram of unmanned ship in embodiment 1 of the present application; wherein, 11 is a ship body, and the arrow direction is the signal flow direction;

[0050] Figure 3 It is the framework diagram of underwater actuator in embodiment 1 of the present application; wherein, the arrow direction is the signal flow direction;

[0051] Figure 4 It is the flow chart of a kind of collaborative control method of unmanned ship supporting multiple underwater actuators in embodiment 2 of the present application;

[0052] Figure 5 It is the DDPG deep reinforcement learning network architecture diagram based on Actor-Critic framework in embodiment 2 of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose and technical scheme of the present application clearer and more clear, the present application will be described in detail below in combination with the drawings and embodiments.

[0054] Embodiment 1: According to the regional water quality image taken by satellite, it is found that the blue-green algae cells in the water body grow and reproduce in large quantities to form water bloom, and after analysis, it is necessary to put copper sulfate into multiple points in the region to control blue-green algae water bloom. In view of this scene, the present application provides "a kind of unmanned ship supporting multiple underwater actuators".

[0055] In combination with Figure 1 , unmanned ship (1), N underwater actuators (2), cable (3) are constituted; the unmanned ship (1) and single underwater actuator (2) are connected to realize communication and power supply through cable (3).

[0056] In combination with Figure 2The unmanned ship (1) is composed of a ship body (11), a power system (12), a positioning module (13), an image sensor (14), a power supply (15), a communication module 1 (16) and a distributed central processing unit 1 (17) installed on the ship body (11); the power system (12), the positioning module (13), the image sensor (14) and the communication module 1 (16) are respectively connected with the distributed central processing unit 1 (17); the power supply (15) is respectively connected with the power system (12), the positioning module (13), the image sensor (14), the communication module 1 (16), the distributed central processing unit 1 (17) and the cable (3) for power supply; the power system (12) is an engine; the positioning system (13) comprises a GPS positioning system and a Beidou positioning system; the image sensor (14) is a camera; the communication module 1 (16) comprises a wired communication module and a wireless communication module, wherein the wired communication module is connected with the cable (3) and the wireless communication module is connected with the client; and the distributed central processing unit 1 (17) is a microprocessor.

[0057] In combination Figure 3 Any one of the underwater execution mechanism (2) is composed of a state sensing sensor (21), an execution device (22), a communication module 2 (23) and a distributed central processing unit 2 (24); the state sensing sensor (21), the execution device (22) and the communication module 2 (23) are respectively connected with the distributed central processing unit 2 (24); the state sensing sensor (21) is composed of an inertial sensor, a positioning sensor, a depth sensor, a sonar and an infrared camera; the sonar module is a multi-beam centimeter-level ultrahigh resolution sonar, which can be used for high-definition underwater terrain detection collected by the unmanned ship; the inertial sensor is a gyroscope; the communication module 2 (23) is a wired communication module connected with the cable (3); the execution device (22) is a motor-driven mechanism for driving and sulfuric acid copper throwing operation; and the distributed central processing unit 2 (24) is a microprocessor.

[0058] When working, the unmanned ship (1) is driven by the power system (12) and assisted by the positioning module (13) and the image sensor (14) to travel to the designated area. Then all the underwater execution mechanisms (2) are thrown and connected through the cable (3) to realize the cooperative control and power supply of the underwater execution mechanisms (2). The underwater execution mechanisms (2) are driven by the execution device (22) and assisted by the state sensing sensor (21) to travel to the designated place to throw sulfuric acid and return to the unmanned ship (1) after the operation is completed.

[0059] Embodiment 2: Based on the scene and system of Embodiment 1, in order to realize automatic, optimized and collaborative control of multiple underwater actuators in an unsupervised complex environment, the present application provides a "collaborative control method of an unmanned ship supporting multiple underwater actuators".

[0060] In combination Figure 4 , comprising the following steps:

[0061] S1: initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) end and the experience buffer, and initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) of the underwater actuator (2) end and the experience buffer.

[0062] The DDPG deep reinforcement learning network based on the Actor-Critic framework adopts a Markov decision process as a gradient policy, and contains a triple (S, A, R), wherein S is a perception state space of a virtual machine, A is a virtual machine task scheduling action space, and R is a reward.

[0063] In combination Figure 5 , the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) end is composed of an actor network μ(s|θ μ ), a critic network Q(s, a|θ Q ), an actor target network μ'(s|θ μ′ ), and a critic target network Q'(s, a|θ Q′ ); wherein: θ μ , θ Q , θ μ′ , θ Q′ are weight coefficients of the four networks, respectively; s is the current state of all underwater actuators; and a is a task scheduling action.

[0064] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) of the underwater actuator (2) end is composed of an actor network critic network actor target network critic target network ; wherein: are weight coefficients of the four networks, respectively; is the real-time state of the corresponding underwater actuator; is an execution action.

[0065] The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) end and the distributed central processing unit 2 (24) at the underwater actuator end (2) are mutually parameter-independent; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in each distributed central processing unit 2 (24) at the underwater actuator (2) end are shared.

[0066] S2: The unmanned ship (1) receives a client instruction to reach a work area and releases underwater actuators; the work task includes a work position three-dimensional coordinate and work content.

[0067] S3: The unmanned ship (1) receives a work task from a client, and issues the task to all underwater actuators (2) according to the high-definition underwater terrain collected by the unmanned ship and a work period T.

[0068] The current state is the coordinate position of all underwater actuators (2), and is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) end.

[0069] S4: At each work period T moment, all underwater actuators (2) obtain the current state through the state perception sensor (21) and feed back to the distributed central processing unit 1 (17) at the unmanned ship (1) end.

[0070] S5: According to the constraint conditions of the current state, power consumption, and task waiting time, a reward of the constraint condition is constructed. The reward of the constraint condition is:

[0071] Wherein, J is the total number of underwater actuators; I is the total number of work tasks in the last work period T, T j is the average waiting time of the historical work tasks of the jth underwater actuator, D j is the average power consumption per unit time of the historical work tasks of the jth underwater actuator, α1, α2, α3 are artificial set hyperparameters, L ij is the driving distance of the jth underwater actuator to execute the work task i in the work period.

[0072] S6: The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned ship (1) end generates a work task allocation action and issues an instruction to the corresponding underwater actuator (2). Specifically:

[0073] S601: Selecting a number of state-action transition pairs from the experience buffer according to the current states of all underwater actuators (2);

[0074] S602: Calculate the constraints according to the task assignment actions of the state-action transition pairs and the current states of all underwater actuators (2).

[0075] rewards;

[0076] S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) are updated and iterated.

[0077] Furthermore, the update iteration described in step S603 is specifically as follows:

[0078] S6031: Using loss function Update the critic network Q(s,a|θ) Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transfer pairs, λ is a hyperparameter, and t is the corresponding current moment s in the experience buffer t The subscript of the state-action transition pair;

[0079] S6032: Update the actor network μ(s|θ using the gradient strategy μ )’s network parameters θ μ ;

[0080] S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0081] S7: In one operation cycle T, the corresponding underwater actuator (2) obtains the real-time status through the status perception sensor (21) in real time.

[0082] The real-time state obtained by the underwater actuator through the state sensing sensor includes: the real-time position [x1, x2, x3] of the underwater actuator obtained by the positioning sensor, the real-time speed [x1', x2', x3'] and the real-time acceleration [x1'', x2'', x3''] of the underwater actuator obtained by the inertial sensor, and the obstacle coordinates detected by the sonar and infrared camera Where j is the number of obstacles.

[0083] The obstacle coordinates detected by the sonar and infrared camera are existing mature technologies, such as YOLO network or the invention patent "a fixed-wing unmanned aerial vehicle trajectory planning method" (ZL201910169620), which can directly call existing algorithms to realize.

[0084] The depth sensor can assist the positioning sensor to correct the coordinate position of the underwater actuator (2), specifically by adding a redundant equation and solving it by the least square method.

[0085] S8: Based on the variational principle, the steepest path of the underwater actuator (2) and the distance of the obstacle are constructed, and the reward of the physical information is constructed.

[0086] Specifically, the steepest descent line in three directions is established according to the variational principle, and then the gravitational field and the repulsive field are used to construct the reward of the physical information in step S8.

[0087]

[0088] Where β1, β2, β3 are artificial set hyperparameters; is the steepest descent line formula, is the gravitational field formula, is the repulsive field formula, is the three-dimensional coordinate of the work position.

[0089] S9: Using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the corresponding underwater actuator (2) end generates an execution action and executes through the execution device (22).

[0090] Specifically,

[0091] S901: Selecting a plurality of state-action transition pairs from the experience buffer according to the current state of the underwater actuator (2) corresponding to the work task;

[0092] S902: According to the execution action of the state-action transition pair and the current state of the corresponding underwater actuator (2), the reward of the physical information is calculated

[0093] ;

[0094] S903: Update iteration of the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the end of the corresponding underwater executive mechanism (2) using the reward of the physical information.

[0095] The update iteration in step S903 is specifically:

[0096] S9031: Update iteration of the parameters of the critic network using the loss function Update the critic network Parameters θ Q ; wherein, N is the number of state-action transition pairs, is a hyperparameter, t is the index of the corresponding current time s t of the state-action transition pair in the experience buffer;

[0097] S9032: Update the actor network using the gradient policy Network parameters of the actor network

[0098] S9033: Update the actor target network Network parameters of the critic target network Wherein, is a hyperparameter.

[0099] S10: Repeat steps S3-S9 until all job tasks are completed.

[0100] Finally, it should be pointed out that the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present application.​

Claims

1. A collaborative control method for an unmanned vessel supporting multiple underwater actuators, characterized in that: The invention is realized by an unmanned ship supporting multiple underwater actuators, wherein the unmanned ship supporting multiple underwater actuators comprises an unmanned ship (1), multiple underwater actuators (2), and a cable (3); the unmanned ship (1) and a single underwater actuator (2) are connected via a cable (3) to realize communication and power supply; the unmanned ship (1) comprises a hull (11), and a power system (12), a positioning module (13), an image sensor (14), a power supply (15), a communication module 1 (16), and a distributed central processing unit 1 (17) installed on the hull (11); the power system (12), the positioning module (13), the image sensor (14), and the communication module 1 (16) are respectively connected to the distributed central processing unit 1 (17); the power supply (15) is respectively connected to the power system (12), the positioning module (13), the image sensor (14), the communication module 1 (16), the distributed central processing unit 1 (17), and the cable (3) to provide power; the power system (12) is an engine; the positioning module (13) is a plurality of underwater actuators (2), and a plurality of underwater actuators (2) are connected to the cable (3) to realize communication and power supply ... The module includes a GPS positioning system and a Beidou positioning system; the communication module 1 (16) includes a wired communication module and a wireless communication module, wherein the wired communication module is connected to the cable (3) and the wireless communication module is connected to the client; the distributed central processing unit 1 (17) is a microprocessor; any one of the underwater actuators (2) is composed of a state perception sensor (21), an actuator (22), a communication module 2 (23) and a distributed central processing unit 2 (24); the state perception sensor (21), the actuator (22) and the communication module 2 (23) are respectively connected to the distributed central processing unit 2 (24); the state perception sensor (21) is composed of an inertial sensor, a positioning sensor, a depth sensor, a sonar and an infrared camera; the inertial sensor is a gyroscope; the communication module 2 (23) is a wired communication module connected to the cable (3); the actuator (22) is a motor-driven mechanism for traveling and working; the distributed central processing unit 2 (24) is a microprocessor; Specifically, the method comprises the following steps: S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer (Replay Buffer) in the distributed central processing unit 1 (17) on the unmanned ship (1) and the DDPG deep reinforcement learning network based on the Actor-Critic framework and the experience buffer in the distributed central processing unit 2 (24) on the underwater actuator (2); S2: The unmanned vessel (1) receives the client's instruction and arrives at the operation area, and releases the underwater actuator (2); S3: The unmanned vessel (1) receives the client's task and issues the task to all underwater actuators (2) according to the task cycle T; S4: At each operation cycle T, all underwater actuators (2) obtain the current status through the state perception sensor (21) and feed it back to the distributed central processing unit 1 (17) on the unmanned ship (1); S5: Construct a reward based on the constraints of the current state, power consumption of the power supply (15), and task waiting time; S6: The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) generates the task assignment action and issues the instruction to the corresponding underwater actuator (2); S7: Within an operation cycle T, the corresponding underwater actuator (2) obtains the real-time status through the status perception sensor (21); S8: Based on the fastest path of the underwater actuator (2) constructed by the variational principle and the distance to the obstacle, construct the reward of physical information; S9: Using the physical information reward, the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the corresponding underwater actuator (2) generates an execution action and executes it through the execution device (22); S10: Repeat steps S3 to S9 until all tasks are completed; Furthermore, the DDPG deep reinforcement learning network based on the Actor-Critic framework adopts the Markov Decision Process (MDP) as the gradient strategy, which includes a triple (S, A, R), where S is the perception state space of the virtual machine, A is the virtual machine task scheduling action space, and R is the reward; The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) is composed of the actor network μ(s|θ μ ), critic network Q(s,a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ ) is composed of; where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s is the current state of all underwater actuators (2); a is the task scheduling action; The DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator (2) is composed of an actor network critic network actor target network critic target network constitute; of which: are the weight coefficients of the four networks respectively; is the real-time status of the corresponding underwater actuator (2); To perform an action; The operation task includes the three-dimensional coordinates of the operation position and the operation content; The current state described in step S3 is the coordinate position of all underwater actuators (2), and is also the state of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) on the unmanned ship (1); The reward for the constraint condition described in step S5 is: Where J is the total number of underwater actuators (2); I is the total number of operation tasks in the previous operation cycle T, T j is the average waiting time of the jth underwater actuator (2) in the history of executing the task, D j is the average power consumption per unit time of the jth underwater actuator (2) in the history of executing the operation task, α1, α2, α3 are artificially set hyper parameters, L ij The distance traveled by the jth underwater actuator (2) during the operation cycle to perform task i; The real-time state of the underwater actuator (2) in step S7 obtained by the state perception sensor (21) in real time includes: the real-time position [x1, x2, x3] of the underwater actuator (2) obtained by the positioning sensor, the real-time speed [x1′, x2′, x3′] and real-time acceleration [x1″, x2″, x3″] of the underwater actuator (2) obtained by the inertial sensor, and the coordinates of the obstacles detected by the sonar and infrared camera. Where j is the number of obstacles; The reward for the physical information in step S8 is: Among them, β1, β2, and β3 are artificially set hyperparameters; is the brachistochrone formula, is the gravitational field formula, is the repulsive field formula, is the three-dimensional coordinate of the working position.

2. The method for cooperative control of an unmanned vessel supporting multiple underwater actuators according to claim 1, characterized in that: The parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) at the unmanned boat (1) end and the distributed central processing unit 2 (24) at the underwater actuator (2) end described in step S1 are independent of each other; the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 2 (24) at the underwater actuator (2) end are shared.

3. The method for cooperative control of an unmanned vessel supporting multiple underwater actuators according to claim 1, characterized in that: The steps described S6 specifically: S601: Selecting a number of state-action transition pairs from the experience buffer according to the current states of all underwater actuators (2); S602: Calculate the reward of the constraint condition according to the task allocation action of the state-action transition pair and the current state of all underwater actuators (2); S603: Using the reward of the constraint condition, the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework in the distributed central processing unit 1 (17) of the unmanned ship (1) are updated and iterated.

4. The method for cooperative control of an unmanned vessel supporting multiple underwater actuators according to claim 3, characterized in that: The update iteration described in step S603 is specifically as follows: S6031: Using loss function Update the critic network Q(s,a|θ Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ), N is the number of state-action transfer pairs, λ is a hyperparameter, and t is the corresponding current moment s in the experience buffer t The subscript of the state-action transition pair; S6032: Update the actor network μ(s|θ using the gradient strategy μ )’s network parameters θ μ ; S6033: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s,a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

5. The method for cooperative control of an unmanned vessel supporting multiple underwater actuators according to claim 1, characterized in that: The depth sensor can assist the positioning sensor in correcting the coordinate position of the underwater actuator (2).

Images