A collaborative hunting method of multiple unmanned boats based on deep reinforcement learning algorithm with channel attention mechanism

Abstract

The present invention discloses a multi-unmanned boat collaborative capture method based on a channel attention mechanism deep reinforcement learning algorithm, comprising the following steps: establishing a maritime unmanned boat capture game environment, determining the dynamic model and observation space of the capture boat, and the escape strategy of the escape boat defines the conditions for successful capture; fixing the dimension of the observation space of the unmanned boat to the friendly party based on the channel attention network; constructing a multi-agent deep deterministic policy gradient algorithm framework, and designing the reward function of distance collaboration and angle collaboration in combination with the capture problem; using a centralized training and distributed execution training framework to generate a multi-unmanned boat collaborative capture strategy. The collaborative capture strategy obtained by the present invention takes into account the influence of the number of different capture boats, and can adapt to the future changes in the capture environment of an uncertain number. At the same time, the reasonable design of the monomer collaborative reward function can improve the success rate of the capture task.

Description

Technical Field

[0001] The present invention relates to the field of collaborative capture control of multiple unmanned boats, and in particular to a collaborative capture method of multiple unmanned boats based on a channel attention mechanism deep reinforcement learning algorithm. Background Art

[0002] With the rapid development of research on unmanned platforms at home and abroad, unmanned boats, as a small unmanned mission platform at sea, have the characteristics of high speed, intelligence, flexibility and concealment, and have significant military application value; with the accelerated development of unmanned equipment, unmanned combat will become an important combat style in future wars. The ability of unmanned swarms to autonomously perform tasks is the key to improving their combat effectiveness; compared with a single unmanned boat, multiple unmanned boats have the characteristics of strong load capacity, wide coverage, and strong information processing capabilities. The problem of collaborative hunting by multiple unmanned boats is a hot issue in the research of multiple unmanned systems at home and abroad, and is of great significance in the military field.

[0003] Compared with the operation of a single unmanned boat at sea, there is still a lot of room for improvement in the coordinated tasks of multiple unmanned boats. The traditional control method does not take into account the complex dynamic environment changes at sea, and the efficiency of task completion is low. How to improve the decision-making ability of unmanned boat clusters in complex and changeable marine environments is a key factor affecting the popularization of unmanned boats in military applications. The coordinated capture of multiple unmanned boats mainly faces the following difficulties: First, the behavior of escaped targets at sea is relatively complex. The current decision-making in the game process between the two sides needs to fully consider the impact results of the previous and next stages. The traditional path planning guidance is not suitable for dynamic target capture tasks; secondly, the changeable confrontation game environment at sea requires that the unmanned boat cluster has a certain robustness when performing collaborative tasks, and it still needs to have the ability to continue to perform tasks when a small number of unmanned boats are damaged due to emergencies; finally, the strategy network needs to have a certain scalability to adapt to the dynamic changes in the number of unmanned boat clusters in performing different tasks. Summary of the invention

[0004] In view of the above-mentioned difficulties faced by the collaborative capture of multiple unmanned boats, the present invention proposes a collaborative capture method of multiple unmanned boats based on a deep reinforcement learning algorithm with a channel attention mechanism. By improving the channel attention mechanism and the reward function design combining clusters and monomers, combined with the structural characteristics of the problem, the difficulties faced by the problem, such as the complex behavior of dynamic escape targets, the stability of cluster task execution, and the dynamic number changes of clusters, are overcome.

[0005] The technical solution to achieve the purpose of the present invention is: a multi-unmanned boat collaborative capture method based on a channel attention mechanism deep reinforcement learning algorithm, comprising the following steps:

[0006] Step 1: Establish a sea environment for cooperative capture by multiple unmanned boats; parameterize the island and reef information in the sea environment and the position angle information of each unmanned boat, and give a strategy function for the escape ship;

[0007] Step 2: Model the interaction information of the unmanned boats into a Markov decision model, and design the action space, observation space and reward function of the multi-unmanned boat collaborative capture task;

[0008] Step 3: construct a channel attention network to process the observation information of the friendly unmanned boat state and then splice it with the observation information of the escape boat and its own state and input it into the algorithm module;

[0009] Step 4: Build a target network based on Actor-Critic, store the actions, states and rewards of the unmanned boat as batch experiences in the experience pool. During the training process, the centralized evaluation function Critic uses the joint experience in the experience pool to update the network parameters, and the Actor function updates the strategy based on the Q value given by Critic.

[0010] Step 5: In the execution phase, the updated Actor decision function is used to make decentralized decisions. In the execution phase, only local observations of the actor are used to obtain the strategy.

[0011] Furthermore, step 1 includes:

[0012] Step 1-1, establishing a spatial coordinate system of the sea area and specifying the positive direction of the spatial coordinate system, setting the boundary of the sea area and the position coordinate information of each unmanned boat and island;

[0013] Step 1-2, define the following conditions for successful roundup:

[0014] (1)

[0015] in, represents the Euclidean distance from our capture boat to the escape boat, represents the capture radius of the escape boat, It represents the angle between our adjacent capture boats;

[0016] Steps 1-3, establish the following escape strategy:

[0017] (2)

[0018] in, represents the escape velocity vector of the enemy escape craft, Indicates the location coordinates of our capture boat. Indicates the location coordinate information of the escape boat, represents the Euclidean distance from our capture boat to the escape boat, is the gravitational adjustment coefficient, Indicates the location coordinate information of islands and reefs.

[0019] Furthermore, step 2 includes:

[0020] Step 2-1: The multi-unmanned boat cooperative capture task can be described as a Markov game, using the tuple express, is the global state, and the joint observation space of the capture boat is is the joint action space of the capture boat, Indicates in status Take strategic actions Then transfer to the next state The transition probability, is the decay coefficient of the cumulative discount reward;

[0021] Step 2-2, establish the second-order kinematic equation of the capture boat and define the continuous action space:

[0022] (3)

[0023] (4)

[0024] Among them, in formula (3) and They represent the linear velocity and angular velocity of the unmanned boat respectively, and They represent the derivatives of the unmanned boat’s linear velocity and angular velocity with respect to time, and They represent the linear acceleration and angular acceleration of the unmanned boat, Unmanned boat The location coordinates on the sea area, and Unmanned Boat The instantaneous velocity in the horizontal and vertical directions, Unmanned boat The heading angle, Unmanned boat The instantaneous heading angle of Unmanned boat exist The position coordinates on the sea area at the moment, is the two-dimensional continuous action space of the unmanned boat, where For unmanned boats The instantaneous speed of the output in the axis direction, For unmanned boats Instantaneous speed output in the axis direction, time interval , Unmanned boat exist Time interval The position coordinates of the unmanned boat in the sea area; the strategy network outputs the action Update the location according to formula (4);

[0025] Step 2-3, for There are unmanned boats, and the observation space is defined as follows:

[0026] (5)

[0027] (6)

[0028] Among them, formula (5) represents the unmanned boat The observation space Includes: Escape Boat Observation , observation of friendly unmanned boats participating in the same coordinated capture mission And its own state observation ; Formula (6) represents the components of these three parts of observations, where and Indicated as escape boat Compared to unmanned boats side angle and its rate of change; distance from the escape boat Distance and its rate of change ; Enemy's speed and its rate of change ; and Unmanned boat A distance from a friendly unmanned boat The distance and rate of change, represents the mutual side angle and its rate of change, Unmanned boat and unmanned boats Angle of The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance between ;

[0029] Step 2-4, the reward function for the unmanned boat swarm collaborative capture task is designed as follows:

[0030] (7)

[0031] (8)

[0032] (9)

[0033] (10)

[0034] (11)

[0035] Among them, formula (7) represents the unmanned boat The reward function Rewarded by a single entity and cluster rewards Composition; Formula (8) represents the single reward It consists of 4 parts: distance reward , collision avoidance reward , Angle Reward And additional rewards for entering the roundup ; Formula (9) represents the cluster reward It consists of two parts: distance collaboration reward Collaboration Rewards with Angle ;In formula (10) The number of all our capture boats participating in the capture mission, Unmanned boat A distance from a friendly unmanned boat The shortest distance, It means the reward for any of our capture boats entering the enemy's capture area. represents the reward obtained when all our capture boats enter the enemy capture area; in formula (11) and Represents distance cooperation reward and angle cooperation reward respectively, unmanned boat The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance of .

[0036] Furthermore, step 3 includes:

[0037] Step 3-1: Read each unmanned boat’s observation information of friendly unmanned boats ;

[0038] Step 3-2, observe the information Different types of attributes are reconstructed to make the angle and distance observations on the same quantitative scale, and N groups are obtained. dimensional vector;

[0039] Step 3-3, group N The dimensional feature vector passes through three fully connected layers of different depths to obtain three different scale expressions of the same set of observation states;

[0040] In step 3-4, the state expressions of three different scales are used as the input of the channel attention module. After the compression excitation layer, the weight relationship between the seven channels is obtained, and finally the weighted average output is performed with the input information.

[0041] Furthermore, step 4 includes:

[0042] Step 4-1, Initialization The value network of unmanned boats , policy network parameters Network parameters and , and and Copy the parameters passed to the corresponding target network; initialize the experience library , where the value network Indicated in the parameter Confirm that the unmanned boat is in the state Start by selecting an action After that, the expected cumulative reward that can be obtained; Strategy network Indicated in the parameter Confirm, the network is in the current state As input, it outputs a parameterized distribution of actions, and Denotes deployed on the 1st to 2nd ships respectively. Our team is capturing an unmanned boat on a valuable network and policy network Parameters;

[0043] Step 4-2, when the experience library When the stored data is greater than the preset threshold, a random Group Sequence Train the online network as a batch;

[0044] Step 4-3, Group Sequence Define the target as an example The values ​​are as follows:

[0045] (12)

[0046] in The experience data stored in the experience pool is the current state ,action ,award and the next moment state The four-tuple of target Value is , in For the reward of the present moment, is the decay coefficient of the cumulative discount reward, Unmanned boat From the next moment state Get started with the Strategy Network Select Action , the expected cumulative reward that can be obtained; Formula (12) is used as the target Q value, and the loss function is defined as follows:

[0047] (13)

[0048] minimize Update online critic network parameters ,in is the total number of sequence groups drawn from the experience pool, Indicates that the unmanned boat is in state Start by selecting an action After that, the expected cumulative reward that can be obtained;

[0049] Step 4-4, the sample policy gradient is as follows:

[0050] (14)

[0051] Represents the target policy function Ask about The gradient of the network is updated by gradient descent ,in Unmanned boat In Status The policy network function under The gradient of Indicates from the state Start by using various strategies to capture unmanned boats Get joint action After that, the expected cumulative reward that can be obtained is, according to the chain rule of gradient, Represents the value network Ask about The gradient of

[0052] Step 4-5, sliding method to update target network parameters , :

[0053] (15)

[0054] in To update the hyperparameters, and To update the previous Critic network and Actor network parameters.

[0055] Furthermore, step 5 includes:

[0056] Each unmanned boat takes appropriate actions based on its own local observations according to the Actor decision function, without the need to interact with the states of other intelligent agents.

[0057] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0058] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0059] A computer program product comprises a computer program, which implements the steps of the above method when executed by a processor.

[0060] Due to the adoption of the above technical scheme, the present invention can achieve the following technical effects: the established Markov decision model for collaborative capture control of multiple unmanned boats can effectively characterize the future uncertain capture strategy requirements for dynamic escape targets in complex environments, while taking into account the impact of environmental state changes before and after the execution of the capture strategy, thereby improving the cluster's adaptability to the capture of dynamic escape targets in complex environments. Based on the above Markov decision model, an improved channel attention mechanism is proposed, which, combined with the different dimensional observation types after feature reconstruction, can handle the difficulty of dynamic changes in the number of clusters. At the same time, this algorithm designs a reward function that combines clusters and monomers. Compared with the traditional controller method, it can continue to perform tasks when a small number of unmanned boats are damaged due to emergencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a principle block diagram of the collaborative capture method of multiple unmanned boats of the present invention.

[0062] Figure 2 It is a schematic diagram of successful roundup of the present invention.

[0063] Figure 3 It is a schematic diagram of the observation space of the present invention.

[0064] Figure 4 Schematic diagram of the improved channel attention module of the present invention.

[0065] Figure 5 It is the reward convergence curve of our capture boat in the 3V1 coordinated capture of the present invention.

[0066] Figures 6 to 8 It is a schematic diagram of the stages of the 3V1 collaborative roundup trajectory process of the present invention.

[0067] Fig. 9 It is the reward convergence curve of our capture boat in the 4V1 coordinated capture of the present invention.

[0068] Figures 10 to 12 It is a schematic diagram of the stages of the 4V1 collaborative roundup trajectory process of the present invention.

[0069] Fig.13 It is the reward convergence curve of our capture boat in the 5V1 coordinated capture of the present invention.

[0070] Figures 14 to 16 It is a schematic diagram of the stages of the 5V1 collaborative roundup trajectory process of the present invention.

[0071] Figures 17 to 20 It is a schematic diagram of the collaborative encirclement trajectory process stages in the 4V1 collaborative encirclement process of the present invention, simulating a situation where a small number of unmanned boats lose communication and then resume communication. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0073] like Figure 1 , Figure 4 As shown, the present invention provides a multi-unmanned boat collaborative capture method based on a channel attention mechanism deep reinforcement learning algorithm, and the specific implementation steps are as follows:

[0074] Step 1: If Figure 2 As shown in the figure, a successful model of multi-unmanned boat collaborative capture is established, and the position angle information of each unmanned boat is parameterized, where Figure 2 Medium Simulation For our three capture boats, is the escape target. The model's capture success condition can be expressed as:

[0075] (1)

[0076] in, Figure 2 middle represents the Euclidean distance from our capture boat to the escape boat, represents the capture radius of the escape boat, It represents the angle between our adjacent capture boats.

[0077] The escape strategy of the escape boat can be expressed as:

[0078] (2)

[0079] in, represents the escape velocity vector of the enemy escape craft, Indicates the location coordinates of our capture boat. Indicates the location coordinate information of the escape boat. represents the Euclidean distance from our capture boat to the escape boat, is the gravitational adjustment coefficient, Indicates the location coordinate information of islands and reefs.

[0080] Step 2: Markov decision model. Define the model state as express, is the global state, and the joint observation space of the capture boat is , is the joint action space of the capture boat, Indicates in status Take strategic actions Then transfer to the next state The transition probability, is the reward function of each capture boat, is the decay coefficient of the cumulative discounted reward. According to the second-order kinematic equation of the capture boat, the continuous action space is defined as:

[0081] (3)

[0082] (4)

[0083] Among them, in formula (3) and They represent the linear velocity and angular velocity of the unmanned boat respectively, and They represent the derivatives of the unmanned boat’s linear velocity and angular velocity with respect to time, and They represent the linear acceleration and angular acceleration of the unmanned boat, Unmanned boat The location coordinates on the sea area, and Unmanned Boat The instantaneous velocity in the horizontal and vertical directions, Unmanned boat The heading angle, Unmanned boat The instantaneous heading angle of Unmanned boat exist The position coordinates on the sea area at the moment, is the two-dimensional continuous action space of the unmanned boat, where For unmanned boats The instantaneous speed of the output in the axis direction, For unmanned boats Instantaneous speed output in the axis direction, time interval , Unmanned boat exist Time interval The position coordinates of the unmanned boat in the sea area; the strategy network outputs the action Update the location according to formula (4);

[0084] like Figure 3 As shown for There are unmanned boats, and the observation space is defined as follows:

[0085] (5)

[0086] (6)

[0087] Among them, formula (5) represents the unmanned boat The observation space Includes: Escape Boat Observation , observation of friendly unmanned boats participating in the same coordinated capture mission And its own state observation ; like Figure 3 As shown, simulating our unmanned boat With the escape boat The capture observation environment, including Our unmanned boat their respective speeds; Figure 3 The remaining parameters are observation quantities, and now we use the unmanned boat For example, and Indicated as escape boat Compared to unmanned boats side angle and its rate of change; distance from the escape boat Distance and its rate of change ;Enemy escape craft Speed and its rate of change ; and Unmanned boat A distance from a friendly unmanned boat The distance and rate of change, Our unmanned boat and unmanned boats The relative shear angles and their rates of change, Unmanned boat and unmanned boats Angle of The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance of .

[0088] The reward function designed for the unmanned boat swarm collaborative capture task is as follows:

[0089] (7)

[0090] (8)

[0091] (9)

[0092] (10)

[0093] (11)

[0094] Among them, formula (7) represents the unmanned boat The reward function Rewarded by a single entity and cluster rewards Composition; Formula (8) represents the single reward It consists of 4 parts: distance reward , collision avoidance reward , Angle Reward And additional rewards for entering the roundup ; Formula (9) represents the cluster reward It consists of two parts: distance collaboration reward Collaboration Rewards with Angle ;In formula (10) The number of all our capture boats participating in the capture mission, Unmanned boat A distance from a friendly unmanned boat The shortest distance, It means the reward for any of our capture boats entering the enemy's capture area. represents the reward obtained when all our capture boats enter the enemy capture area; in formula (11) and Represents distance cooperation reward and angle cooperation reward respectively, unmanned boat The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance of .

[0095] Step 3: Construct a channel attention network to process the dynamically changing state observation information of the friendly unmanned boat and then splice it with the escape boat and its own state observation information and input it into the algorithm module. Figure 4 As shown, first read the original observation information of each unmanned boat on the friendly unmanned boat , and then the observation information Different types of attributes are reconstructed to make the angle, distance and other observations on the same quantitative scale, and N groups are obtained. dimensional observation feature vector. Then N groups The dimensional feature vector passes through three fully connected layers of different depths to obtain three different scale expressions of the same set of observation states. Finally, the three different scale state expressions are used as the input of the channel attention module. After the compression excitation layer, the weight relationship between the seven channels is obtained and the weighted average output is performed with the input information.

[0096] Step 4: Build an Actor-Critic based target network. The actions, states and rewards of the unmanned boat are stored in the experience pool as batch experiences. First, initialize The value network of an unmanned boat , policy network parameters Network parameters and ; and and Copy the parameters passed to the corresponding target network; initialize the experience library ; Among them, the value network Indicated in the parameter Confirm that the unmanned boat is in the state Start by selecting an action After that, the expected cumulative reward that can be obtained; Strategy network Indicated in the parameter Confirm, the network is in the current state As input, it outputs a parameterized distribution of actions, and Denotes deployed on the 1st to 2nd ships respectively. Our team is capturing an unmanned boat on a valuable network and policy network When the experience library When the stored data is greater than the preset threshold, a random Group Sequence The online network is trained as a batch, with Group Sequence Define the target as an example value:

[0097] (12)

[0098] in The experience data stored in the experience pool is the current state ,action ,award and the next moment state The four-tuple of target Value is , in For the reward of the present moment, is the decay coefficient of the cumulative discount reward, Unmanned boat Next moment status Get started with the Strategy Network Select Action , the expected cumulative reward that can be obtained. Formula (12) is used as the target Q value and the loss function is defined as:

[0099] (13)

[0100] minimize Update online critic network parameters ,in is the total number of sequence groups drawn from the experience pool, Indicates that the unmanned boat is in state Start by selecting an action After that, the expected cumulative reward that can be obtained;

[0101] The sample policy gradient is as follows:

[0102] (14)

[0103] Represents the target policy function Ask about The gradient of the network is updated by gradient descent ,in Unmanned boat In Status The policy network function under The gradient of Indicates from the state Start by using various strategies to capture unmanned boats Get joint action After that, the expected cumulative reward that can be obtained is, according to the chain rule of gradient, Represents the value network Ask about The gradient of

[0104] Finally, the sliding method is used to update the target network parameters , :

[0105] (15)

[0106] Step 5: During the execution phase, each unmanned boat takes appropriate actions based on its own local observations according to the Actor decision function, without interacting with the states of other agents.

[0107] The simulation results of the present invention are as follows Figures 5 to 20 As shown in the figure, the hardware configuration of the experimental platform is CPU i9-10980XE, GPU RTX4 070 Ti, memory 128 GB, Pytorch is used as the deep learning training framework, and OpenAIGym is used as the reinforcement learning environment framework. Figures 5 to 16 As shown in the figure, a simulation experiment of the coordinated capture strategy was conducted on three different numbers of capture boats: 3 capture boats, 4 capture boats, and 5 capture boats. Figures 6 to 8 , Figures 10 to 12 , Figures 14 to 16 The horizontal and vertical coordinates in the capture trajectory process record diagram determine the scope of the sea area, as well as the location information of the island and the unmanned boat; Figure 5 , Fig. 9 , Fig.13 It can be seen from the reward curve that at the beginning of exploration, the unmanned boat reward curve is in a climbing state, and the reward tends to be stable between 1.5 million and 2 million steps. The encirclement and capture strategy is continuously optimized until convergence during the exploration process; from the encirclement and capture trajectory process records, it can be seen that under the three different numbers of game environments, the encirclement and capture boats can successfully implement encirclement and capture before the escape boat reaches the island reef, and some unmanned boats emerge and insert behind the escape boat to cooperate with other friendly unmanned boats to encircle and capture it.

[0108] like Figures 17 to 20 The figure shows that at the beginning of the capture mission, the position information of the unmanned boat is given by the horizontal and vertical coordinates. One of our capture boats close to the island reef adopts a random strategy to simulate damage and does not participate in the capture mission. The remaining three capture boats under the invention are not affected by the boat and continue to approach the escape target. The adjacent friendly boats penetrate from the rear and form an encirclement of the escape boat. In the subsequent execution steps, the disconnected unmanned boat is reconnected to the cluster. It can be seen from the capture trajectory process record that the unmanned boat can still cooperate with the remaining three boats to enter the encirclement, which verifies that when a small number of unmanned boats are damaged due to emergencies, the remaining cluster still has the ability to continue to perform the mission.

[0109] The Markov decision model of the collaborative capture control of multiple unmanned boats of the present invention can effectively characterize the future uncertain capture strategy requirements of dynamic escape targets in complex environments, takes into account the impact of environmental state changes before and after the execution of the capture strategy, and improves the success rate of cluster capture of dynamic escape targets in complex environments; when inputting cluster state features, the proposed channel attention mechanism is combined with the different dimensional observation types after feature reconstruction to solve the difficulty of dealing with dynamic changes in the number of clusters; finally, the present invention designs a reward function that combines clusters and monomers. Compared with the traditional controller method, it can continue to perform tasks when a small number of unmanned boats are damaged due to emergencies.

[0110] The present invention provides a multi-unmanned boat collaborative capture method based on a channel attention mechanism deep reinforcement learning algorithm. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A multi-unmanned boat collaborative capture method based on a channel attention mechanism deep reinforcement learning algorithm, characterized in that: The steps include: Step 1: Establish a sea environment for cooperative capture by multiple unmanned boats; parameterize the island and reef information in the sea environment and the position angle information of each unmanned boat, and give a strategy function for the escape ship; Step 2: Model the interaction information of the unmanned boats into a Markov decision model, and design the action space, observation space and reward function of the multi-unmanned boat collaborative capture task; specifically including: Step 2-1: The task of multiple unmanned boats coordinating and capturing is described as a Markov game, using the tuple express, is the global state, and the joint observation space of the capture boat is , is the joint action space of the capture boat, Indicates in status Take strategic actions Then transfer to the next state The transition probability, is the reward function of each capture boat, is the decay coefficient of the cumulative discount reward; Step 2-2, establish the second-order kinematic equation of the capture boat and define the continuous action space: （3） （4） Among them, in formula (3) and They represent the linear velocity and angular velocity of the unmanned boat respectively, and They represent the derivatives of the unmanned boat’s linear velocity and angular velocity with respect to time, and They represent the linear acceleration and angular acceleration of the unmanned boat respectively, Unmanned boat The location coordinates on the sea area, and Unmanned Boat The instantaneous velocity in the horizontal and vertical directions, Unmanned boat The heading angle, Unmanned boat The instantaneous heading angle of Unmanned boat exist The position coordinates on the sea area at the moment, is the two-dimensional continuous action space of the unmanned boat, where For unmanned boats The instantaneous speed of the output in the axis direction, For unmanned boats Instantaneous speed output in the axis direction, time interval , Unmanned boat exist Time interval The position coordinates of the unmanned boat in the sea area; the strategy network outputs the action Update the location according to formula (4); Step 2-3, for There are unmanned boats, and the observation space is defined as follows: （5） （6） Among them, formula (5) represents the unmanned boat The observation space Includes escape boat Observation , observation of friendly unmanned boats participating in the same coordinated capture mission And its own state observation ; Formula (6) represents the components of these three parts of observations, where and Indicated as escape boat Compared to unmanned boats side angle and its rate of change; distance from the escape boat Distance and its rate of change ; Enemy's speed and its rate of change ; and Unmanned boat A distance from a friendly unmanned boat The distance and rate of change, represents the mutual side angle and its rate of change, Unmanned boat and unmanned boats Angle of The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance between ; Step 2-4, the reward function for the unmanned boat swarm collaborative capture task is designed as follows: （7） （8） （9） （10） （11） Among them, formula (7) represents the unmanned boat The reward function Rewarded by a single entity and cluster rewards Composition; Formula (8) represents the single reward It consists of 4 parts: distance reward , collision avoidance reward , Angle Reward And additional rewards for entering the roundup ; Formula (9) represents the cluster reward It consists of two parts: distance collaboration reward Collaboration Rewards with Angle ;In formula (10) The number of all our capture boats participating in the capture mission, Unmanned boat A distance from a friendly unmanned boat The shortest distance, It means the reward for any of our capture boats entering the enemy's capture area. represents the reward obtained when all our capture boats enter the enemy capture area; in formula (11) and Represents distance cooperation reward and angle cooperation reward respectively, unmanned boat The angle difference between the left and right adjacent friendly unmanned boats is , Expressed as the distance between the capture boat group and the escape boat The mean distance between ; Step 3: construct a channel attention network to process the observation information of the friendly unmanned boat state and then splice it with the observation information of the escape boat and its own state and input it into the algorithm module; specifically, it includes: Step 3-1: Read each unmanned boat’s observation information of friendly unmanned boats ; Step 3-2, observe the information Different types of attributes are reconstructed to make the angle and distance observations on the same quantitative scale, and N groups are obtained. dimensional vector; Step 3-3, group N The dimensional feature vector passes through three fully connected layers of different depths to obtain three different scale expressions of the same set of observation states; Step 3-4: The state expressions of three different scales are used as the input of the channel attention module. After the compression excitation layer, the weight relationship between the seven channels is obtained, and finally the weighted average output is performed with the input information. Step 4: Build a target network based on Actor-Critic, store the actions, states and rewards of the unmanned boat as batch experiences in the experience pool. During the training process, the centralized evaluation function Critic uses the joint experience in the experience pool to update the network parameters, and the Actor function updates the strategy based on the Q value given by Critic. Step 5: In the execution phase, the updated Actor decision function is used to make decentralized decisions. In the execution phase, only local observations of the actor are used to obtain the strategy.

2. The multi-unmanned boat collaborative capture method based on the channel attention mechanism deep reinforcement learning algorithm according to claim 1 is characterized in that: Step 1 specifically includes: Step 1-1, establishing a spatial coordinate system of the sea area and specifying the positive direction of the spatial coordinate system, setting the boundary of the sea area and the position coordinate information of each unmanned boat and island; Step 1-2, define the following conditions for successful roundup: （1） in, represents the Euclidean distance from our capture boat to the escape boat, represents the capture radius of the escape boat, It represents the angle between our adjacent capture boats; Steps 1-3, establish the following escape strategy: （2） in, represents the escape velocity vector of the enemy escape craft, Indicates the location coordinates of our capture boat. Indicates the location coordinate information of the escape boat. represents the Euclidean distance from our capture boat to the escape boat, is the gravitational adjustment coefficient, Indicates the location coordinate information of islands and reefs.

3. The multi-unmanned boat collaborative capture method based on the channel attention mechanism deep reinforcement learning algorithm according to claim 1 is characterized in that: Step 4 specifically includes: Step 4-1, Initialization The value network of unmanned boats , policy network Network parameters and , and and Copy the parameters passed to the corresponding target network; initialize the experience library , where the value network Indicated in the parameter Confirm that the unmanned boat is in the state Start by selecting an action After that, the expected cumulative reward that can be obtained; Strategy network Indicated in the parameter Confirm, the network is in the current state As input, it outputs a parameterized distribution of actions, and Denotes deployed on the 1st to 2nd ships respectively. Our team is capturing an unmanned boat on a valuable network and policy network Parameters; Step 4-2, when the experience library When the stored data is greater than the preset threshold, a random Group Sequence Train the online network as a batch; Step 4-3, Group Sequence Define the target as an example The values ​​are as follows: （12） in The experience data stored in the experience pool is the current state ,action ,award and the next moment state The four-tuple of target Value is , in For the reward of the present moment, is the decay coefficient of the cumulative discount reward, Unmanned boat From the next moment state Get started with the Strategy Network Select Action , the expected cumulative reward that can be obtained; Formula (12) is used as the target Q value, and the loss function is defined as follows: （13） minimize Update online network parameters ,in is the total number of sequence groups drawn from the experience pool, Indicates that the unmanned boat is in state Start by selecting an action After that, the expected cumulative reward that can be obtained; Step 4-4, the sample policy gradient is as follows: （14） Represents the target policy function Ask about The gradient of the network is updated by gradient descent ,in Unmanned boat In Status The policy network function under The gradient of Indicates from the state Start by using various strategies to capture unmanned boats Get joint action After that, the expected cumulative reward that can be obtained is, according to the chain rule of gradient, Represents the value network Ask about The gradient of Step 4-5, sliding method to update target network parameters , : （15） in To update the hyperparameters, and To update the previous Critic network and Actor network parameters.

4. The multi-unmanned boat collaborative capture method based on the channel attention mechanism deep reinforcement learning algorithm according to claim 1 is characterized in that: In step 5, each unmanned boat takes action based on its own local observation according to the Actor decision function, without interacting with the states of other intelligent agents.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images