A reinforcement learning-based method, program, device, and storage medium for full-domain coverage of unmanned swarms.

Abstract

This invention belongs to the field of underwater acoustic detection technology, specifically relating to a method, program, device, and storage medium for full-area coverage of unmanned swarms based on reinforcement learning. This invention constructs the task of full-area coverage of a target sea area by an unmanned swarm as a distributed partially observable Markov decision process, builds a detection and perception map with a revisit mechanism, quantitatively represents the coverage distribution state of the full-area nautical chart, and effectively guides the surface unmanned swarm to exhibit intelligent emergent behavior of "rejecting densely covered areas and being attracted to unknown or low-attention areas." Addressing the specific needs of the full-area coverage task of surface unmanned swarms, this invention designs reinforcement learning framework elements adapted to the task, embedding complex acoustic physical constraints and spatial-temporal states into the mathematical model of reinforcement learning, significantly reducing the exploration difficulty for the agent in complex environments and accelerating the convergence process of the MATD3 algorithm.

Description

Technical Field

[0001] This invention belongs to the field of underwater acoustic detection technology, specifically relating to a method, program, device, and storage medium for full-domain coverage of unmanned swarms based on reinforcement learning. Background Technology

[0002] Traditional single-platform operation modes are no longer sufficient to meet the increasingly complex task requirements, and single-platform operations are gradually shifting towards collaborative operations of multiple types of unmanned platforms. Existing reinforcement learning-based coverage path planning techniques have been studied to some extent in surface cooperative detection missions. For example, the paper "Coverage Path Planning for AUVs Cooperative Environment Detection in Integrated Underwater Acoustic Communication and Detection Networks" introduces communication and detection models to coordinate the detection and information sharing process among surface autonomous vehicles, aiming to maximize coverage and minimize path length. It employs a multi-agent reinforcement learning algorithm to achieve coverage path planning in AUV cooperative environment detection. Another example is "Coverage Path Planning for AUVs Cooperative Environment Detection in Integrated Underwater Acoustic Communication and Detection Networks," which focuses on USVs with sensing capabilities. This research constructs multi-scale boundary maps based on line-of-sight angles and combines them with a multi-agent reinforcement learning algorithm for coverage path planning, improving coverage while reducing redundant path generation. The above research shows that reinforcement learning methods can improve the autonomous decision-making ability of unmanned platforms in coverage path planning to a certain extent.

[0003] However, existing full-coverage path planning methods still have the following shortcomings when facing complex water surface target search tasks:

[0004] (1) Insufficient detection and perception capabilities result in poor flexibility and environmental adaptability of path planning.

[0005] Modeling heterogeneous detection capabilities is oversimplified. Existing methods for describing the joint detection effectiveness of unmanned surface swarms typically use a fixed-radius "geometric Boolean circle" model to characterize the platform's operational range, and often treat heterogeneous platforms as homogenous. This simplification completely severs the differences in detection mechanisms between different sonar operating modes, such as purely passive and combined transmit / receive, making it difficult to truly reflect the asymmetric capabilities of heterogeneous platforms in collaborative detection.

[0006] The lack of a multi-dimensional parameter coupling mechanism is a significant issue. Traditional exploration planning often treats the marine environment as an ideal homogeneous space, lacking a deep coupling between the environment and target characteristics. The failure to effectively incorporate marine hydrological data with spatiotemporal evolution characteristics (such as sound velocity profiles and seabed topography) and the target's own acoustic characteristics leads to a severe disconnect between theoretical exploration models and the real, complex surface acoustic environment.

[0007] (2) The lack of a dynamic decision-making mechanism for long-term uncovered areas leads to insufficient target search capabilities.

[0008] Existing reinforcement learning-based environment awareness methods typically lack a dynamic mechanism to characterize the timeliness of area coverage and guide the cluster to prioritize searching long-uncovered areas. For moving target search tasks, certain areas, after a prolonged period without being searched, often have a higher probability of target reappearance. If the path planning process lacks proactive guidance for these areas, it can easily lead to unreasonable allocation of search resources, thus affecting the overall search performance.

[0009] (3) Existing reinforcement learning-based full-domain coverage models are not well adapted to the needs of actual tasks.

[0010] Currently, there is still relatively little research on reinforcement learning in the field of water surface path planning. Most existing methods are based on specific assumptions and simplified task scenarios, and the generalization ability of the model design is limited. They lack key modeling elements that can adapt to complex and ever-changing water surface environments, multi-platform collaborative mechanisms, and full-domain coverage task requirements, making it difficult to directly meet the requirements of practical applications. Summary of the Invention

[0011] The purpose of this invention is to address the problems of insufficient detection and perception capabilities and lack of decision-making mechanisms in existing methods for full-domain target search of unmanned surface swarms, and to provide a method, program, device and storage medium for full-domain coverage of unmanned swarms based on reinforcement learning.

[0012] A reinforcement learning-based method for full-domain coverage of unmanned swarms includes the following steps:

[0013] The task of achieving full coverage of the target sea area by the unmanned swarm is constructed as a distributed partially observable Markov decision process. The target sea area is divided into grids, a detection and perception map with a revisit mechanism is constructed, and a reward function, a loss function, and a policy network and value network of each surface unmanned platform in the unmanned swarm are constructed.

[0014] During the process of the unmanned swarm performing the task of full coverage of the target sea area, for each grid in the detection and perception map, if the grid is within the effective detection range of the unmanned swarm at the current moment, the attraction return factor and repulsion return factor of the grid are updated; based on the updated attraction return factor and repulsion return factor of each grid, the comprehensive pheromone concentration of each grid is updated, and the comprehensive pheromone concentration of each grid is adjusted according to the binary threshold, and the detection and perception map is transformed into a detection binary map.

[0015] In an unmanned swarm, a surface unmanned platform extracts a local observation map vector from the binary map of the detection system. The position, heading, speed, local observation map vector of the surface unmanned platform, and the positions of all other surface unmanned platforms in the swarm are used to construct the local observation space of the surface unmanned platform. The local observation space of the surface unmanned platform at the current moment is input into the policy network of the surface unmanned platform updated at the previous moment to obtain the actions of the surface unmanned platform.

[0016] Furthermore, the effective detection range of the unmanned swarm at the current moment is calculated as follows:

[0017] All surface unmanned platforms in the unmanned cluster are divided into multiple groups of active and passive dual-platform detection units or single passive platform detection units with heterogeneous detection characteristics.

[0018] For each detection unit, taking the current position of all surface unmanned platforms in the detection unit as the center point, the surrounding sea area is evenly divided in the horizontal direction according to the azimuth angle, the sound propagation loss of each azimuth angle is obtained, the received signal-to-noise ratio of the detection unit is calculated according to the sonar equation, and combined with the set detection threshold, the received signal-to-noise ratio is converted into the detection probability of the detection unit at the current moment.

[0019] The rank-K fusion criterion is adopted to perform spatial information fusion on the detection probabilities of each detection unit to obtain the spatial distribution of joint detection probabilities in the target sea area; an effective detection threshold is preset, and the grid set in the target sea area with a joint detection probability greater than the effective detection threshold is taken as the effective detection range of the unmanned swarm at the current moment.

[0020] Furthermore, for each grid in the detection and perception map, if the grid is within the effective detection range of the unmanned swarm at the current moment, the attraction return factor and repulsion return factor of the grid are updated, specifically as follows:

[0021] If the detection revisit map shows the first Grid in the current If the unmanned swarm is within its effective detection range at any given time, then update the [number of] [times]. The most recent probe time of the grid is the current time. At any given moment, calculate the current time. Time distance from the first The time interval between the last probe on the grid Update # Grid's attraction and return visit factor and rejection return visit factor :

[0022] ,

[0023] in, and Adjustable weights;

[0024] If the detection revisit map shows the first The grid is not in the current state. Within the effective detection range of the unmanned swarm at any given time, then the first... The grid's attraction and repulsion factors remain unchanged, i.e. , .

[0025] Furthermore, the attraction return factor based on the updated grid is... and rejection return visit factor Update the overall pheromone concentration of each grid. :

[0026]

[0027] in, To detect the first in the perception map Grid in the current The concentration of attraction pheromones at any given moment; To detect the first in the perception map Grid in the current The concentration of repulsive pheromones at any given moment;

[0028]

[0029]

[0030] in, To attract pheromone evaporation coefficient, The pheromone evaporation coefficient is used to exclude pheromones. To attract pheromone propagation coefficient, The repulsive pheromone propagation coefficient; To detect the first in the perception map The grid's preset attracts pheromones. To detect the first in the perception map The grid's preset rejection of pheromones; To detect the first in the perception map Grid from Time's up The constant transmission of attraction pheromones To detect the first in the perception map Grid from Time's up The constant transmission of repulsive pheromones;

[0031]

[0032]

[0033] in, To detect the first in the perception map The set of neighboring grids of a grid; To detect the first in the perception map The number of neighboring grids of the grid;

[0034] The binary threshold Adjust the overall pheromone concentration of each grid to convert the detection and sensing map into a detection binary map, specifically as follows:

[0035] Based on the overall pheromone concentration of each grid in the updated detection and perception map The corresponding comprehensive pheromone concentration The largest front The overall pheromone concentration of the grid becomes 1, and the overall pheromone concentration of the other grids becomes 0.

[0036] Furthermore, the unmanned surface platform in the unmanned swarm extracts local observation map vectors from the binary map of the detection, specifically as follows:

[0037] For the first in the unmanned cluster Unmanned surface platform, with the first Unmanned surface platforms Location at any moment Centered on, in probing the binary map direction and Detection distance thresholds are extracted in each direction. The grid within the area forms a local detection binary map; It is a positive integer; For the first Detection distance threshold for unmanned surface platforms;

[0038] The local detection binary map is arranged according to the side length Divide the map evenly into sub-maps and obtain the number of grid cells with a combined pheromone concentration of 1 within each sub-map. And the number of grid cells with a total pheromone concentration of 0. ; , For the first Unmanned surface platforms The number of sub-maps divided by the local probe binary map at any given time;

[0039] Calculate the first The first local detection binary map of the unmanned surface platform The number of grid cells in the submap with a combined pheromone concentration of 1. Total number of grids in the target sea area proportion ;

[0040]

[0041] All submaps Constructed as the first Unmanned surface platforms Local observation map vector at time .

[0042] Furthermore, the reward function for:

[0043]

[0044] in, and These are the weighting coefficients;

[0045] For the first Unmanned surface platform Time-based coverage rewards;

[0046]

[0047] For the first Unmanned surface platform Penalties for collisions at any moment;

[0048]

[0049] For the first Unmanned surface platform and the first Collision risk penalties for unmanned surface platforms;

[0050]

[0051] Furthermore, the loss function includes the loss function of the policy network and the loss function of the value network;

[0052] The current unmanned surface platform Local observation space at time Input the policy network of the unmanned surface platform updated at the previous time step. The actions of the unmanned surface platform were obtained during the process. ;

[0053] The loss function of the policy network is:

[0054]

[0055] in, It is the mean operator; For the first Unmanned surface platform Constantly updated policy network Based on the input local observation space Output Action The probability of; For the first Unmanned surface platform Constantly updated policy network Based on the input local observation space Output Action The probability of; For the clipping function, These are the trimming parameters; For the first Unmanned surface platform The generalized dominance estimate at time t represents the first time. Actual actions performed by unmanned surface platforms The relative value compared to the expected action;

[0056] The loss function of the value network is:

[0057]

[0058] in, For the first Unmanned surface platform Cumulative rewards over time ; For the first Unmanned surface platform A constantly updated value network Based on the input local observation space The output reward value.

[0059] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described reinforcement learning-based unmanned swarm full-domain coverage method.

[0060] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based unmanned swarm full-domain coverage method.

[0061] A computer program product includes computer instructions that, when executed by a processor, implement the steps of the above-described reinforcement learning-based unmanned swarm full-domain coverage method.

[0062] The beneficial effects of this invention are as follows:

[0063] This invention constructs a dynamic perception map that integrates a return visit mechanism and digital pheromones. By deeply integrating the return visit mechanism and the digital pheromone mechanism, it releases repulsive or attractive pheromones adaptively to environmental conditions, forming a return visit cooperation mechanism within the cluster. This can quantitatively characterize the coverage distribution of the entire nautical chart and effectively guide the surface unmanned cluster to generate intelligent emergent behavior of "rejecting densely covered areas and being attracted to unknown or low-attention (long-term unvisited) areas." This provides a reliable spatial memory and decision guidance basis for the dynamic collaboration of the cluster in complex environments, and significantly improves the cluster's ability to recognize and coordinate the global environmental situation.

[0064] This invention addresses the specific requirements of unmanned surface swarm full-domain coverage tasks by designing reinforcement learning framework elements adapted for such tasks. Based on a distributed partially observable Markov decision process, it transforms a high-fidelity detection and perception map into a network observation space, combines a realistic kinematic model to constrain the action space, and designs a joint reward function driving the learning objective, consisting of "coverage increment" and "collision avoidance penalty." This invention embeds complex acoustic physical constraints and spatial-temporal states into the mathematical model of reinforcement learning, significantly reducing the exploration difficulty for agents in complex environments and accelerating the convergence process of the MATD3 algorithm. Attached Figure Description

[0065] Figure 1 This is a diagram of the overall architecture of the present invention. Detailed Implementation

[0066] The present invention will now be further described with reference to the accompanying drawings.

[0067] This invention addresses the shortcomings of existing methods for full-domain target search in unmanned surface swarms, such as insufficient detection and perception capabilities and a lack of decision-making mechanisms. It provides a reinforcement learning-based method for full-domain target search in unmanned swarms, modeling the task as a distributed, partially observable Markov decision process and designing core modules adapted to task requirements.

[0068] (1) Environmental status: A heterogeneous multi-node detection performance model with multi-parameter coupling of platform-environment-target was constructed. At the same time, a detection perception map with a revisiting mechanism was established, which combines a detection revisiting map with digital pheromones.

[0069] (2) Observation space: obtained by dimensionality reduction mapping of sensory map features, and observation information suitable for multi-agent reinforcement learning input;

[0070] (3) Action space: The heading and speed dual-degree-of-freedom continuous parameters are used as the platform's motion control quantities to meet the continuous motion control requirements of the unmanned surface swarm;

[0071] (4) Reward function: Designed with the goal of maximizing detection coverage and avoiding platform collisions, combining coverage incentives and collision penalties to guide the unmanned swarm on the water surface to achieve efficient collaborative coverage.

[0072] After training, this invention can generate a collaborative path planning scheme for unmanned surface swarms in continuous time and space, improving the efficiency of full-domain coverage and the probability of target detection.

[0073] A reinforcement learning-based method for full-domain coverage of unmanned swarms includes the following steps:

[0074] Step 1: Construct the task of full coverage of the target sea area by the unmanned swarm as a distributed partially observable Markov decision process, divide the target sea area into grids, construct a detection and perception map with a revisit mechanism, and construct the reward function, loss function, and policy network and value network of each surface unmanned platform in the unmanned swarm.

[0075] Unmanned clusters include One unmanned surface platform; the target sea area is divided into grids, with the center of each grid being […]. superscript For grid index, , Total number of grid cells;

[0076] Step 2: Initialization ;

[0077] Initialize the attraction and revisit factors of all grids in the sensor map. Rejection factors Comprehensive pheromone concentration ;

[0078] Initialize the policy network for each unmanned surface platform With value network ;

[0079] Step 3: In the current At any given moment, obtain the effective detection range of the unmanned swarm;

[0080] A detection performance analysis model is constructed based on the multi-parameter coupling of platform, environment, and target under multiple detection modes. These multiple detection modes include multi-base mode, active-passive mode, and an active + multi-base mode for scenarios where the platform's turning causes receiver failure. Based on the platform's functional configuration relationships under different cooperative detection modes, [the model is then implemented]. Each surface unmanned platform is divided into multiple groups of active and passive dual-platform detection units or single passive platform detection units with heterogeneous detection characteristics.

[0081] For any detection unit, with the current At any given moment, using the positions of all unmanned surface platforms in this detection unit as the center point, the surrounding sea area is evenly divided horizontally according to azimuth angles to obtain the current position. The sound propagation loss at various azimuth angles at any given time is used to calculate the received signal-to-noise ratio (SNR) of the detection unit based on the sonar equations. Combined with a set detection threshold, the received SNR is then converted into the current... The detection probability of the detection unit at any given time;

[0082] Among them, subscript For indexing unmanned surface platforms, ;

[0083] The rank-K fusion criterion is used to fuse the spatial information of the detection probabilities of each detection unit to obtain the spatial distribution of the joint detection probability of the target sea area. An effective detection threshold is preset, and the grid set in the target sea area with a joint detection probability greater than this threshold is used as the current detection probability. The effective detection range of unmanned swarms at all times;

[0084] Step 4: For the first detection in the perception map Grid If the first Grid in the current If the unmanned cluster is within its effective detection range at any given time, then the most recent detection time of that grid is updated to the current time. The time interval between the current grid and the last detection time is calculated. Update the grid's attraction return factor. and rejection return visit factor ;

[0085] If the first The grid is not in the current state. Within the effective detection range of the unmanned cluster at any given time, the attraction and repulsion factors of the grid remain unchanged, i.e. , ;

[0086] in, and All weights are adjustable;

[0087] Step 5: Based on the updated attraction and return visit factors of each grid. and rejection return visit factor Based on the previous moment, that is The concentration of attracting pheromones in each grid of the constantly updated detection and perception map. and repulsive pheromone concentration Update the overall pheromone concentration of each grid in the detection and perception map. ;

[0088]

[0089] in, To detect the first in the perception map Grid in the current The concentration of attraction pheromones at any given moment; To detect the first in the perception map Grid in the current The concentration of repulsive pheromones at any given moment;

[0090]

[0091]

[0092] in, To attract pheromone evaporation coefficient, The pheromone evaporation coefficient is used to exclude pheromones. To attract pheromone propagation coefficient, The repulsive pheromone propagation coefficient; To detect the first in the perception map The grid's preset attracts pheromones. To detect the first in the perception map The grid's preset rejection of pheromones; To detect the first in the perception map Grid from Time's up The constant transmission of attraction pheromones To detect the first in the perception map Grid from Time's up The constant transmission of repulsive pheromones;

[0093]

[0094]

[0095] in, For the first The set of neighboring grids of a grid; For the first The number of neighboring grids of the grid;

[0096] Based on the overall pheromone concentration of each grid in the updated detection and perception map Set binary threshold The corresponding comprehensive pheromone concentration The largest front The overall pheromone concentration of the grid becomes 1, and the overall pheromone concentration of the other grids becomes 0, thereby converting the updated detection and perception map into a detection binary map.

[0097] Step 6: Using the first Unmanned surface platforms Location at any moment Centered on, in probing the binary map direction and Detection distance thresholds are extracted in each direction. The grid within the area forms a local detection binary map; It is a positive integer; For the first Detection distance threshold for unmanned surface platforms;

[0098] The local detection binary map is arranged according to the side length Divide the map evenly into sub-maps and obtain the number of grid cells with a combined pheromone concentration of 1 within each sub-map. And the number of grid cells with a total pheromone concentration of 0. ; , For the first Unmanned surface platforms The number of sub-maps divided by the local probe binary map at any given time;

[0099] Calculate the first The first local detection binary map of the unmanned surface platform The number of grid cells in the submap with a combined pheromone concentration of 1. The proportion of the total number of grids in the target sea area , ; All submaps Constructed as the first Unmanned surface platforms Local observation map vector at time ;

[0100] The first Unmanned surface platforms Location at any moment ,course Speed Local observation map vector and the location of all other unmanned surface platforms. Constructed as the first Unmanned surface platforms Local observation space at time ;

[0101] Local observation space of all unmanned surface platforms Constructed as a global observation space ;

[0102] Step 7: Unmanned surface platforms will provide localized observation space Input up to the previous time step, i.e. A constantly updated self-policy network In the middle, the action is obtained. ;

[0103] Calculate the reward function ;

[0104]

[0105] in, and These are the weighting coefficients;

[0106] For the first Unmanned surface platform The coverage reward at time step is calculated by... Unmanned surface platform Compared to the number of grids with a comprehensive pheromone concentration of 0 in the local detection binary map at a given time. The increment at time, i.e.:

[0107]

[0108] For the first Unmanned surface platform Penalties for collisions at any moment;

[0109] ,

[0110] Step 8: Update the first step using the loss function Strategy Network for Unmanned Surface Platforms With value network ;

[0111] Loss function of policy network:

[0112]

[0113] in, It is the mean operator; For the first Before the update of the unmanned surface platform, Timing-based strategy networks Based on local observation space Output Action The probability of; For the first After the unmanned surface platform is updated, Timing-based strategy networks Based on local observation space Output Action The probability of; For the clipping function, that is... The value is restricted to the range Inside; These are the trimming parameters used to control the update range; For the first Unmanned surface platform The generalized advantage estimate at a given moment represents the action taken. The relative value compared to the expected value;

[0114] Loss function of value network:

[0115]

[0116] in, For the first Unmanned surface platform Cumulative rewards over time ; For value networks After entering the number Unmanned surface platform Local observation space at time The output after that;

[0117] Step 9: If If the task of achieving full coverage of the target sea area has been completed, then stop the iteration; otherwise, let... Return to step 3.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for full-domain coverage of unmanned swarms based on reinforcement learning, characterized in that: The task of achieving full coverage of the target sea area by the unmanned swarm is constructed as a distributed partially observable Markov decision process. The target sea area is divided into grids, a detection and perception map with a revisit mechanism is constructed, and a reward function, a loss function, and a policy network and value network of each surface unmanned platform in the unmanned swarm are constructed. During the process of the unmanned swarm performing the task of full coverage of the target sea area, for each grid in the detection and perception map, if the grid is within the effective detection range of the unmanned swarm at the current moment, the attraction return factor and repulsion return factor of the grid are updated. For each grid in the detection and perception map, if the grid is within the effective detection range of the unmanned swarm at the current moment, the attraction return factor and repulsion return factor of the grid are updated, specifically as follows: If the detection revisit map shows the first Grid in the current If the unmanned swarm is within its effective detection range at any given time, then update the [number of] [times]. The most recent probe time of the grid is the current time. At any given moment, calculate the current time. Time distance from the first The time interval between the last probe on the grid Update # Grid's attraction and return visit factor and rejection return visit factor : ， in, and Adjustable weights; If the first grid in the detection return visit map is not in the effective detection range of the current moment unmanned cluster, the attraction return visit factor and the repulsion return visit factor of the first grid are unchanged, that is , ; According to the updated attractive revisit factor of each grid and repulsive revisit factor , update the comprehensive pheromone concentration of each grid : in, To detect the first in the perception map Grid in the current The concentration of attraction pheromones at any given moment; To detect the first in the perception map Grid in the current The concentration of repulsive pheromones at any given moment; in, To attract pheromone evaporation coefficient, The pheromone evaporation coefficient is used to exclude pheromones. To attract pheromone propagation coefficient, The repulsive pheromone propagation coefficient; To detect the first in the perception map The grid's preset attracts pheromones. To detect the first in the perception map The grid's preset rejection of pheromones; To detect the first in the perception map Grid from Time's up The constant transmission of attraction pheromones To detect the first in the perception map Grid from Time's up The constant transmission of repulsive pheromones; in, To detect the first in the perception map The set of neighboring grids of a grid; To detect the first in the perception map The number of neighboring grids of the grid; According to the binary threshold The integrated pheromone concentration of each grid is adjusted, and the detection perception map is converted into a detection binary map, specifically: Based on the overall pheromone concentration of each grid in the updated detection and perception map The corresponding comprehensive pheromone concentration The largest front The overall pheromone concentration of the grid becomes 1, and the overall pheromone concentration of the other grids becomes 0. The effective detection range of the unmanned swarm at the current moment is calculated as follows: All surface unmanned platforms in the unmanned cluster are divided into multiple groups of active and passive dual-platform detection units or single passive platform detection units with heterogeneous detection characteristics. For each detection unit, taking the current position of all surface unmanned platforms in the detection unit as the center point, the surrounding sea area is evenly divided in the horizontal direction according to the azimuth angle, the sound propagation loss of each azimuth angle is obtained, the received signal-to-noise ratio of the detection unit is calculated according to the sonar equation, and combined with the set detection threshold, the received signal-to-noise ratio is converted into the detection probability of the detection unit at the current moment. The rank-K fusion criterion is adopted to perform spatial information fusion on the detection probabilities of each detection unit to obtain the spatial distribution of joint detection probabilities in the target sea area; an effective detection threshold is preset, and the grid set in the target sea area with a joint detection probability greater than the effective detection threshold is taken as the effective detection range of the unmanned swarm at the current moment. Based on the updated attraction and repulsion factors of each grid, the comprehensive pheromone concentration of each grid is updated, and the comprehensive pheromone concentration of each grid is adjusted according to the binary threshold, thus transforming the detection perception map into a detection binary map. In an unmanned swarm, a surface unmanned platform extracts a local observation map vector from the binary map of the detection system. The position, heading, speed, local observation map vector of the surface unmanned platform, and the positions of all other surface unmanned platforms in the swarm are used to construct the local observation space of the surface unmanned platform. The local observation space of the surface unmanned platform at the current moment is input into the policy network of the surface unmanned platform updated at the previous moment to obtain the actions of the surface unmanned platform.

2. The method for full-domain coverage of unmanned swarms based on reinforcement learning according to claim 1, characterized in that: The unmanned surface platforms in the unmanned swarm extract local observation map vectors from the binary map they are exploring, specifically as follows: For the first in the unmanned cluster Unmanned surface platform, with the first Unmanned surface platforms Location at any moment Centered on, in probing the binary map direction and Detection distance thresholds are extracted in each direction. The grid within the area forms a local detection binary map; It is a positive integer; For the first Detection distance threshold for unmanned surface platforms; The local detection binary map is arranged according to the side length Divide the map evenly into sub-maps and obtain the number of grid cells with a combined pheromone concentration of 1 within each sub-map. And the number of grid cells with a total pheromone concentration of 0. ; , For the first Unmanned surface platforms The number of sub-maps divided by the local probe binary map at any given time; Calculate the first The first local detection binary map of the unmanned surface platform The number of grid cells in the submap with a combined pheromone concentration of 1. Total number of grids in the target sea area proportion ; All submaps Constructed as the first Unmanned surface platforms Local observation map vector at time .

3. The global coverage method for unmanned cluster based on reinforcement learning according to claim 2, characterized in that: The reward function is: wherein with is a weight coefficient; For the first Unmanned surface platform Time-based coverage rewards; For the first Water unmanned platforms in Risk of collision at all times punishment; For the first Water surface unmanned platform and the first Water surface unmanned platform collision risk penalty; 。 4. The method for full-domain coverage of unmanned swarms based on reinforcement learning according to claim 2, characterized in that: The loss function includes the loss function of the policy network and the loss function of the value network; The current unmanned surface platform Local observation space at time Input the policy network of the unmanned surface platform updated at the previous time step. The actions of the unmanned surface platform were obtained during the process. ; The loss function of the policy network is: in, It is the mean operator; For the first Unmanned surface platform Constantly updated policy network Based on the input local observation space Output Action The probability of; For the first Unmanned surface platform Constantly updated policy network Based on the input local observation space Output Action The probability of; For the clipping function, These are the trimming parameters; For the first Unmanned surface platform The generalized dominance estimate at time t represents the first time. Actual actions performed by unmanned surface platforms The relative value compared to the expected action; The loss function of the value network is: in, For the first Unmanned surface platform Cumulative rewards over time ; For the first Unmanned surface platform A constantly updated value network Based on the input local observation space The output reward value.

5. A computer device comprising a memory, a processor, and a computer program stored on the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

6. A computer readable storage medium having stored thereon a computer program, characterized in that: When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4.

7. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1 to 4.

Images