Cross-domain multi-agent collaborative search method for dynamic marine environment

Abstract

The invention relates to a cross-domain multi-agent collaborative search method for a dynamic marine environment, which comprises the following steps of: constructing an initial probability distribution diagram by taking a target position as a center, and generating a dynamic probability distribution diagram evolved along with time in combination with real-time ocean current data and a diffusion probability model; adaptive region allocation is carried out by adopting an improved weighted clustering algorithm, and each sub-region is allocated to the unmanned aerial vehicle to execute a search task; each unmanned aerial vehicle performs target searching and positioning in the distributed area, and transmits the identified target position information to the unmanned surface vessel; and after the unmanned surface vehicle receives the target positions, dynamically traversing all the target positions and implementing accurate rescue by adopting a path planning algorithm based on a prediction-correction-tracking strategy. According to the method, the whole process from target searching, positioning to rescue is coordinated, and after all targets are successfully traversed and rescue is completed, the task is ended. And through an autonomous controllable intelligent algorithm and a heterogeneous robot cooperation mechanism, the search efficiency and the dynamic adaptability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous unmanned system collaborative control, and in particular to a cross-domain multi-agent collaborative search method for dynamic ocean environments. Background Art

[0002] In maritime search and rescue scenarios, a rapidly responsive, highly efficient, and collaborative intelligent rescue system is a key requirement for safeguarding lives and property and advancing the modernization of emergency management. However, current technologies still face significant challenges in multi-agent collaborative control and adaptability to dynamic environments. First, traditional search and rescue methods rely on static environmental assumptions, making them difficult to account for the impact of dynamic factors such as ocean currents and waves on target locations. However, existing region division strategies are still based on initial target distributions and lack real-time response to target drift and sudden position changes, resulting in an imbalanced allocation of task sub-regions. Furthermore, existing path planning algorithms lack the ability to integrate dynamic probability maps, making them unable to adapt to the non-uniform diffusion of targets in maritime disaster scenarios driven by ocean currents, thus limiting search and rescue efficiency. Second, existing methods exhibit significant deficiencies in multi-agent collaborative mechanisms. Heterogeneous agents (such as drones and unmanned boats) often experience overlapping or incomplete task areas due to a lack of global information sharing and dynamic coordination strategies. This inefficient collaborative model fails to meet the requirements of an intelligent, networked, and collaborative emergency rescue system. To address these challenges, this paper proposes a multi-agent collaborative search method for dynamic ocean environments, combining deep reinforcement learning, dynamic probability diffusion models, and cross-domain collaborative control technology. By leveraging autonomous and controllable intelligent algorithms and heterogeneous robot collaboration, this method significantly improves search efficiency and dynamic adaptability. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention provides a cross-domain multi-agent collaborative search method for dynamic ocean environments. This method addresses issues such as overlapping regions, delayed dynamic target response, and low collaborative efficiency in multi-agent collaborative search tasks. This method integrates an improved weighted K-means algorithm, a deep reinforcement learning path planning strategy, and a target traversal algorithm using a prediction-correction-tracking mechanism to construct a highly efficient and collaborative search and rescue system. First, the search area is modeled as a two-dimensional space, and an initial Gaussian distribution probability map is constructed centered on the target location. This map is dynamically updated based on target motion using real-time ocean current data and a diffusion model, generating a probability distribution map that evolves over time. This probability map is then used to divide the task into sub-areas. An improved weighted K-means algorithm is then used for adaptive region allocation, and each sub-area is assigned to an unmanned aerial vehicle (UAV) for search tasks. Within the assigned area, each UAV searches and locates the target using a deep reinforcement learning strategy and transmits the identified target location information to an unmanned surface vehicle (UAV). Upon receiving the target information, the UAV uses a path planning algorithm based on a prediction-correction-tracking strategy to dynamically traverse all target locations and conduct precise rescue operations.

[0004] To solve the above technical problems, the present invention provides the following technical solution: a cross-domain multi-agent collaborative search method for a dynamic ocean environment, comprising the following steps:

[0005] S1. Model the search area as a two-dimensional space and construct an initial probability distribution map centered on the target location. Combine real-time ocean current data with the diffusion probability model to dynamically update the target motion and generate a dynamic probability distribution map that evolves over time.

[0006] S2: Divide the task sub-areas based on the dynamic probability distribution map, use the improved weighted K-means clustering algorithm for adaptive area allocation, and assign each sub-area to the UAV to perform the search mission; each UAV searches and locates the target in the assigned area based on the deep reinforcement learning strategy, and transmits the identified target location information to the surface unmanned vehicle;

[0007] S3. After receiving the target location, the surface unmanned boat adopts a path planning algorithm based on the prediction-correction-tracking strategy to dynamically traverse all target locations and implement precise rescue.

[0008] Furthermore, the specific process of step S1 includes the following steps:

[0009] S11, receive the accident coordinates from GPS positioning, and obtain the coordinates of the accident center x0, y0; calculate the probability value of each grid according to the Gaussian distribution formula, and set the attenuation coefficient σ to generate the initial target existence probability P i,j (0);

[0010] S12, satellite or sensor acquires ocean current vector field in real time, loads real-time ocean current data, and obtains ocean current velocity vector V c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c (i,j)]; where v c (i,j) represents the ocean current velocity at the grid (i,j), θ c (i, j) indicates the direction of ocean current;

[0011] S13. Calculate the transfer weight of adjacent grids according to the direction of ocean current to obtain the diffusion weight w from grid m,n to i,j m,n→i,j ;

[0012] S14, perform diffusion probability calculation, set the time step, and update the diffusion probability distribution;

[0013] S15. Perform update cycle judgment. If no update is required, output the dynamic probability distribution map of the high probability band along the ocean current direction. If update is required, repeat S12-S15.

[0014] Furthermore, in step S15, a dynamic probability distribution diagram of the high probability band along the ocean current direction is output. The specific process includes: after each time step Δt, the system adjusts the probability value of each grid according to the ocean current direction, considering the driving effect of the ocean current on the target position. The target drifts with the ocean current, resulting in dynamic changes in the probability distribution. The probability diffusion model needs to reflect the dominant role of the ocean current direction. The probability diffusion direction and rate are dominated by the ocean current vector field. The probability diffusion model is expressed as:

[0015]

[0016] γ=k·||V c (i,j)||

[0017] in, Represents the probability of inflow, the probability P of the neighborhood grid m,n m,n (t) According to the diffusion weight w m,n→i,j Flow into the current grid i,j; γ·P i,j (t) represents the probability of the current grid i, j flowing out to other grids at rate γ, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, represents the set of adjacent grids of the current grid i, j, w m,n→i,j represents the diffusion weight from grid m,n to i,j, which only allows diffusion along the direction of ocean current and its lateral direction, and the diffusion weight against the ocean current is 0, ∑ (m,n) w m,n→i,j Represents the sum of the diffusion weights of all neighboring grids to the current grid, normalizes the probability contribution of the neighboring grid, and avoids the probability value exceeding the reasonable range due to weight difference. c (m,n) represents the ocean current velocity vector of the neighborhood grid m,n, V c (i, j) represents the ocean current velocity vector of the current grid i, j, k represents the proportional coefficient, which adjusts the contribution weight of the ocean current velocity to the diffusion rate, and γ is the diffusion rate, which is proportional to the ocean current velocity at the current grid. The stronger the ocean current, the more significant the probability diffusion.

[0018] Furthermore, in step S2, the specific process includes the following steps:

[0019] S21, UAV receives the dynamic probability distribution map of the high probability band by the improved weighted K-means algorithm and divides it into sub-region probability distribution matrix P k (t) and initial dynamic probability map;

[0020] S22, rasterize the probability matrix, and convert the sub-area probability distribution matrix P of the partition that the current drone is responsible for k (t) is converted into a two-dimensional tensor and the normalized relative coordinates of the UAV and the time step are concatenated to obtain the state feature vector;

[0021] S23, taking the state feature vector as the input of the neural network, outputs the Q value of 8 movement directions through the fully connected layer to determine the movement direction of the drone;

[0022] S24. Set the initial exploration rate ∈ and the decay rate λ, select an action with the strategy ε-greedy, and determine whether the action is safe. If the action is unsafe, select the suboptimal action. If the action is safe, execute the action and update the state.

[0023] S25. The interaction data of all drones are stored in the shared experience pool D, and priority sampling is used to update the experience pool;

[0024] S26. Determine whether the UAV has found the target. If so, transmit the target's location information to the surface unmanned boat. If not, repeat steps S21-S26.

[0025] Furthermore, step S21 specifically includes: designing a clustering objective function by using a weighted K-means clustering algorithm:

[0026]

[0027] in, is the normalized probability of the target existing at grid point i, j in the entire search area, C k is the kth cluster, μ k is the cluster center; α, β, δ are weight coefficients, which respectively represent the regional geographical compactness, probability distribution balance and the degree of ocean current influence; DriftCost (C k ) indicates that the area far away from the cluster center and with high ocean current speed is penalized, and the downstream high probability area is preferentially assigned to the UAV to prevent it from being carried away from the mission area by the ocean current. Finally, the sub-area probability distribution matrix P is updated to obtain k (t).

[0028] Furthermore, the specific process of step S23 includes the following steps:

[0029] Each drone independently maintains a DQN network, whose input state feature vector is composed of the normalized sub-region probability distribution matrix P k (t), drone status and time step composition, the network structure is divided into two parts: feature extraction and decision output: the feature extraction layer uses convolutional neural network CNN to process the normalized sub-region probability distribution matrix CNN consists of two layers of convolution and one layer of maximum pooling, which are used to extract spatial features in the probability map. The decision output layer combines the feature vector output by CNN with the normalized relative coordinates of the drone x. rel ,y rel and time step After concatenation, the input is fed into the fully connected layer FC to generate the Q value. The final output is the Q value corresponding to the movement action in eight directions, namely up, down, left, right and four diagonal directions. The formula is as follows:

[0030]

[0031] in, is the sub-region state input, a i Represents the historical actions of drone i, is the probability distribution matrix assigned to the i-th sub-region of the k-th UAV, and the sub-region probability distribution matrix is normalized to Ensure that the value range is [0,1], the relative coordinate x of the drone in the sub-area rel ,y rel Normalized to [0,1], the current time step t is normalized to Time step t is used for probability diffusion model update.

[0032] Furthermore, for step S24, an adaptive ε-greedy strategy is adopted to balance exploration and exploitation. In the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas. As the number of training steps increases, the exploration rate ∈ is adjusted according to exponential decay:

[0033] ∈(t)=∈ min +(∈ max -∈ min )·e -λt

[0034] Among them, λ controls the decay rate and eventually stabilizes at ∈ min ; Add action correction. If the drone detects an obstacle, it blocks the direction of action that may cause a collision and reselects a safe action with a suboptimal Q value. In multi-agent collaborative search tasks, the reward function must take into account both individual efficiency and team collaboration. The collaborative reward function includes: search reward, area coverage reward, and collaborative penalty. The specific design is as follows:

[0035] Search reward: When a drone scans a target for the first time, it will be given a fixed positive reward to encourage exploration behavior; if the target has been marked by another drone, it will only be given zero reward to avoid wasting resources:

[0036]

[0037] Where λ1 is the reward coefficient, r baseAs a basic reward;

[0038] Area coverage reward: weighted by probability density, based on the probability P of the target existing at the current grid point i, j in the entire search area i,j Dynamically adjust rewards, and gain higher rewards for covering high-probability areas; add a time decay factor: gradually increase the reward value for areas that have not been searched for a long time to avoid ignoring low-probability areas:

[0039] R coverage =λ2·∑ (i,j)∈Visited (P i,j (t)·e -η·Δt )

[0040] Among them, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, λ2 is the coverage bonus coefficient, η is the time decay rate, and Δt is the number of time steps that the grid has not been searched;

[0041] Collaboration penalty: If multiple drones enter the same sub-area at the same time step, a penalty mechanism is triggered; the penalty value is proportional to the number of overlaps, strengthening the collaborative awareness of decentralized search:

[0042]

[0043] Where λ3 is the penalty coefficient, OverlapCount(k) is the number of overlapping drones in the kth sub-region;

[0044] The weighted sum of the three rewards balances individual exploration and team collaboration to obtain the final reward function:

[0045] R total =R discover +R coverage +R penalty .

[0046] Furthermore, for step S25, the specific design is as follows:

[0047] Centralized experience pool: interaction data of all drones i ,a i ,r i ,s' i Stored in the shared experience pool D; during training, batch data is randomly sampled to break data correlation, and priority sampling is used to improve the utilization of high TD error samples and accelerate convergence:

[0048] D={(s i ,a i ,r i ,s′ i )|i=1,2,…,N}

[0049] Among them, s i is the sub-region state input, s' i is the next state, containing the updated sub-region probability distribution matrix

[0050] Global state perception: During the training phase, the Q network of drone i can access the historical actions of other drones {a j} j≠i , dynamically weighting the impact of different drones’ actions through the attention mechanism:

[0051]

[0052] where {a j} Obtained through shared experience pool D, s i represents the state vector of UAV i, ∑ j≠i α j a j represents the weighted sum of other UAV actions, reflecting their collaborative impact on the current UAV, α j is the attention weight, reflecting the importance of UAV j to the collaboration of i, a i represents the historical actions of drone i;

[0053] The joint loss function is that the training goal of each drone is to minimize the temporal difference error containing collaborative information, and the loss is minimized by gradient descent to update the current network parameters θ i , improve the accuracy of action decision:

[0054]

[0055] Among them, θ i Represents the parameters of the current network, updated by gradient descent, r i Indicates that drone i is in state s i Execute action a i After the immediate reward, s' i Indicates that drone i is in the current state s i Next, perform action a i The next state that the environment transfers to is a' i Indicates that drone i is in the next state s' i The optimal action selected under i is the target network parameter, φ is the discount factor; using the soft update strategy, the target network parameter θ' i Synchronize from the current network every C steps:

[0056] θ′ i ←τθ i +(1-τ)θ′i .

[0057] Wherein, τ represents the soft update coefficient 0<τ<<1, which controls the update rate of the target network parameters.

[0058] Furthermore, the specific process of step S3 includes the following steps:

[0059] S31. The surface unmanned boat obtains all target location information searched by the UAV;

[0060] S32, performing Kalman filtering on the target location information searched by the UAV to predict the next location information of the target;

[0061] S33. Sort the traversal order of the multiple targets, calculate the Manhattan distance, and prioritize searching for targets moving toward the surface unmanned boat based on prioritizing traversal of targets with a short Manhattan distance;

[0062] S34, perform path planning, select the A* algorithm, generate the shortest path, and avoid known obstacles;

[0063] S35, determine whether the prediction error is greater than the set threshold, if less, continue to traverse the target, if greater, re-plan the traversal path, and repeat S34-S35;

[0064] S36. Continue traversing the target and make real-time path corrections. If the target suddenly accelerates or turns, use the Manhattan distance gradient descent method to adjust the direction of the surface unmanned boat.

[0065] S37, judging whether all targets have been traversed. If the surface unmanned boat has traversed all targets and rescued all targets, the mission is completed. Otherwise, repeat S34-S37.

[0066] By means of the above technical solution, the present invention provides a cross-domain multi-agent collaborative search method for dynamic ocean environments, which has at least the following beneficial effects:

[0067] Compared with the existing technology, the present invention has the following advantages: the present invention realizes adaptive division of task areas and load balancing through an improved weighted K-means algorithm combined with a probability diffusion mechanism; it integrates deep reinforcement learning with probability graphs and designs a collaborative reward function to take into account both individual exploration and team collaboration; the surface unmanned boat adopts a prediction-correction-tracking strategy, combined with Kalman filtering and A* algorithm, to achieve efficient traversal and precise rescue of multiple dynamic targets, significantly improving search and rescue efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0069] Figure 1 A schematic block diagram of the multi-agent collaborative search system provided by the present invention;

[0070] Figure 2 A schematic diagram of the dynamic probability map generation process provided by the present invention;

[0071] Figure 3 A schematic diagram of the target search process of the UAV provided by the present invention;

[0072] Figure 4 This is a schematic diagram of the target traversal process of the unmanned boat provided by the present invention. DETAILED DESCRIPTION

[0073] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0074] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] Please refer to Figures 1-4 This example illustrates a specific implementation of this embodiment, which coordinates the entire process from target search and positioning to rescue. The mission ends when all targets are successfully traversed and rescue is complete. By leveraging autonomous and controllable intelligent algorithms and heterogeneous robot collaboration, search efficiency and dynamic adaptability are significantly improved.

[0076] Please refer to Figure 1 This embodiment proposes a cross-domain multi-agent collaborative search method for a dynamic ocean environment, which includes the following steps:

[0077] S1. Model the search area as a two-dimensional space and construct an initial probability distribution map centered on the target location. Combine real-time ocean current data with the diffusion probability model to dynamically update the target motion and generate a dynamic probability distribution map that evolves over time.

[0078] As a preferred embodiment of step S1, Figure 2 As shown, the specific process of step S1 includes the following steps:

[0079] S11. First, discretize the sea surface search area into M×N grid areas to obtain the grid coordinate set X={(x1,y1),(x2,y2),…,(x i ,y j )}, where (x i ,y j )∈R 2 Represents a two-dimensional grid coordinate. After the accident occurs, the accident coordinates are received from GPS positioning, and the coordinates of the accident center are obtained. The probability value of each grid is calculated according to the Gaussian distribution formula, and the attenuation coefficient σ is set to generate the initial target existence probability P i,j (0), marking high-priority areas where targets may exist, which serves as an important basis for subsequent search path planning. Assuming that the initial distribution of targets is concentrated near the accident center, the Gaussian distribution model is used to model the probability of the initial target existence, and the probability decays exponentially with the increase of the distance from the center:

[0080]

[0081] Among them, P i,j represents the probability of the target existing in the grid (i, j), P max represents the maximum probability value, that is, the highest probability of the target existing in the accident center area. σ controls the probability decay range. A larger value indicates a more dispersed target distribution. The farther the grid is from the accident center, the lower the probability of the target existing, forming a probability distribution with the accident center as the "hotspot".

[0082] S12. The ocean current vector field is the core input of the probability diffusion mechanism. Satellites or sensors obtain the ocean current vector field in real time to ensure that the probability map can reflect the drift trend of the target. In order to quantify the impact of ocean currents on the target position, real-time ocean current data is loaded, the ocean current vector field is modeled, and a gridded ocean current vector field is constructed to obtain the ocean current velocity vector: V c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c (i,j)]; where v c (i,j) represents the ocean current velocity at the grid (i,j), θ c (i, j) represents the direction of the ocean current; the ocean current velocity is decomposed into components in the x and y directions to calculate the probability diffusion direction;

[0083] S13. Calculate the transfer weight of adjacent grids according to the direction of ocean current to obtain the diffusion weight w from grid m,n to i,j m,n→i,j;

[0084] S14, perform diffusion probability calculation, set the time step, and update the diffusion probability distribution;

[0085] S15. Perform update cycle judgment. If no update is required, output the dynamic probability distribution map of the high probability band along the ocean current direction. If update is required, repeat S12-S15.

[0086] More specifically, in step S15, a dynamic probability distribution diagram of the high probability band along the ocean current direction is output. The specific process includes: after each time step Δt, the system adjusts the probability value of each grid according to the ocean current direction, considering the driving effect of the ocean current on the target position. The target drifts with the ocean current, resulting in dynamic changes in the probability distribution. The probability diffusion model needs to reflect the dominant role of the ocean current direction. The probability diffusion direction and rate are dominated by the ocean current vector field. The probability diffusion model is expressed as:

[0087]

[0088] γ=k·||V c (i,j)||

[0089] in, Represents the probability of inflow, the probability P of the neighborhood grid m,n m,n (t) According to the diffusion weight w m,n→i,j Flow into the current grid i,j; γ·P i,j (t) represents the probability of the current grid i, j flowing out to other grids at rate γ, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, represents the set of adjacent grids of the current grid i, j, w m,n→i,j represents the diffusion weight from grid m,n to i,j, which only allows diffusion along the direction of ocean current and its lateral direction, and the diffusion weight against the ocean current is 0, ∑ (m,n) w m,n→i,j Represents the sum of the diffusion weights of all neighboring grids to the current grid, normalizes the probability contribution of the neighboring grid, and avoids the probability value exceeding the reasonable range due to weight difference. c (m,n) represents the ocean current velocity vector of the neighborhood grid m,n, V c (i, j) represents the ocean current velocity vector of the current grid i, j, k represents the proportional coefficient, which adjusts the contribution weight of the ocean current velocity to the diffusion rate, and γ is the diffusion rate, which is proportional to the ocean current velocity at the current grid. The stronger the ocean current, the more significant the probability diffusion. This mechanism forms a high-probability belt along the direction of the ocean current, reflecting the possible drift trajectory of the target.

[0090] S2. To further improve search efficiency, the task sub-areas are divided based on the dynamic probability distribution map. An improved weighted K-means clustering algorithm is used for adaptive area allocation. Each sub-area is assigned to a UAV to perform the search task. Each UAV searches and locates the target within the assigned area based on a deep reinforcement learning strategy and transmits the identified target location information to the surface unmanned vehicle.

[0091] As a preferred embodiment of step S2, in step S2, as Figure 3 As shown, the specific process includes the following steps:

[0092] S21, UAV receives the dynamic probability distribution map of the high probability band by the improved weighted K-means algorithm and divides it into sub-region probability distribution matrix P k (t) and initial dynamic probability map;

[0093] More specifically, step S21 specifically includes: designing a clustering objective function by introducing target probability distribution, ocean current influence, and load balancing weights through a weighted K-means clustering algorithm:

[0094]

[0095] in, is the normalized probability of the target existing at grid point i, j in the entire search area, C k is the kth cluster, μ k is the cluster center; α, β, δ are weight coefficients, which respectively represent the regional geographical compactness, probability distribution balance and the degree of ocean current influence; DriftCost (C k ) indicates that the area far away from the cluster center and with high ocean current speed is penalized, and the downstream high probability area is preferentially assigned to the UAV to prevent it from being carried away from the mission area by the ocean current. Finally, the sub-area probability distribution matrix P is updated to obtain k (t).

[0096] In this embodiment, in order to find the target faster, the distance of each point is weighted by its probability, so that the points with high probability have a greater impact on the cluster center, ensuring that the high probability area is more compact; in order to promote the balance of the total probability of each cluster, that is, to ensure the load balance of each drone and avoid overload of tasks in some areas, that is, the total probability of each area is close, the balance of probability distribution is taken into account, and the square of the difference between the total probability of each cluster and the ideal average value is calculated.

[0097] S22. After completing the area division and assigning the sub-areas to each drone, each drone must prioritize the search for high-probability target areas within its responsible area. This invention uses the DQN algorithm as its core to guide drones to optimize their search strategies based on their current state. The drones construct a high-dimensional input state vector by perceiving the local probability map and their own state (position, time) in real time. The probability map is the core input of the DQN algorithm and needs to be structured to represent the data in a form that can be processed by the neural network. Therefore, the probability matrix is rasterized and the probability distribution matrix P of the sub-region of the partition (M×M grid) that the current drone is responsible for is converted into k (t) is converted to a two-dimensional tensor:

[0098]

[0099] Secondly, the probability P of the target existing at the current grid point i, j in the entire search area at time step t is required i,j (t) is normalized to ensure that the value range is within [0,1], so the probability calculation formula for each grid point in the matrix is as follows:

[0100]

[0101] Finally, construct the input tensor and put all the normalized probability points in the grid The normalized probability distribution matrix is obtained by aggregation Flatten it into a vector and concatenate it with the drone state (position, time) to form the complete input state:

[0102]

[0103] Among them, the relative coordinates of the UAV in the partition (x rel ,y rel ) is normalized to [0,1], and the current time step t is normalized to The M×M probability matrix of the current partition of the drone is normalized and flattened to M 2 dimensional vector, and the relative coordinates of the drone (x rel ,y rel ) and time step t to form M 2 +3D input state. At this point, this complete state feature vector is used as input and transmitted to the neural network of the DQN algorithm.

[0104] In this embodiment, this design enables the neural network to simultaneously capture the probability distribution and spatiotemporal dynamic characteristics of the environment.

[0105] S23, taking the state feature vector as the input of the neural network, outputting the Q value of 8 movement directions through the fully connected layer to determine the movement direction of the drone;

[0106] To solve the problem of efficient collaborative search for dynamic targets by multiple drones in an unknown environment, this paper proposes a multi-agent collaborative search method that integrates deep reinforcement learning (DQN) and dynamic probabilistic graphs. This method comprehensively utilizes a dynamic probabilistic mapping mechanism to guide search area division, and combines information sharing mechanisms and strategy coupling mechanisms between multiple drones to achieve task collaboration and path optimization among drones. More specifically, step S23 includes the following steps:

[0107] Each drone independently maintains a DQN network, whose input state feature vector is composed of the normalized sub-region probability distribution matrix P k (t), drone status and time step composition, the network structure is divided into two parts: feature extraction and decision output: the feature extraction layer uses convolutional neural network CNN to process the normalized sub-region probability distribution matrix CNN consists of two layers of convolution and one layer of maximum pooling, which are used to extract spatial features in the probability map. The decision output layer combines the feature vector output by CNN with the normalized relative coordinates of the drone x. rel ,y rel and time step After concatenation, the input is fed into the fully connected layer FC to generate the Q value. The final output is the Q value corresponding to the movement action in eight directions, namely up, down, left, right and four diagonal directions. The formula is as follows:

[0108]

[0109] in, is the sub-region state input, a i Represents the historical actions of drone i, is the probability distribution matrix assigned to the i-th sub-region of the k-th UAV, and the sub-region probability distribution matrix is normalized to Ensure that the value range is [0,1], the relative coordinate x of the drone in the sub-area rel ,y rel Normalized to [0,1], the current time step t is normalized to Time step t is used for probability diffusion model update.

[0110] S24. Set the initial exploration rate ∈ and the decay rate λ, select an action with the strategy ε-greedy, and determine whether the action is safe. If the action is unsafe, select the suboptimal action. If the action is safe, execute the action and update the state.

[0111] More specifically, for step S24, in order to better select the action of the UAV, an adaptive ε-greedy strategy is adopted to balance exploration and exploitation. In the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas. As the number of training steps increases, the exploration rate ∈ is adjusted according to exponential decay:

[0112] ∈(t)=∈ min +(∈ max -∈ min )·e -λt

[0113] Among them, λ controls the decay rate and eventually stabilizes at ∈ min ; Add action correction. If the drone detects an obstacle, it blocks the direction of action that may cause a collision and reselects a safe action with a suboptimal Q value. In multi-agent collaborative search tasks, the reward function must take into account both individual efficiency and team collaboration. The collaborative reward function includes: search reward, area coverage reward, and collaborative penalty. The specific design is as follows:

[0114] Search reward: When a drone scans a target for the first time, it will be given a fixed positive reward to encourage exploration behavior; if the target has been marked by another drone, it will only be given zero reward to avoid wasting resources:

[0115]

[0116] Where λ1 is the reward coefficient, r base As a basic reward;

[0117] Area coverage reward: weighted by probability density, based on the probability P of the target existing at the current grid point i, j in the entire search area i,j Dynamically adjust rewards, and gain higher rewards for covering high-probability areas; add a time decay factor: gradually increase the reward value for areas that have not been searched for a long time to avoid ignoring low-probability areas:

[0118] R coverage =λ2·∑ (i,j)∈Visited (P i,j (t)·e -η·Δt )

[0119] Among them, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, λ2 is the coverage bonus coefficient, η is the time decay rate, and Δt is the number of time steps that the grid has not been searched;

[0120] Collaboration penalty: If multiple drones enter the same sub-area at the same time step, a penalty mechanism is triggered; the penalty value is proportional to the number of overlaps, strengthening the collaborative awareness of decentralized search:

[0121]

[0122] Where λ3 is the penalty coefficient, OverlapCount(k) is the number of overlapping drones in the kth sub-region;

[0123] The weighted sum of the three rewards balances individual exploration and team collaboration to obtain the final reward function:

[0124] R total =R discover +R coverage +R penalty .

[0125] S25. The interaction data of all drones are stored in the shared experience pool D, and priority sampling is used to update the experience pool;

[0126] In order to improve the efficiency of multi-UAV collaboration, a centralized experience pool and a global state perception mechanism are adopted. More specifically, step S25 is designed as follows:

[0127] Centralized experience pool: interaction data of all drones i ,a i ,r i ,s' i Stored in the shared experience pool D; during training, batch data is randomly sampled to break data correlation, and priority sampling is used to improve the utilization of high TD error samples and accelerate convergence:

[0128] D={(s i ,a i ,r i ,s′ i )|i=1,2,…,N}

[0129] Among them, s i is the sub-region state input, s' i is the next state, containing the updated sub-region probability distribution matrix

[0130] Global state perception: During the training phase, the Q network of drone i can access the historical actions of other drones {a j} j≠i , dynamically weighting the impact of different drones’ actions through the attention mechanism:

[0131]

[0132] where {a j} Obtained through shared experience pool D, s i represents the state vector of UAV i, ∑ j≠i α j a j represents the weighted sum of other UAV actions, reflecting their collaborative impact on the current UAV, α j is the attention weight, reflecting the importance of UAV j to the collaboration of i, a i represents the historical actions of drone i;

[0133] The joint loss function is that the training goal of each drone is to minimize the temporal difference error containing collaborative information, and the loss is minimized by gradient descent to update the current network parameters θ i , improve the accuracy of action decision:

[0134]

[0135] Among them, θ i Represents the parameters of the current network, updated by gradient descent, r i Indicates that drone i is in state s i

[0136] Execute action a i After the immediate reward, s' i Indicates that drone i is in the current state s i Next, perform action a i The next state that the environment transfers to is a' i Indicates that drone i is in the next state s' i The optimal action selected under i is the target network parameter, φ is the discount factor; using the soft update strategy, the target network parameter θ' i Synchronize from the current network every C steps:

[0137] θ′ i ←τθ i +(1-τ)θ′ i .

[0138] Wherein, τ represents the soft update coefficient (0<τ<<1), which controls the update rate of the target network parameters.

[0139] S26. Determine whether the UAV has found the target. If so, transmit the target's location information to the surface unmanned boat. If not, repeat steps S21-S26.

[0140] S3. After receiving the target location, the surface unmanned boat adopts a path planning algorithm based on the prediction-correction-tracking strategy to dynamically traverse all target locations and implement precise rescue.

[0141] As a preferred embodiment of step S3, Figure 3 As shown, the specific process of step S3 includes the following steps:

[0142] S31. The surface unmanned boat obtains all target location information searched by the UAV;

[0143] S32. Based on the target position information searched by the UAV, a Kalman filter is used to predict the next position information of the target. First, the dynamic target position is predicted, the target position prediction is corrected according to the ocean current, and the target position within the next τ time step is predicted using the Kalman filter algorithm:

[0144]

[0145] Among them, g r (t) represents the target position at the current moment, represents the predicted future position, v r Indicates the target's own drift speed, which can be ignored, that is, set to 0, V c (g r (t)) represents the ocean current velocity vector at the current position of the target;

[0146] S33. Sort the traversal order of multiple targets and calculate the Manhattan distance. Since there are multiple dynamic targets, it is necessary to assign priorities to the dynamic targets and sort them according to the following rules. First, the distance priority needs to be considered, that is, the target with the closest Manhattan distance is traversed first. On the basis of prioritizing the traversal of the target with the shortest Manhattan distance, the movement trend priority needs to be considered, and the target moving toward the surface unmanned boat is searched first.

[0147] S34. After obtaining the predicted target position, the surface unmanned vehicle needs to generate a traversal path. To achieve the shortest and safest path planning, path planning is performed, and the A* algorithm is selected to generate the shortest path while avoiding known obstacles. The present invention adopts the heuristic search algorithm A* and constructs a path cost function based on obstacle avoidance:

[0148]

[0149] in, is the path length, is the avoidance cost of the obstacle area passed by the path, and α is the cost weight coefficient, which is used to balance the shortest path and safety.

[0150] S35, determine whether the prediction error is greater than the set threshold, if less, continue to traverse the target, if greater, re-plan the traversal path, repeat S34-S35; Considering the uncertainty of dynamic targets, the surface unmanned vehicle needs to make continuous corrections when executing the path. Every Δt time step, the surface unmanned vehicle receives the current actual position of the target g r (t), calculate the prediction error:

[0151]

[0152] Among them, g r (t) represents the actual position coordinate of the dynamic target at time step t, represents the predicted position coordinates of the dynamic target at time step t. If the error e(t) exceeds the threshold, the path replanning mechanism is triggered, the target predicted position and model parameters are updated, and the A* path planning is re-executed.

[0153] S36. Continue traversing the target and make real-time path corrections. If the target suddenly accelerates or turns, the Manhattan distance gradient descent method is used to adjust the direction of the surface unmanned vehicle. The actual action selection of the surface unmanned vehicle is dominated by the predicted path and dynamically adjusted according to the movement of the target: the surface unmanned vehicle preferentially executes the next action along the planned path. If the target suddenly deviates or accelerates, the Manhattan distance gradient descent method is used to quickly adjust the direction to shorten the distance between the current position and the target. The specific calculation is as follows:

[0154]

[0155] in, Indicates the current position of the surface unmanned boat; The coordinate component representing the target predicted position; sign(·) is a sign function whose output is -1, 0, or +1, which determines the direction of movement.

[0156] S37, judging whether all targets have been traversed. If the surface unmanned boat has traversed all targets and rescued all targets, the mission is completed. Otherwise, repeat S34-S37.

[0157] In this embodiment, after multiple drones complete their search for all dynamic targets and obtain their precise locations, the surface unmanned vehicle (UAV) takes over the subsequent rapid traversal and rescue missions. This invention designs an efficient traversal strategy for dynamic targets, integrating target state prediction, path planning, and dynamic correction. This prediction-correction-tracking strategy ensures that the surface UAV can rapidly traverse multiple targets and achieve precise and efficient rescue in complex environments.

[0158] The beneficial effects of the present invention are: compared with the existing technology, it has the following advantages: the present invention realizes adaptive division of task areas and load balancing through an improved weighted K-means algorithm combined with a probability diffusion mechanism; integrates deep reinforcement learning and probability graphs, and designs a collaborative reward function to take into account both individual exploration and team collaboration; the surface unmanned boat adopts a prediction-correction-tracking strategy, combined with Kalman filtering and A* algorithm, to achieve efficient traversal and precise rescue of multiple dynamic targets, significantly improving search and rescue efficiency.

[0159] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0161] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A cross-domain multi-agent collaborative search method for dynamic ocean environments, characterized by: The following steps are involved: S1. Model the search area as a two-dimensional space and construct an initial probability distribution map centered on the target location. Combine real-time ocean current data with the diffusion probability model to dynamically update the target motion and generate a dynamic probability distribution map that evolves over time. S2: Divide the task sub-areas based on the dynamic probability distribution map, use the improved weighted K-means clustering algorithm for adaptive area allocation, and assign each sub-area to the UAV to perform the search mission; each UAV searches and locates the target in the assigned area based on the deep reinforcement learning strategy, and transmits the identified target location information to the surface unmanned vehicle; S3. After receiving the target location, the surface unmanned boat adopts a path planning algorithm based on the prediction-correction-tracking strategy to dynamically traverse all target locations and implement precise rescue.

2. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 1, characterized in that: The specific process of step S1 includes the following steps: S11, receive the accident coordinates from GPS positioning, and obtain the coordinates of the accident center x0, y0; calculate the probability value of each grid according to the Gaussian distribution formula, and set the attenuation coefficient σ to generate the initial target existence probability P i,j (0); S12, satellite or sensor acquires ocean current vector field in real time, loads real-time ocean current data, and obtains ocean current velocity vector V c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c (i,j)]; where v c (i,j) represents the ocean current velocity at grid i,j, θ c (i, j) indicates the direction of ocean current; S13. Calculate the transfer weight of adjacent grids according to the direction of ocean current to obtain the diffusion weight w from grid m,n to i,j m,n→i,j ; S14, perform diffusion probability calculation, set the time step, and update the diffusion probability distribution; S15. Perform update cycle judgment. If no update is required, output the dynamic probability distribution map of the high probability band along the ocean current direction. If update is required, repeat S12-S15.

3. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 2, characterized in that: In step S15, the dynamic probability distribution diagram of the high probability zone along the ocean current direction is output. The specific process includes: after each time step Δt, the system adjusts the probability value of each grid according to the ocean current direction, considering the driving effect of the ocean current on the target position. The target drifts with the ocean current, resulting in dynamic changes in the probability distribution. The probability diffusion model needs to reflect the dominant role of the ocean current direction. The probability diffusion direction and rate are dominated by the ocean current vector field. The probability diffusion model is expressed as: γ=k·||V c (i,j)|| in, Represents the probability of inflow, the probability P of the neighborhood grid m,n m,n (t) According to the diffusion weight w m,n→i,j Flow into the current grid i,j; γ·P i,j (t) represents the probability of the current grid i, j flowing out to other grids at rate γ, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, represents the set of adjacent grids of the current grid i, j, w m,n→i,j represents the diffusion weight from grid m,n to i,j, which only allows diffusion along the direction of ocean current and its lateral direction, and the diffusion weight against the ocean current is 0, ∑ (m,n) w m,n→i,j Represents the sum of the diffusion weights of all neighboring grids to the current grid, normalizes the probability contribution of the neighboring grid, and avoids the probability value exceeding the reasonable range due to weight difference. c (m,n) represents the ocean current velocity vector of the neighborhood grid m,n, V c (i, j) represents the ocean current velocity vector of the current grid i, j, k represents the proportional coefficient, which adjusts the contribution weight of the ocean current velocity to the diffusion rate, and γ is the diffusion rate, which is proportional to the ocean current velocity at the current grid. The stronger the ocean current, the more significant the probability diffusion.

4. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 1, characterized in that: In step S2, the specific process includes the following steps: S21, UAV receives the dynamic probability distribution map of the high probability band by the improved weighted K-means algorithm and divides it into sub-region probability distribution matrix P k (t) and initial dynamic probability map; S22, rasterize the probability matrix, and convert the sub-area probability distribution matrix P of the partition that the current drone is responsible for k (t) is converted into a two-dimensional tensor and the normalized relative coordinates of the UAV and the time step are concatenated to obtain the state feature vector; S23, taking the state feature vector as the input of the neural network, outputting the Q value of 8 movement directions through the fully connected layer to determine the movement direction of the drone; S24. Set the initial exploration rate ∈ and the decay rate λ, select an action with the strategy ε-greedy, and determine whether the action is safe. If the action is unsafe, select the suboptimal action. If the action is safe, execute the action and update the state. S25. The interaction data of all drones are stored in the shared experience pool D, and priority sampling is used to update the experience pool; S26. Determine whether the UAV has found the target. If so, transmit the target's location information to the surface unmanned boat. If not, repeat steps S21-S26.

5. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 4, characterized in that: Step S21 specifically includes: designing a clustering objective function by using a weighted K-means clustering algorithm: in, is the normalized probability of the target existing at grid point i, j in the entire search area, C k is the kth cluster, μ k is the cluster center; α, β, δ are weight coefficients, which respectively represent the regional geographical compactness, probability distribution balance and the degree of ocean current influence; DriftCost (C k ) indicates that the area far away from the cluster center and with high ocean current speed is penalized, and the downstream high probability area is preferentially assigned to the UAV to prevent it from being carried away from the mission area by the ocean current. Finally, the sub-area probability distribution matrix P is updated to obtain k (t).

6. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 4, characterized in that: The specific process of step S23 includes: Each drone independently maintains a DQN network, whose input state feature vector is composed of the normalized sub-region probability distribution matrix P k (t), drone status and time step composition, the network structure is divided into two parts: feature extraction and decision output: the feature extraction layer uses convolutional neural network CNN to process the normalized sub-region probability distribution matrix CNN consists of two layers of convolution and one layer of maximum pooling, which are used to extract spatial features in the probability map. The decision output layer combines the feature vector output by CNN with the normalized relative coordinates of the drone x. rel ,y rel and time step After concatenation, the input is fed into the fully connected layer FC to generate the Q value. The final output is the Q value corresponding to the movement action in eight directions, namely up, down, left, right and four diagonal directions. The formula is as follows: in, is the sub-region state input, a i Represents the historical actions of drone i, is the probability distribution matrix assigned to the i-th sub-region of the k-th UAV, and the sub-region probability distribution matrix is normalized to Ensure that the value range is [0,1], the relative coordinate x of the drone in the sub-area rel ,y rel Normalized to [0,1], the current time step t is normalized to Time step t is used for probability diffusion model update.

7. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 4, characterized in that: For step S24, an adaptive ε-greedy strategy is adopted to balance exploration and exploitation; in the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas; As the number of training steps increases, the exploration rate ∈ is adjusted exponentially: ∈(t)=∈ min +(∈ max -∈ min )·e -λt Among them, λ controls the decay rate and eventually stabilizes at ∈ min ; Add action correction. If the drone detects an obstacle, it blocks the direction of action that may cause a collision and reselects a safe action with a suboptimal Q value. In multi-agent collaborative search tasks, the reward function must take into account both individual efficiency and team collaboration. The collaborative reward function includes: search reward, area coverage reward, and collaborative penalty. The specific design is as follows: Search reward: When a drone scans a target for the first time, it will be given a fixed positive reward to encourage exploration behavior; if the target has been marked by another drone, it will only be given zero reward to avoid wasting resources: Where λ1 is the reward coefficient, r base As a basic reward; Area coverage reward: weighted by probability density, based on the probability P of the target existing at the current grid point i, j in the entire search area i,j Dynamically adjust rewards, and gain higher rewards for covering high-probability areas; add a time decay factor: gradually increase the reward value for areas that have not been searched for a long time to avoid ignoring low-probability areas: R coverage =λ2·∑ (i,j)∈Visited (P i,j (tea -η·Δt ) Among them, P i,j (t) represents the probability of the target existing at the current grid point i, j in the entire search area at time step t, λ2 is the coverage bonus coefficient, η is the time decay rate, and Δt is the number of time steps that the grid has not been searched; Collaboration penalty: If multiple drones enter the same sub-area at the same time step, a penalty mechanism is triggered; the penalty value is proportional to the number of overlaps, strengthening the collaborative awareness of decentralized search: Where λ3 is the penalty coefficient, OverlapCount(k) is the number of overlapping drones in the kth sub-region; The weighted sum of the three rewards balances individual exploration and team collaboration to obtain the final reward function: R total =R discover +R coverage +R penalty 。 8. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 6, characterized in that: For step S25, the specific design is as follows: Centralized experience pool: interaction data of all drones i ,a i ,r i ,s' i Stored in the shared experience pool D; during training, batch data is randomly sampled to break data correlation, and priority sampling is used to improve the utilization of high TD error samples and accelerate convergence: D={(s i ,a i ,r i ,s′ i )∣i=1,2,…,N} Among them, s i is the sub-region state input, s' i is the next state, containing the updated sub-region probability distribution matrix Global state perception: During the training phase, the Q network of drone i can access the historical actions of other drones {a j } j≠i , dynamically weighting the impact of different drones’ actions through the attention mechanism: where {a j } Obtained through shared experience pool D, s i represents the state vector of UAV i, Σ j≠i α j a j represents the weighted sum of other UAV actions, reflecting their collaborative impact on the current UAV, α j is the attention weight, reflecting the importance of UAV j to the collaboration of i, a i represents the historical actions of drone i; The joint loss function is that the training goal of each drone is to minimize the temporal difference error containing collaborative information, and the loss is minimized by gradient descent to update the current network parameters θ i , improve the accuracy of action decision: Among them, θ i Represents the parameters of the current network, updated by gradient descent, r i Indicates that drone i is in state s i Execute action a i After the immediate reward, s' i Indicates that drone i is in the current state s i Next, perform action a i The next state that the environment transfers to is a' i Indicates that drone i is in the next state s' i The optimal action selected under i is the target network parameter, φ is the discount factor; using the soft update strategy, the target network parameter θ' i Synchronize from the current network every C steps: θ′ i ←tth i +(1-τ)θ i Where τ represents the soft update coefficient 0<τ<<1, which controls the update rate of the target network parameters.

9. The cross-domain multi-agent collaborative search method for a dynamic ocean environment according to claim 1, characterized in that: The specific process of step S3 includes the following steps: S31. The surface unmanned boat obtains all target location information searched by the UAV; S32, performing Kalman filtering on the target location information searched by the UAV to predict the next location information of the target; S33. Sort the traversal order of the multiple targets, calculate the Manhattan distance, and prioritize searching for targets moving toward the surface unmanned boat based on prioritizing traversal of targets with a short Manhattan distance; S34, perform path planning, select the A* algorithm, generate the shortest path, and avoid known obstacles; S35, determine whether the prediction error is greater than the set threshold, if less, continue to traverse the target, if greater, re-plan the traversal path, and repeat S34-S35; S36. Continue traversing the target and make real-time path corrections. If the target suddenly accelerates or turns, use the Manhattan distance gradient descent method to adjust the direction of the surface unmanned boat. S37, judging whether all targets have been traversed. If the surface unmanned boat has traversed all targets and rescued all targets, the mission is completed. Otherwise, repeat S34-S37.

Images