A cross-domain multi-agent cooperative search method for dynamic marine environment

Abstract

The application relates to a cross-domain multi-agent cooperative search method for a dynamic marine environment, which comprises the following steps: constructing an initial probability distribution graph with a target position as the center, combining real-time ocean current data with a diffusion probability model to generate a dynamic probability distribution graph evolving over time; adopting an improved weighted clustering algorithm for adaptive regional allocation, and distributing each sub-region to a UAV for search task execution; each UAV searches and locates the target in the allocated region, and transmits the identified target position information to a surface unmanned boat; after receiving the target position, the surface unmanned boat adopts a path planning algorithm based on a prediction-correction-tracking strategy, dynamically traverses all target positions and implements precise rescue. The application cooperates in the whole process from target search and positioning to rescue, and the task ends when all targets are successfully traversed and rescue is completed. Through the autonomous controllable intelligent algorithm and the heterogeneous robot cooperative mechanism, the search efficiency and dynamic adaptability are significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of autonomous unmanned system cooperative control, and particularly relates to a cross-domain multi-agent cooperative search method for dynamic marine environment. BACKGROUND

[0002] In the marine search and rescue scene, an intelligent rescue system with fast response and efficient cooperation is the key requirement to protect people's life and property safety and improve the modernization level of emergency management. However, the current technology still faces major challenges in multi-agent cooperative control and dynamic environment adaptability. First, the traditional search and rescue method relies on the assumption of static environment and is difficult to cope with the influence of dynamic factors such as ocean currents and waves on the target position. However, the existing regional division strategy is still based on the initial target distribution and lacks real-time response capability to target drift and sudden position changes, resulting in unbalanced task sub-region allocation. In addition, the existing path planning algorithm lacks the ability to integrate dynamic probability maps and is difficult to adapt to the trend of non-uniform diffusion of targets along the ocean current in the marine distress scene, which restricts the search and rescue efficiency. Second, the existing method has obvious deficiencies in multi-agent cooperative mechanism. Due to the lack of global information sharing and dynamic coordination strategy, heterogeneous agents (such as unmanned aerial vehicles and unmanned surface vehicles) often have problems of overlapping task areas or incomplete coverage. This inefficient cooperation mode cannot meet the requirements of an intelligent, networked and collaborative emergency rescue system. In view of the above challenges, the present application combines deep reinforcement learning, dynamic probability diffusion model and cross-domain cooperative control technology to propose a multi-agent cooperative search method for dynamic marine environment. Through autonomous controllable intelligent algorithms and heterogeneous robot cooperative mechanisms, the search efficiency and dynamic adaptability are significantly improved. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a cross-domain multi-agent cooperative search method for dynamic marine environment. In view of the problems of region overlap, dynamic target response lag and low cooperative efficiency in multi-agent cooperative search tasks, the method integrates an improved weighted K-means algorithm, a deep reinforcement learning path planning strategy and a target traversal algorithm based on prediction-correction-tracking mechanism to build an efficient cooperative search and rescue system. First, the search area is modeled as a two-dimensional space, and an initial Gaussian distribution probability map is constructed with the target position as the center. Real-time ocean current data and diffusion model are combined to dynamically update the target motion, generating a probability distribution map that evolves over time. Based on the probability map, task sub-regions are divided, and the improved weighted K-means algorithm is used for adaptive region allocation, and each sub-region is assigned to an unmanned aerial vehicle for search task. Each unmanned aerial vehicle searches and locates the target based on the deep reinforcement learning strategy in the assigned region, and transmits the identified target position information to the unmanned surface vehicle. After receiving the target information, the unmanned surface vehicle uses a path planning algorithm based on the prediction-correction-tracking strategy to dynamically traverse all target positions and implement precise rescue.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a cross-domain multi-agent cooperative search method for dynamic marine environments, comprising the following steps:

[0005] S1. Model the search area as a two-dimensional space, and construct an initial probability distribution map with the target location as the center. Combine real-time ocean current data and diffusion probability model to dynamically update the target motion and generate a dynamic probability distribution map that evolves over time.

[0006] S2. Based on the dynamic probability distribution map, the task sub-regions are divided, and an improved weighted K-means clustering algorithm is used for adaptive region allocation. Each sub-region is assigned to the UAV to perform the search task. Each UAV performs target search and localization in the assigned region based on a deep reinforcement learning strategy and transmits the identified target location information to the surface unmanned vessel.

[0007] S3. After receiving the target location, the unmanned surface vessel (USV) uses a path planning algorithm based on a prediction-correction-tracking strategy to dynamically traverse all target locations and implement precise rescue.

[0008] Furthermore, step S1 specifically includes the following steps:

[0009] S11. Receive the accident coordinates from GPS positioning to obtain the accident center coordinates x0, y0; calculate the probability values ​​of each grid according to the Gaussian distribution formula, set the attenuation coefficient σ, and generate the initial target existence probability P. i,j (0);

[0010] S12. Satellites or sensors acquire ocean current vector fields in real time, load real-time ocean current data, and obtain the ocean current velocity vector V. c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c [(i,j)]; where v c (i,j) represents the ocean current velocity at grid (i,j), θ c (i,j) represents the direction of the ocean current;

[0011] S13. Calculate the transfer weights of adjacent grids based on the ocean current direction to obtain the diffusion weight w from grids m,n to i,j. m,n→i,j ;

[0012] S14. Perform diffusion probability calculation, set the time step, and update the diffusion probability distribution;

[0013] S15, a period of updating is performed, if no updating is needed, a dynamic probability distribution map of the high-probability zone along the ocean current direction is output, if updating is needed, S12-S15 are repeated.

[0014] Further, the dynamic probability distribution map of the high-probability zone along the ocean current direction is output in step S15, and the specific process includes the following steps: every time step Δt, the system adjusts the probability value of each grid according to the ocean current direction, the driving effect of the ocean current on the target position is considered, the target drifts along the ocean current, which causes the dynamic change of the probability distribution, the probability diffusion model needs to reflect the dominant effect of the ocean current direction, the probability diffusion direction and rate are dominated by the ocean current vector field, and then the probability diffusion model is represented as:

[0015]

[0016] γ=k·||V c (i,j)|

[0017] wherein, represents the probability inflow, the probability P m,n (t) of the adjacent grid m,n flows into the current grid i,j according to the diffusion weight w m,n→i,j ; γ·P i,j (t) represents that the probability of the current grid i,j flows out to other grids according to the rate γ, P i,j (t) represents the target existence probability of the current grid point i,j in the entire search area at the time step t, represents the adjacent grid set of the current grid i,j, w m,n→i,j represents the diffusion weight from the grid m,n to i,j, only diffusion along the ocean current direction and the lateral diffusion is allowed, and the diffusion weight in the reverse direction of the ocean current is 0, ∑ (m,n) w m,n→i,j represents the sum of the diffusion weights of all adjacent grids to the current grid, the probability contribution of the adjacent grid is normalized, the probability value is prevented from exceeding the reasonable range due to the weight difference, V c (m,n) represents the ocean current velocity vector of the adjacent grid m,n, V c (i,j) represents the ocean current velocity vector of the current grid i,j, k represents a proportional coefficient, which adjusts the contribution weight of the ocean current velocity to the diffusion rate, γ is the diffusion rate, which is proportional to the ocean current velocity at the current grid, and the stronger the ocean current, the more significant the probability diffusion.

[0018] Further, in step S2, the specific process includes the following steps:

[0019] S21, the unmanned aerial vehicle receives the dynamic probability distribution map of the high-probability zone, which is regionally divided into a sub-region probability distribution matrix P k (t) by the improved weighted K-means algorithm and the initial dynamic probability map.

[0020] S22, rasterize the probability matrix, the sub-region probability distribution matrix P k (t) is converted into a two-dimensional tensor, and the normalized UAV relative coordinates and time steps are spliced to obtain a state feature vector;

[0021] S23, taking the state feature vector as the input of the neural network, outputting the Q value of 8 moving directions through the fully connected layer to determine the moving direction of the UAV;

[0022] S24, setting the initial exploration rate ∈ and the decay rate λ to select the action by the strategy ε-greedy; and judging whether the action is safe, if the action is not safe, selecting a suboptimal action, if the action is safe, executing the action and updating the state;

[0023] S25, the interaction data of all UAVs is stored in the shared experience pool D, and the experience pool is updated by using priority sampling;

[0024] S26, judging whether the UAV has searched the target, if the target is searched, the position information of the target is transmitted to the surface unmanned boat, if not, repeating steps S21-S26.

[0025] Further, step S21 specifically comprises: designing a clustering objective function through a weighted K-means clustering algorithm:

[0026]

[0027] wherein, is the target existence probability of the grid point i,j in the normalized entire search area, C k is the kth cluster, μ k is the cluster center; α, β, δ represent weight coefficients, respectively representing the geographical compactness of the region, the probability distribution balance and the current drift degree; DriftCost(C k ) represents the region that is punished for being far away from the cluster center and having a large current drift speed, preferentially allocating the downstream high-probability region to the UAV to avoid being carried away from the task area by the current, and finally updating the sub-region probability distribution matrix P k (t).

[0028] Further, the specific process of step S23 comprises the following steps:

[0029] Each UAV independently maintains a DQN network, and the input state feature vector of the network is composed of the normalized sub-region probability distribution matrix P k (t), the UAV state and the time step, and the network structure is divided into two parts of feature extraction and decision output: the feature extraction layer adopts a convolutional neural network CNN to process the normalized sub-region probability distribution matrix CNN contains two layers of convolution and one layer of max-pooling for extracting spatial features in the probability map; the decision output layer concatenates the feature vector output by CNN with the normalized relative coordinates x rel ,y rel and time step t of the UAV, and inputs the fully connected layer FC to generate the Q value, and finally outputs the moving actions of 8 directions, i.e., the Q values corresponding to the up, down, left, right and four diagonal directions, which can be expressed as follows:

[0030]

[0031] wherein, is the state input of the sub-region, a i represents the historical action of the UAV i, is the probability distribution matrix assigned to the i-th sub-region to which the k-th UAV belongs, and the sub-region probability distribution matrix is normalized as to ensure the numerical range in [0, 1], the relative coordinates x rel ,y rel of the UAV in the sub-region are normalized to [0, 1], and the current time step t is normalized as Time step t is used for probability diffusion model updating.

[0032] Further, for step S24, an adaptive ε-greedy strategy is used to balance exploration and utilization; in the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas; as the number of training steps increases, the exploration rate ε is adjusted exponentially:

[0033] ε(t) = ε min + (ε max - ε min )·e -λt

[0034] wherein λ controls the decay rate, and finally stabilizes at ε min ; action correction is added, if the UAV detects an obstacle, the action direction leading to collision is shielded, and a safe action with suboptimal Q value is selected again; in the multi-agent cooperative search task, the reward function needs to consider both individual efficiency and team cooperation; the cooperation reward function includes: search reward, region coverage reward and cooperation penalty, and the specific design is as follows:

[0035] Search reward: when the UAV first scans the target, a fixed positive reward is given to encourage exploration behavior; if the target has been marked by other UAVs, only zero reward is given to avoid waste of resources:

[0036]

[0037] wherein λ1 is the reward coefficient, r baseBasic rewards;

[0038] Area coverage reward: Weighted by probability density, based on the probability P of the target existing at the current grid point i,j in the entire search area. i,j Rewards are dynamically adjusted, with higher rewards for covering high-probability areas; a time decay factor is added: reward values ​​for areas that have not been searched for a long time are gradually increased to avoid ignoring low-probability areas.

[0039] R coverage =λ2·∑ (i,j)∈Visited (P i,j (t)·e -η·Δt )

[0040] Among them, P i,j (t) represents the probability that the target exists at the current grid point i,j in the entire search area at time step t, λ2 is the coverage reward coefficient, η is the time decay rate, and Δt is the number of time steps in which the grid has not been searched.

[0041] Collaboration Penalty: If multiple drones enter the same sub-region at the same time step, a penalty mechanism is triggered; the penalty value is proportional to the number of overlaps, reinforcing the collaborative awareness of decentralized search.

[0042]

[0043] Where λ3 is the penalty coefficient, and OverlapCount(k) is the number of overlapping drones in the k-th sub-region;

[0044] By weighted summing of the three reward components to balance individual exploration and team collaboration, the final reward function is obtained:

[0045] R total =R discover +R coverage +R penalty .

[0046] Furthermore, for step S25, the specific design is as follows:

[0047] Centralized experience pool: Interaction data of all drones i ,a i ,r i ,s' i Data is stored in a shared experience pool D. During training, batches of data are randomly sampled to break data correlations, and priority sampling is used to improve the utilization of high TD error samples, thereby accelerating convergence.

[0048] D={(s i ,a i ,r i ,s′ i )|i=1,2,…,N}

[0049] where s i is the sub-region state input, s' i is the next state, containing the updated sub-region probability distribution matrix

[0050] Global state awareness: in the training phase, the Q network of drone i has access to the historical actions of other drones {a j} j≠i , dynamically weighting the influence of different drone actions through an attention mechanism:

[0051]

[0052] where {a j} are obtained through a shared experience pool D, s i represents the state vector of drone i, ∑ j≠i α j a j represents the weighted sum of other drone actions, reflecting their collaborative influence on the current drone, α j is the attention weight, reflecting the collaborative importance of drone j to i, a i represents the historical actions of drone i;

[0053] The joint loss function is the training target for each drone to minimize the temporal difference error containing collaboration information. By gradient descent, the loss is minimized to update the current network parameters θ i , improving the accuracy of action decision:

[0054]

[0055] where θ i represents the parameters of the current network, updated by gradient descent, r i represents the immediate reward obtained by drone i after performing action a i in state s i , s' i represents the next state to which the environment transitions after drone i performs action a i in the current state s i , a' i represents the optimal action chosen by drone i in the next state s' i , θ' i is the target network parameter, and φ is the discount factor; a soft update strategy is adopted, and the target network parameter θ' i is synchronized from the current network every C steps:

[0056] θ′ i ←τθ i +(1-τ)θ′i .

[0057] Wherein, τ represents a soft update coefficient 0 < τ < < 1, control the update rate of target network parameters.

[0058] Further, the specific process of step S3 includes the following steps:

[0059] S31, the water surface unmanned ship gets all the target position information searched by the unmanned aerial vehicle;

[0060] S32, on the basis of the target position information searched by the unmanned aerial vehicle, Kalman filter is used to predict the next step position information of the target;

[0061] S33, the traversal order of multiple targets is sorted, and the Manhattan distance is calculated, on the basis of preferentially traversing the target with short Manhattan distance, the target moving towards the water surface unmanned ship is preferentially searched;

[0062] S34, path planning is carried out, A* algorithm is selected, and the shortest path is generated while avoiding known obstacles;

[0063] S35, it is judged whether the prediction error is greater than the set threshold value, if less than, continue to traverse the target, if greater than, re-plan the traversal path, and repeat S34-S35;

[0064] S36, continue to traverse the target, and real-time path correction is carried out, if the target suddenly accelerates or turns, Manhattan distance gradient descent method is used to adjust the direction of the water surface unmanned ship;

[0065] S37, it is judged whether all targets are traversed, if the water surface unmanned ship traverses all targets, realizes the rescue of all targets, the task is completed, otherwise, repeat S34-S37.

[0066] By the above technical scheme, the application provides a cross-domain multi-agent cooperative search method for dynamic marine environment, which has at least the following beneficial effects:

[0067] Compared with the prior art, the application has the following advantages: the improved weighted K-means algorithm combined with the probability diffusion mechanism is used to realize adaptive division and load balancing of the task area; deep reinforcement learning and probability graph are fused, and a cooperative reward function is designed to balance individual exploration and team cooperation; the water surface unmanned ship adopts a prediction-correction-tracking strategy, combines Kalman filter and A* algorithm, realizes efficient traversal and accurate rescue of multiple dynamic targets, and significantly improves the search and rescue efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0069] Figure 1 A schematic block diagram of a multi-agent cooperative search system provided by the present application is shown in the figure;

[0070] Figure 2 A schematic flow diagram of a dynamic probability map generation process provided by the present application is shown in the figure;

[0071] Figure 3 A schematic flow diagram of a target search process of a UAV provided by the present application is shown in the figure;

[0072] Figure 4 A schematic flow diagram of a target traversal process of an unmanned ship provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0073] In order to make the above objectives, features and advantages of the present application more apparent, further detailed description will be given below with reference to the accompanying drawings and specific embodiments. The realization process of how to apply technical means to solve technical problems and achieve technical effects of the present application can be fully understood and implemented.

[0074] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of a computer usable storage medium (including but not limited to a disk memory, a CD-ROM, an optical memory, etc.) containing computer usable program codes.

[0075] Please refer to Figures 1-4 , which shows a specific embodiment of the present embodiment, the present embodiment cooperates from target search, positioning to rescue the whole process, and the task ends after all targets are successfully traversed and rescue is completed. Through the self-controllable intelligent algorithm and the heterogeneous robot cooperative mechanism, the search efficiency and dynamic adaptability are significantly improved.

[0076] Please refer to Figure 1 , the present embodiment proposes a cross-domain multi-agent cooperative search method for dynamic marine environment, which includes the following steps:

[0077] S1, model the search area as a two-dimensional space, and construct an initial probability distribution map with the target position as the center, combine real-time ocean current data with a diffusion probability model to dynamically update the target motion, and generate a dynamic probability distribution map evolving over time;

[0078] As a preferred embodiment of step S1, such as Figure 2 As shown, step S1 specifically includes the following steps:

[0079] S11. First, discretize the sea surface search area into an M×N grid region, obtaining the grid coordinate set X={(x1,y1),(x2,y2),…,(x i ,y j )}, where (x i ,y j )∈R 2 This represents a two-dimensional grid coordinate system. After an accident occurs, the GPS coordinates of the accident center (x0, y0) are received. The probability values ​​for each grid cell are calculated using the Gaussian distribution formula, and an attenuation coefficient σ is set to generate the initial target existence probability P. i,j (0) marks high-priority areas where targets may exist, serving as an important basis for subsequent search path planning. Assuming the initial target distribution is concentrated near the accident center, a Gaussian distribution model is used to model the probability of initial target existence, with the probability decreasing exponentially with increasing distance from the center:

[0080]

[0081] Among them, P i,j P represents the probability of the target existing in grid (i,j). max This represents the maximum probability value, i.e., the highest probability of the target's presence in the accident's central area. σ controls the probability decay range; a larger value indicates a more dispersed target distribution. The further away from the accident center the grid, the lower the probability of the target's presence, forming a probability distribution with the accident center as a "hotspot."

[0082] S12. The ocean current vector field is the core input of the probability diffusion mechanism. Satellites or sensors acquire the ocean current vector field in real time to ensure that the probability map reflects the target's drift trend. To quantify the impact of ocean currents on the target's position, real-time ocean current data is loaded, and the ocean current vector field is modeled to construct a gridded ocean current vector field, yielding the ocean current velocity vector: V c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c [(i,j)]; where v c (i,j) represents the ocean current velocity at grid (i,j), θ c (i,j) represents the direction of the ocean current; the ocean current velocity is decomposed into components in the x and y directions to calculate the probability diffusion direction;

[0083] S13. Calculate the transfer weights of adjacent grids based on the ocean current direction to obtain the diffusion weight w from grids m,n to i,j. m,n→i,j;

[0084] S14, performing diffusion probability calculation and setting time step, updating diffusion probability distribution;

[0085] S15, performing update cycle judgment, if no update is needed, outputting the final dynamic probability distribution map of high probability band along the ocean current direction, if update is needed, repeating S12-S15.

[0086] More specifically, in step S15, the final dynamic probability distribution map of high probability band along the ocean current direction is output, and the specific process includes: every time step Δt, the system adjusts the probability value of each grid according to the ocean current direction, considers the driving effect of the ocean current on the target position, and the target drifts along the ocean current, resulting in dynamic changes of the probability distribution, the probability diffusion model needs to reflect the dominant role of the ocean current direction, the probability diffusion direction and rate are dominated by the ocean current vector field, and then the probability diffusion model is represented as:

[0087]

[0088] γ=k·||V c (i,j)|

[0089] wherein, P m,n (t) represents the probability of the adjacent grid m,n flowing into the current grid i,j, P m,n→i,j (t) represents the probability of the current grid i,j flowing out to other grids at a rate of γ, P i,j (t) represents the probability of the current grid i,j flowing out to other grids at a rate of γ, P i,j P V m,n→i,j V (m,n) w m,n→i,j V c V c V

[0090] S2, in order to further improve the search efficiency, based on the dynamic probability distribution map, the improved weighted K-means clustering algorithm is used for adaptive region allocation, and each sub-region is allocated to the unmanned aerial vehicle to perform the search task; each unmanned aerial vehicle searches and locates the target in the allocated region based on the deep reinforcement learning strategy, and transmits the identified target position information to the surface unmanned ship;

[0091] As a preferred embodiment of step S2, in step S2, as shown in Figure 3 The specific process includes the following steps:

[0092] S21, the unmanned aerial vehicle receives the dynamic probability distribution map of the high probability band by the improved weighted K-means algorithm to divide the region into a sub-region probability distribution matrix P k (t) and the initial dynamic probability graph;

[0093] More specifically, step S21 specifically includes: introducing target probability distribution, ocean current influence and load balancing weight into the clustering objective function by weighted K-means clustering algorithm:

[0094]

[0095] Wherein, is the target existence probability of the grid point i,j in the normalized entire search region, C k is the kth cluster, μ k is the cluster center; α, β, δ represent the weight coefficients, which represent the region geographical compactness, probability distribution balance and ocean current influence degree respectively; DriftCost(C k ) represents the region that is punished away from the cluster center and has large ocean current speed, which preferentially allocates the downstream high probability region to the unmanned aerial vehicle to avoid being taken away from the task area by the ocean current, and finally updates the sub-region probability distribution matrix P k (t).

[0096] In this embodiment, in order to find the target faster, the distance of each point is weighted by its probability, so that the high probability point has greater influence on the cluster center, ensuring that the high probability region is more compact; in order to promote the total probability balance of each cluster, that is, to ensure the load balance of each unmanned aerial vehicle and avoid excessive task in some regions, that is, the total probability of each region is close, the probability distribution balance is considered, and the square of the difference between the total probability of each cluster and the ideal average value is calculated.

[0097] S22, after completing the region division and allocating the sub-regions to each unmanned aerial vehicle, each unmanned aerial vehicle needs to preferentially cover and search the high probability target region in its responsible region. The present application takes DQN algorithm as the core to guide the unmanned aerial vehicle to optimize the search strategy according to the current state. The unmanned aerial vehicle constructs a high-dimensional input state vector by real-time sensing of the local probability map and its own state (position, time) As the core input of the DQN algorithm, the probability map needs to be structured into a form that can be processed by the neural network. Therefore, the grid-based probability matrix is used to represent the probability distribution matrix P of the sub-area of the current partition (M x M grid) responsible for by the UAV k (t) is converted into a two-dimensional tensor:

[0098]

[0099] Secondly, the target existence probability P i,j (t) of the current grid point i, j in the entire search area at the t time step needs to be normalized to ensure that the numerical range is within [0, 1], so the calculation formula of the probability of each grid point in the matrix is as follows:

[0100]

[0101] Finally, the input tensor is constructed by collecting all the normalized probability points in the grid to obtain the normalized probability distribution matrix which is flattened into a vector and concatenated with the UAV state (position, time) to form a complete input state:

[0102]

[0103] where the relative coordinates (x rel ,y rel ) of the UAV in the partition are normalized to [0, 1], and the current time step t is normalized to The M x M probability matrix of the current partition of the UAV is normalized and flattened into an M 2 dimensional vector, which is concatenated with the relative coordinates (x rel ,y rel ) of the UAV and the time step t to form an M 2 +3 dimensional input state. At this time, this complete state feature vector is taken as input and transmitted to the neural network of the DQN algorithm.

[0104] In this embodiment, this design enables the neural network to capture both the environmental probability distribution and the spatiotemporal dynamic characteristics.

[0105] S23, taking the state feature vector as the input of the neural network, outputting 8 Q values of moving directions through a fully connected layer to determine the moving direction of the UAV;

[0106] To solve the efficient cooperative search problem of multiple unmanned aerial vehicles (UAVs) for dynamic targets in unknown environment, the application proposes a multi-agent cooperative search method combining deep reinforcement learning (DQN) and dynamic probability graph. The method comprehensively utilizes the dynamic probability mapping mechanism to guide the search area division, and combines the information sharing mechanism and strategy coupling mechanism among multiple UAVs to realize the task cooperation and path optimization among UAVs. More specifically, the specific process of step S23 includes the following steps:

[0107] Each UAV independently maintains a DQN network, and the input state feature vector of the network is composed of the normalized sub-region probability distribution matrix P k (t), the UAV state and the time step, and the network structure is divided into two parts of feature extraction and decision output: the feature extraction layer adopts a convolutional neural network (CNN) to process the normalized sub-region probability distribution matrix CNN includes two convolutional layers and one max-pooling layer for extracting spatial features in the probability graph; the decision output layer concatenates the feature vector output by CNN with the normalized relative coordinates x rel ,y rel of the UAV and the time step to input a fully connected layer (FC) to generate a Q value, and finally outputs eight moving actions corresponding to the Q values of the up, down, left, right and four diagonal directions, which can be expressed as follows:

[0108]

[0109] Among them, is the sub-region state input, a i represents the historical action of the UAV i, is the probability distribution matrix allocated to the i-th sub-region of the k-th UAV, and the sub-region probability distribution matrix is normalized to to ensure that the numerical range is within [0, 1], the relative coordinates x rel ,y rel of the UAV in the sub-region are normalized to [0, 1], and the current time step t is normalized to Time step t is used for probability diffusion model updating.

[0110] S24, set the initial exploration rate ∈ and the decay rate λ, select the action by the strategy ε-greedy; and determine whether the action is safe, if the action is not safe, select a suboptimal action, if the action is safe, execute the action and update the state;

[0111] More specifically, for step S24, in order to better select the action of the UAV, an adaptive ε-greedy strategy is adopted to balance exploration and utilization; in the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas; with the increase of the training step number, the exploration rate ∈ is adjusted by exponential decay:

[0112] ∈(t)=∈ min +(∈ max -∈ min )·e -λt

[0113] where λ controls the decay rate, and finally stabilizes at ∈ min ; Add action correction, if the UAV detects obstacles, shield the action direction that leads to collision, and reselect the Q value suboptimal safe action; In the multi-agent cooperative search task, the reward function needs to consider individual efficiency and team cooperation; The cooperative reward function includes: search reward, area coverage reward and cooperation penalty, the specific design is as follows:

[0114] Search reward: when the UAV scans the target for the first time, give a fixed positive reward to encourage exploration behavior; If the target has been marked by other UAVs, only give zero reward to avoid waste of resources:

[0115]

[0116] where λ1 is the reward coefficient, r base is the basic reward;

[0117] Area coverage reward: through probability density weighting, according to the target existence probability P i,j of the current grid point i,j in the whole search area, adjust the reward dynamically, and get higher reward in high probability area coverage; Add time decay factor: gradually increase the reward value of the area that has not been searched for a long time, avoid ignoring low probability area:

[0118] R coverage =λ2·∑ (i,j)∈Visited (P i,j (t)·e -η·Δt )

[0119] where P i,j (t) represents the target existence probability of the current grid point i,j in the whole search area at t time step, λ2 is the coverage reward coefficient, η is the time decay rate, and Δt is the time step number of the grid that has not been searched;

[0120] Cooperation penalty: if multiple UAVs enter the same sub-area at the same time step, trigger the penalty mechanism; The penalty value is proportional to the overlap number, and the cooperation awareness of dispersed search is strengthened:

[0121]

[0122] where λ3 is the penalty coefficient, and OverlapCount(k) is the number of overlapping UAVs in the kth sub-area;

[0123] By weighted summing of the three reward components to balance individual exploration and team collaboration, the final reward function is obtained:

[0124] R total =R discover +R coverage +R penalty .

[0125] S25. All interactive data of UAVs is stored in the shared experience pool D, and the experience pool is updated by priority sampling.

[0126] To improve the efficiency of multi-drone collaboration, a centralized experience pool and a global state awareness mechanism are adopted. More specifically, for step S25, the design is as follows:

[0127] Centralized experience pool: Interaction data of all drones i ,a i ,r i ,s' i Data is stored in a shared experience pool D. During training, batches of data are randomly sampled to break data correlations, and priority sampling is used to improve the utilization of high TD error samples, thereby accelerating convergence.

[0128] D={(s i ,a i ,r i ,s′ i )|i=1,2,…,N}

[0129] Among them, s i For sub-region state input, s' i The next state contains the updated sub-region probability distribution matrix.

[0130] Global State Awareness: During the training phase, the Q-network of drone i can access the historical actions of other drones {a} j} j≠i The impact of different drone actions is dynamically weighted through an attention mechanism:

[0131]

[0132] Where {a j} Acquired through shared experience pool D, s i Let ∑ represent the state vector of drone i. j≠i α j a j α represents the weighted sum of the actions of other drones, reflecting their impact on the current drone's cooperation. j For attention weights, reflecting the collaborative importance of drone j to i, a i This indicates the historical actions of drone i;

[0133] The joint loss function is to minimize the time difference error containing cooperation information for each UAV, and the current network parameters θ are updated by minimizing the loss through gradient descent i , to improve the accuracy of action decision:

[0134]

[0135] Where θ i represents the parameters of the current network, which is updated by gradient descent, r i represents the immediate reward obtained after the UAV i performs action a i

[0136] in state s i , s' i represents the next state to which the environment is transferred after the UAV i performs action a i in the current state s i , a' i represents the optimal action selected by the UAV i in the next state s' i , θ' i is the target network parameter, and φ is the discount factor; a soft update strategy is adopted, and the target network parameter θ' i is synchronized from the current network every C steps:

[0137] θ′ i ←τθ i +(1-τ)θ′ i .

[0138] Where τ represents the soft update coefficient (0<τ<<1), which controls the update rate of the target network parameter.

[0139] S26, judge whether the UAV has searched for the target, if the target is searched, the position information of the target is transmitted to the surface unmanned boat, if not, repeat steps S21-S26.

[0140] S3, after the surface unmanned boat receives the target position, a path planning algorithm based on the prediction-correction-tracing strategy is adopted, and all target positions are dynamically traversed and precise rescue is implemented.

[0141] As a preferred embodiment of step S3, as shown in Figure 3 , the specific process of step S3 includes the following steps:

[0142] S31, the surface unmanned boat obtains all the target position information searched by the UAV;

[0143] S32, on the basis of the target position information searched by the unmanned aerial vehicle, Kalman filtering is carried out to predict the next step position information of the target; first, the dynamic target position is predicted, the target position prediction is corrected according to the ocean current, and the Kalman filtering algorithm is used to predict the target position in the future τ time steps:

[0144]

[0145] Wherein, g r (t) represents the current time target position, represents the predicted future position, v r represents the target itself drift speed, which can be ignored, that is, set to 0, V c (g r (t)) represents the ocean current velocity vector of the target current position;

[0146] S33, the traversal order of multiple targets is sorted, and the Manhattan distance is calculated, since there are multiple dynamic targets, the priority of the dynamic target needs to be allocated, and the following rules are sorted, first, the distance priority needs to be considered, that is, the target with the nearest Manhattan distance is preferentially traversed, on the basis of preferentially traversing the target with the nearest Manhattan distance, the moving trend priority also needs to be considered, and the target moving towards the water surface unmanned ship is preferentially searched;

[0147] S34, when the target prediction position is obtained, the water surface unmanned ship needs to generate a traversal path, in order to realize the shortest and safe path planning, path planning is carried out, A* algorithm is selected, the shortest path is generated, and the known obstacles are avoided; the heuristic search algorithm A* is adopted in the application, and the path cost function is constructed on the basis of obstacle avoidance:

[0148]

[0149] Wherein, is the path length, is the avoidance cost of the obstacle region in the path, and alpha is a cost weight coefficient for balancing the shortest path and safety.

[0150] S35, whether the prediction error is greater than the set threshold value is judged, if it is less than, the target is continuously traversed, if it is greater than, the traversal path is re-planned, and S34-S35 are repeated; considering that the dynamic target has uncertainty, the water surface unmanned ship needs to be continuously corrected when the path is executed. Every Δt time step, the water surface unmanned ship receives the target current actual position g r (t), and the prediction error is calculated:

[0151]

[0152] Wherein, g r (t) represents the actual position coordinates of the dynamic target at time step t, represents the predicted position coordinate of the dynamic target at time step t. If the error e(t) exceeds the threshold, the path re-planning mechanism is triggered, the target prediction position and model parameters are updated, and A* path planning is re-executed.

[0153] S36, continue to traverse the target and real-time path correction, if the target suddenly accelerates or turns, the Manhattan distance gradient descent method is used to adjust the direction of the surface unmanned ship; the actual action selection of the surface unmanned ship is dominated by the predicted path, and dynamic adjustment is made according to the target movement; the surface unmanned ship preferentially executes the next step action along the planned path, if the target suddenly deviates or accelerates, the Manhattan distance gradient descent method is used to quickly adjust the direction to shorten the distance between the current position and the target, and the specific calculation is as follows:

[0154]

[0155] wherein, represents the current position of the surface unmanned ship; represents the coordinate component of the target prediction position; sign(·) is a sign function, outputting-1, 0 or +1, to determine the moving direction.

[0156] S37, judge whether all targets are traversed, if the surface unmanned ship traverses all targets to realize the rescue of all targets, the task is completed, otherwise, S34-S37 are repeated.

[0157] In the embodiment, after a plurality of unmanned aerial vehicles complete the search of all dynamic targets and obtain the accurate positions thereof, the surface unmanned ship will take over the subsequent rapid traversal and rescue task. The present application designs an efficient traversal strategy for dynamic targets, which combines target state prediction, path planning and dynamic correction, and through the above prediction-correction-tracking strategy, the surface unmanned ship can realize rapid traversal and accurate and efficient rescue of multiple targets in a complex environment.

[0158] The present application has the following advantages compared with the prior art: the present application realizes adaptive division and load balancing of the task area by using the improved weighted K-means algorithm combined with the probability diffusion mechanism; deep reinforcement learning and probability graph are combined to design a cooperative reward function to balance individual exploration and team cooperation; the surface unmanned ship uses the prediction-correction-tracking strategy combined with Kalman filtering and A* algorithm to realize efficient traversal and accurate rescue of multiple dynamic targets, and significantly improves the search and rescue efficiency.

[0159] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0160] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices.

[0161] The above embodiments have been described in detail, and the principles and embodiments of the present application have been described herein by applying specific examples. The above examples are only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A cross-domain multi-agent cooperative search method for dynamic marine environments, characterized in that, Includes the following steps: S1. Model the search area as a two-dimensional space, and construct an initial probability distribution map with the target location as the center. Combine real-time ocean current data and diffusion probability model to dynamically update the target motion and generate a dynamic probability distribution map that evolves over time. S11. Receive the accident coordinates from GPS positioning to obtain the accident center coordinates x. o ,y o Calculate the probability value of each grid cell according to the Gaussian distribution formula, and set the attenuation coefficient σ to generate the initial target existence probability P. i,j (0); S12. Satellites or sensors acquire ocean current vector fields in real time, load real-time ocean current data, and obtain the ocean current velocity vector V. c (i,j)=v c (i,j)·[cos θ c (i,j),sin θ c (i,j)]; Among them, v c (i,j) represents the ocean current velocity at grid point i,j, θ c (i,j) represents the direction of the ocean current; S13. Calculate the transfer weights of adjacent grids based on the ocean current direction to obtain the diffusion weight w from grids m,n to i,j. m,n→i,j ; S14. Perform diffusion probability calculation, set the time step, and update the diffusion probability distribution; S15. Perform an update cycle judgment. If no update is needed, output the dynamic probability distribution map of the high probability zone along the ocean current direction. If an update is needed, repeat S12-S15. The specific process of the dynamic probability distribution map of the high-probability zone includes: after each time step Δt, the system adjusts the probability values ​​of each grid according to the ocean current direction, considering the driving effect of the ocean current on the target position. The target drifts with the ocean current, causing the probability distribution to change dynamically. The probability diffusion model needs to reflect the dominant role of the ocean current direction. The probability diffusion direction and rate are dominated by the ocean current vector field. Therefore, the probability diffusion model is expressed as: γ=k·||V c (i,j)|| in, P represents the probability of inflow into the neighborhood grids m and n. m,n (t) according to diffusion weight w m,n→i,j Flowing into the current grid i,j; γ·P i,j (t) represents the probability that the current grid i,j flows out to other grids at a rate γ, P i,j (t) represents the probability that the target exists at the current grid point i,j in the entire search area at time step t. w represents the set of neighboring grids of the current grid i,j. m,n→i,j This represents the diffusion weight from grid m,n to i,j, allowing diffusion only along the ocean current direction and its lateral direction; the diffusion weight against the ocean current direction is 0. ∑ (m,n) w m,n→i,i V represents the sum of diffusion weights from all neighboring grids to the current grid, normalizing the probability contribution of neighboring grids to avoid probability values ​​exceeding a reasonable range due to weight differences. c (m,n) represents the ocean current velocity vector of the neighborhood grid m,n, V c (i,j) represents the ocean current velocity vector at the current grid i,j, k represents the scaling factor, which adjusts the contribution weight of the ocean current velocity to the diffusion rate, and γ is the diffusion rate, which is proportional to the ocean current velocity at the current grid. The stronger the ocean current, the more significant the probability diffusion. S2. Based on the dynamic probability distribution map, the task sub-regions are divided, and an improved weighted K-means clustering algorithm is used for adaptive region allocation. Each sub-region is assigned to the UAV to perform the search task. Each UAV performs target search and localization in the assigned region based on a deep reinforcement learning strategy and transmits the identified target location information to the surface unmanned vessel. S3. After receiving the target location, the unmanned surface vessel (USV) uses a path planning algorithm based on Kalman filter to predict the target location, Manhattan distance to sort the traversal order, A* algorithm path planning, and prediction error correction tracking strategy to dynamically traverse all target locations and implement precise rescue.

2. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 1, characterized in that: In step S2, the specific process includes the following steps: S21. The UAV receiver uses an improved weighted K-means algorithm to divide the dynamic probability distribution map of the high-probability band into sub-region probability distribution matrices P. k (t) and the initial dynamic probability plot; S22, Rasterized probability matrix, which represents the probability distribution matrix P of the sub-regions of the area currently handled by the UAV. k (t) is converted into a two-dimensional tensor, and the normalized UAV relative coordinates and time steps are concatenated to obtain the state feature vector; S23. The state feature vector is used as the input of the neural network, and the Q values ​​of the eight movement directions are output through the fully connected layer to determine the movement direction of the drone. S24. Set the initial exploration rate ∈ and the decay rate λ, select an action using the policy ε-greedy, and determine whether the action is safe. If the action is unsafe, select the suboptimal action; if the action is safe, execute the action and update the state. S25. All interactive data of UAVs is stored in the shared experience pool D, and the experience pool is updated by priority sampling. S26. Determine whether the UAV has found the target. If the target is found, transmit the target's location information to the surface unmanned vessel. If not, repeat steps S21-S26.

3. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 2, characterized in that: Step S21 specifically includes: designing a clustering objective function using the weighted K-means clustering algorithm: in, Let C be the probability of the target existing at grid point i,j in the entire search region after normalization. k It is the k-th cluster, μ k It is the cluster center; α, β, δ represent weighting coefficients, which respectively represent regional geographical compactness, probability distribution equilibrium, and the degree of ocean current influence; DriftCost(C k The region P represents the area far from the cluster center and with high ocean current velocity. High-probability downstream areas are prioritized for allocation to UAVs to prevent them from being carried away from the mission area by ocean currents. Finally, the probability distribution matrix P of the sub-regions is updated and obtained. k (t).

4. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 2, characterized in that: Step S23 specifically includes the following process: Each drone independently maintains a DQN network, whose input state feature vector is composed of a normalized sub-region probability distribution matrix P. k (t) The network structure consists of the UAV state and time step, and is divided into two parts: feature extraction and decision output. The feature extraction layer uses a convolutional neural network (CNN) to process the normalized sub-region probability distribution matrix. The CNN consists of two convolutional layers and one max-pooling layer to extract spatial features from the probabilistic map; the decision output layer modifies the feature vector output by the CNN with the normalized relative coordinates x of the UAV. rel ,y rel and time step The input is concatenated, and the fully connected layer (FC) generates Q-values. The final output is the Q-values ​​for movement in eight directions: up, down, left, right, and the four diagonal directions. The formula is expressed as follows: in, For sub-region state input, a i This indicates the historical actions of drone i. This is the probability distribution matrix assigned to the i-th sub-region to which the k-th UAV belongs. The probability distribution matrix of the sub-region is then normalized as follows: Ensure the numerical range is within [0,1], and the relative coordinates x of the UAV within the sub-region. rel ,y rel Normalized to [0,1], the current time step t is normalized to The time step t is used for updating the probability diffusion model.

5. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 2, characterized in that: For step S24, an adaptive ε-greedy strategy is adopted to balance exploration and utilization; in the initial stage, a high exploration rate is set to encourage extensive exploration of unknown areas; As the number of training steps increases, the exploration rate is adjusted exponentially: ∈(t)=∈ min +(∈ max -∈ min )·e -λt Wherein, λ controls the decay rate, eventually stabilizing at ∈ min Action correction is incorporated; if the drone detects an obstacle, the direction of the action that would lead to a collision is blocked, and a safe action with a suboptimal Q-value is selected. In multi-agent cooperative search tasks, the reward function needs to balance individual efficiency and team collaboration. The cooperative reward function includes: search reward, area coverage reward, and cooperation penalty, specifically designed as follows: Search Rewards: A fixed positive reward is given when a drone first scans a target to incentivize exploration; if the target has already been marked by other drones, only zero reward is given to avoid wasting resources. Where λ1 is the reward coefficient, r base Basic rewards; Area coverage reward: Weighted by probability density, based on the probability P of the target existing at the current grid point i,j in the entire search area. i,j Dynamically adjust rewards, with higher rewards for covering high-probability areas; incorporate a time decay factor: gradually increase reward values ​​for areas that haven't been searched for a long time to avoid overlooking low-probability areas. R coverage =λ2·∑ (i,j)∈Visited (P i,j (tea -η·Δt ) Among them, P i,j (t) represents the probability that the target exists at the current grid point i,j in the entire search area at time step t, λ2 is the coverage reward coefficient, η is the time decay rate, and Δt is the number of time steps in which the grid has not been searched. Collaboration Penalty: If multiple drones enter the same sub-region at the same time step, a penalty mechanism is triggered; the penalty value is proportional to the number of overlaps, reinforcing the collaborative awareness of decentralized search. Where λ3 is the penalty coefficient, and OverlapCount(k) is the number of overlapping drones in the k-th sub-region; By weighted summing of the three reward components to balance individual exploration and team collaboration, the final reward function is obtained: R total =R discover +R coverage +R penalty 。 6. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 4, characterized in that: For step S25, the specific design is as follows: Centralized experience pool: Interaction data of all drones i ,a i ,r i ,s 'i Data is stored in a shared experience pool D. During training, batches of data are randomly sampled to break data correlations, and priority sampling is used to improve the utilization of high TD error samples, thereby accelerating convergence. D={(s i ,a i ,r i ,s' i )∣i=1,2,…,N} Among them, s i For sub-region state input, s' i The next state contains the updated sub-region probability distribution matrix. Global State Awareness: During the training phase, the Q-network of drone i can access the historical actions of other drones {a} j } j≠i The impact of different drone actions is dynamically weighted through an attention mechanism: Where {a j } Acquired through shared experience pool D, s i Let ∑ represent the state vector of drone i. j≠i α j a j α represents the weighted sum of the actions of other drones, reflecting their impact on the current drone's cooperation. j For attention weights, reflecting the collaborative importance of drone j to i, a i This indicates the historical actions of drone i; The joint loss function is used to minimize the temporal difference error containing cooperative information for each drone during training. This loss is minimized through gradient descent, and the current network parameters θ are updated accordingly. i To improve the accuracy of action decisions: Where, θ i This represents the current network parameters, updated via gradient descent, r. i This indicates that drone i is in state s. i Perform action a i The immediate reward obtained afterward, s' i This indicates that drone i is in its current state s. i Next, execute action a i The next state after the environment transitions, a' i This indicates that drone i is in the next state s' i The optimal action to be selected is θ'. i Let θ' be the target network parameter, and φ be the discount factor; a soft update strategy is adopted, and the target network parameter θ' is... i Synchronize from the current network every C steps: θ′ i ←tth i +(1-τ)θ′ i Where τ represents the soft update coefficient 0<τ<<1, which controls the update rate of the target network parameters.

7. The cross-domain multi-agent cooperative search method for dynamic marine environments according to claim 1, characterized in that: Step S3 specifically includes the following steps: S31. The unmanned surface vessel obtains all target location information searched by the drone; S32. Based on the target location information searched by the UAV, perform Kalman filtering to predict the target's next location information; S33. Sort the traversal order of multiple targets, calculate the Manhattan distance, and prioritize searching for targets moving towards the unmanned surface vessel while prioritizing the traversal of targets with the closest Manhattan distance. S34. Perform path planning, select the A* algorithm to generate the shortest path, and avoid known obstacles at the same time; S35. Determine whether the prediction error is greater than the set threshold. If it is less than the threshold, continue traversing the target. If it is greater than the threshold, replan the traversal path and repeat S34-S35. S36. Continue to traverse the target and correct the path in real time. If the target suddenly accelerates or turns, use the Manhattan distance gradient descent method to adjust the direction of the unmanned surface vessel. S37. Determine whether all targets have been traversed. If the unmanned surface vessel has traversed all targets and rescued all targets, the mission is completed. Otherwise, repeat S34-S37.

Images