Unmanned cluster multi-target search and pursuit method in high sea state environment

Abstract

This invention discloses a method for multi-target search and pursuit in an unmanned swarm under high sea state conditions. Unmanned aerial vehicles (UAVs) act as the swarm's eyes, responsible for target search; unmanned surface vessels (USVs) act as the swarm's brain, responsible for swarm control, data processing, and target allocation; and pursuit USVs perform target pursuit tasks. Through the collaboration of different UAVs, the swarm's search and pursuit tasks are completed. Furthermore, based on the swarm's collaborative task execution, this invention considers the issue of maneuver decision-making in high sea state environments. On the one hand, it enables precise target location by UAVs when traditional communication technologies are limited and positioning technologies are interfered with in high sea state environments; on the other hand, it enhances information interaction among UAVs within the swarm, maximizing overall efficiency while ensuring each UAV completes its pursuit task.

Description

Technical Field

[0001] This invention belongs to the field of unmanned swarm cooperative search and pursuit technology, specifically relating to a method for unmanned swarm multi-target search and pursuit in high sea state environments. Background Technology

[0002] In recent years, with the rapid development of unmanned equipment, unmanned systems will play an important role in future civilian and military operations. However, when facing complex environments, single unmanned platforms are finding it increasingly difficult to efficiently handle tasks. Heterogeneous unmanned system collaborative technology has become an effective means to improve the intelligence of unmanned swarms and achieve efficient task processing. Different types of intelligent agents can cooperate and divide labor according to their own characteristics, which can effectively improve task processing efficiency.

[0003] Unmanned surface vessels (USVs) have made rapid progress in near-shore search and exploration. However, in high sea states, USVs struggle to accurately acquire information about the surrounding environment and targets amidst turbulence. Unmanned aerial vehicles (UAVs), on the other hand, can leverage their aerial capabilities to ensure search capabilities in complex and changing environments. However, UAVs also suffer from limited endurance and small payload capacity. Therefore, combining these two types of unmanned equipment is a viable solution. UAVs act as the "eyes" of the swarm, handling target search, while USVs act as the "brain," managing swarm control, data processing, and target allocation. Simultaneously, USVs perform target pursuit. This combination compensates for each other's weaknesses and effectively utilizes their strengths for cross-domain collaborative search and pursuit missions. Currently, research on cross-domain collaboration of heterogeneous unmanned systems in collaborative search and pursuit technologies is limited. Most collaborative tasks target only UAV or USV swarms. Furthermore, in high sea states, traditional positioning technologies struggle to guarantee accurate target localization for swarm equipment in complex and changing sea conditions. Furthermore, most existing studies on collaborative search or pursuit of heterogeneous unmanned systems adopt centralized algorithms, that is, a central server assigns tasks to all members in the cluster. This model is not conducive to the robustness and high adaptability of unmanned clusters to the environment. Summary of the Invention

[0004] To address the technical problems mentioned in the background, this invention provides a method for multi-target search and pursuit in unmanned swarms under high sea states. Unmanned aerial vehicles (UAVs) act as the swarm's eyes, responsible for target search; unmanned surface vessels (USVs) act as the swarm's brain, responsible for swarm control, data processing, and target allocation; and pursuit USVs perform target pursuit tasks. Through the collaboration of different UAVs, the swarm's search and pursuit tasks are completed. Furthermore, based on the swarm's collaborative task execution, this invention considers the issue of maneuver decision-making in high sea states. On the one hand, when traditional communication technologies are limited and positioning technologies are interfered with in high sea states, this invention enables UAVs to accurately locate targets. On the other hand, to enhance information interaction among UAVs within the swarm and maximize overall efficiency while ensuring each UAV completes its pursuit task, a method for training unmanned swarms based on distributed multi-agent reinforcement learning is proposed.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0006] A method for multi-target search and pursuit of unmanned swarms in high sea state environments, comprising the following steps:

[0007] S1. Discretize the sea area to be searched using a grid, and model the search environment using the grid method;

[0008] S2. Treat the UAV as a particle moving on a two-dimensional plane in the air. Based on the cooperative coverage search algorithm of the environmental stimulus function, optimize the search path for each UAV. According to the UAV state information and the environmental stimulus function, optimize the next optimal waypoint of the UAV. According to the optimal waypoint, the UAV updates its motion state and moves to the corresponding position. Search the grid within its range in each time step and send the perception information to the communication UAV.

[0009] S3. After the UAV finds the target, it tracks the target and calculates the relative distance, the rate of change of relative distance and the relative speed between itself and the target at each time step. It calculates the relative positioning estimate between itself and the target and further calculates the relative positioning estimate between the UAV and the target based on the positioning estimate between the UAV and the target.

[0010] S4. The UAV records the target status information and transmits it to the communication UAV, constructs a target assignment matrix, and allocates UAV target tasks based on the current target status information and the existing pursuit UAV status information.

[0011] S5. Based on the target task allocation for the pursuit of the unmanned surface vessel (USV), establish a pursuit decision model and a decision learning model for the USV; after completion, the USV execution system will carry out the pursuit task.

[0012] Preferably, in step S1, the entire environment is considered as a planar rectangular region, and the region is divided into L... x ×L y Discrete grid (x,y) Using the x-th row and y-th column of the grid to represent the rectangle, and Δx and Δy to represent the length and width of the unit grid, the entire search environment E can be expressed by the grid set formula as follows:

[0013] E = {Grid (x,y) |m=1,2,…,L x n = 1, 2, ..., L y}

[0014] At time t, Grid (x,y) The state is represented as:

[0015] s (x,y) (t)=[μ (x,y) ,ζ (x,y) ,η (x,y) (t),c (x,y) ]

[0016] In the formula, μ (x,y) Represents Grid (x,y) The coordinates of the center point, ζ (x,y) ∈{0,1} represents Grid (x,y) Does there exist a target, where ζ (x,y) =1 indicates Grid (x,y) The memory contains the search target, ζ (x,y) =0 indicates Grid (x,y) There is no target inside, η (x,y) (t)∈{0,1,2,…,h} represents Grid (x,y) The number of times the object has been searched up to time t, c (x,y) For the search stimulus function, representing Grid (x,y) The level of attraction for drones.

[0017] Preferably, the specific steps of the cooperative coverage search algorithm based on the environmental stimulus function in step S2 for optimizing the search path for each UAV are as follows:

[0018] S21. Initialize the position and state of the UAV, where the state information of UAV i at time t is expressed by the following formula:

[0019] s i (t)=[λ i (t),o i (t)]

[0020] In the formula, λ i (t)=(x i (t),yi (t) represents the position coordinates of UAV i in environment E at time t, o i (t) represents the heading angle of UAV i at time t;

[0021] S22. Calculate the stimulus function c for each grid cell. (x,y) During the cluster search process, the stimulation function c (x,y) The following calculation method will be used for updating:

[0022]

[0023] In the formula, c (x,y) (0) is for Grid (x,y) The initial stimulus value, α∈(0,1), is the attenuation coefficient, when Grid (x,y) The more times it is searched, the lower its search stimulus value;

[0024] Drone i will select the grid cell with the largest search stimulus value among its neighboring grid cells as the next search point, as expressed in the following formula:

[0025]

[0026] Once UAV i locates the target, it records and calculates the target's state, then sends the target state to other UAVs in the same communication group. The target state formula is expressed as:

[0027] s target,j (t)=[λ j (t),v j (t),θ j,i (t)]

[0028] In the formula, λ j (t)=(x j (t),y j (t) represents the position coordinates of target j in environment E at time t, v j (t) represents the velocity of target j, θ j,i (t) represents the deflection angle of target j relative to UAV i.

[0029] Preferably, in step S3, after the UAV locates the target, the UAV uses ultra-wideband ranging and visual odometry to measure the relative distance between the UAV and the unmanned surface vessel in real time at each time step. and relative velocity Based on the measurement data, the formula for estimating the relative positioning between UAV i and target j at the t-th time step is as follows:

[0030]

[0031] In the formula, ε represents the rate of change of relative distance. t , and These are relative velocities. relative distance and the rate of change of relative distance The measurement error at the t-th time step; T is the sampling period of the ultra-wideband sensor, γ∈R + It is a tunable constant gain;

[0032] Based on the status information of the pursuing unmanned surface vessel (USV) and drone (UAV) provided within the cluster, the relative distance between the pursuing USV k and UAV i is obtained. relative speed and relative distance change rate Then, the relative distance between each pursuing unmanned surface vessel and the target was calculated. relative speed and relative distance change rate

[0033] The formula for estimating the relative position of the pursuing unmanned surface vessel k and the target j at the same time step is expressed as follows:

[0034]

[0035] Preferably, in step S4, a total of l pursuit unmanned surface vessels pursue p targets, where l ≥ p, and the target allocation matrix A = [a ij ], when a ij When a = 1, it means that target j is assigned to the pursuing unmanned surface vessel i; when a ij When = 0, it means that target j has not been assigned to the pursuing unmanned surface vessel i. In target allocation, each target should be assigned at least one pursuing unmanned surface vessel, i.e. Furthermore, all pursuing unmanned surface vessels should ultimately be tasked with a pursuit mission, namely...

[0036] The target allocation model, which takes minimizing the initial relative distance between the unmanned surface vessel and the target as the allocation objective, is expressed as follows:

[0037]

[0038]

[0039]

[0040] a ij ∈{0,1}

[0041] In the formula, This indicates the initial relative distance between the pursuit drone and the target. Each pursuit drone calculates its matching degree with the target, and the drone with the highest matching degree is selected to carry out the pursuit mission.

[0042] Preferably, in step S5, a target pursuit model for the unmanned surface vessel is established, which is represented by tuples as follows:

[0043]

[0044] In the formula, S represents the current state space being pursued, which can be shared by all devices within the cluster, and A... i Let T:S×A represent the action space for pursuing the unmanned surface vessel i. l →S represents the deterministic transfer function of the environment, R i : The reward function represents the function for pursuing the unmanned surface vessel;

[0045] The global reward value for the pursuit unmanned surface vessel (USV) formation is defined as the average of the reward values ​​of each USV, expressed by the following formula:

[0046]

[0047] In the formula, r t (s,a) represents the reward value obtained by chasing the unmanned surface vessel formation at time t in state s;

[0048] The maximization strategy formula is expressed as follows:

[0049]

[0050] In the formula, s′≡s t+1 This represents the state at time t+1;

[0051] The reward value for each pursuit drone is set, expressed by the following formula:

[0052] r i (s,a)=r cap +r help +r step

[0053] In the formula, r cap This represents the capture reward (r) obtained when the distance between the pursuit drone and the target is less than the pursuit distance, i.e., when the pursuit drone catches up with the target. help This indicates that when multiple unmanned surface vessels (USVs) are pursuing the same target, an assistance reward is given upon the target's capture. step Represented as step size reward, r step The formula is expressed as follows:

[0054] r step =ω1r1+ω2r2

[0055] ω1+ω2=1

[0056] In the formula, r1 is the pursuit distance bonus, and r2 is the collision bonus;

[0057] The pursuit distance reward r1 is expressed by the following formula:

[0058]

[0059] In the formula, k1 represents the remaining pursuit distance, and r1 is the adjustment coefficient for the reward.

[0060] The collision reward r2 formula is expressed as follows:

[0061]

[0062] In the formula, d min =mind i,j , represents the minimum travel distance between the pursuing unmanned vessels, r2∈(-1,0], k2 is the reward r2 adjustment coefficient;

[0063] Preferably, in step S5, a multi-unmanned surface vessel (USV) pursuit maneuver decision model is established, employing an Actor-Critic structure. The Actor network and Critic network of each pursuit USV are connected via a bidirectional recurrent neural network. The hidden layers of the Actor network and Critic network in a single pursuit USV decision model are used as recurrent units of the bidirectional recurrent neural network, and this model is expanded according to the number of pursuit USVs.

[0064] The objective function formula for pursuing the unmanned surface vessel is expressed as follows:

[0065]

[0066] In the formula, This indicates that action a is taken under the state transition function T. θ The obtained state distribution, For expectations;

[0067] The objective function formula for pursuing the unmanned surface vessel formation is expressed as follows:

[0068]

[0069] The gradient formula for the policy network parameter θ is expressed as follows:

[0070]

[0071] Using parameterized critical function Q ξ (s,a) is used to estimate the state-action function in the above equation. The Critic is trained using a sum-of-squares loss function, Q. ξ The gradient formula for (s,a) is expressed as follows:

[0072]

[0073] In the formula, ξ represents the Q-network parameters;

[0074] The Actor and Critic networks are optimized using stochastic gradient descent. During the interactive learning process, the network parameters are updated using data obtained through trial and error, thus completing the optimization of collaborative search and pursuit.

[0075] Preferably, the training and learning process of the multi-unmanned surface vessel cooperative target pursuit decision model includes the following steps:

[0076] S51. Initialize the online network parameters of Actor and Critic, and assign the online network parameters to the corresponding target network parameters, i.e., θ′←θ and ξ′←ξ, where θ′ and ξ′ are the target parameters of Actor and Critic, respectively. Initialize the experience replay space. Save the data obtained during the exploration;

[0077] S52. Determine the initial state of the training, and set the initial position and speed of the pursuit unmanned surface vessel formation and the target;

[0078] S53. Repeat the training for multiple sets based on the initial state, simulating the following operations:

[0079] Each pursuit drone is based on state s t And a random process ε generates an action And execute;

[0080] After all actions are performed, the state transitions to s. t+1 Calculate the reward value And will pass process variables Stored in the experience replay space During the learning process, a batch of M empirical data points is randomly selected. The target Q value for each pursuit drone is calculated using the following formula:

[0081]

[0082] The gradient estimate of the Critic is calculated using the following formula:

[0083]

[0084] Based on the obtained gradient estimates Δξ and Δθ, the online network parameters of Actor and Critic are updated. Subsequently, the target network parameters are updated, as expressed by the following formula:

[0085]

[0086] In the formula, k∈(0,1).

[0087] The beneficial effects of adopting the above technical solution are as follows:

[0088] (1) The present invention uses a gridded method to model the search environment of unmanned clusters, which facilitates the description of environmental information and reduces the amount of computation;

[0089] (2) The present invention designs a cooperative coverage search algorithm based on environmental stimulus function, which optimizes the search path of the UAV by taking into account the possible locations of the target and the current state of the UAV.

[0090] (3) The present invention adopts a persistent reward relative positioning method to measure the relative position and relative velocity between the unmanned equipment and the target in real time. It does not rely on external infrastructure, ensures accurate positioning in the denied environment, and can cope with the interference of traditional positioning systems in high sea state environments, ensuring the accurate positioning of UAVs and unmanned surface vessels to the target.

[0091] (4) This invention establishes an unmanned cluster collaborative communication model, divides the unmanned cluster into multiple communication groups, and achieves unmanned cluster collaborative search for targets on the sea surface where communication infrastructure and resources are lacking, avoiding collisions between drones or unmanned boats, and ensuring that during the target search process, the search drone cluster can quickly transmit its own status information, environmental information and target status information to the base station deployed on the unmanned boat.

[0092] (5) In the process of unmanned swarm collaborative search and pursuit, this invention ensures that while the UAVs in the swarm are conducting searches, the unmanned surface vessel target tasks are allocated, thereby achieving task coordination in target search and target pursuit, and designs a target task allocation method.

[0093] (6) This invention organizes the individual learning behavior of unmanned surface vessels into unmanned surface vessel cluster collaboration through a coordination mechanism, designs a distributed multi-agent reinforcement learning method based on device communication, ensures that each unmanned surface vessel completes the pursuit task, and maximizes the overall benefits of cluster pursuit. In complex and changeable high sea state environment, it realizes the efficiency of unmanned cluster collaborative pursuit decision-making and ensures the stability and reliability of unmanned cluster pursuit of targets. Attached Figure Description

[0094] Figure 1This invention is a system model based on unmanned swarm collaborative search and pursuit. The system includes an unmanned swarm, a communication unmanned surface vessel swarm, a pursuit unmanned surface vessel swarm, and a target to be searched.

[0095] Figure 2 This is a flowchart of the invention;

[0096] Figure 3 It is a maneuvering model for unmanned surface vessels to pursue targets based on a bidirectional recurrent neural network. Detailed Implementation

[0097] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0098] like Figure 1 This paper presents a system model based on unmanned swarm cooperative search and pursuit, including a swarm of unmanned aerial vehicles (UAVs), a swarm of communication UAVs, a swarm of pursuit UAVs, and the target to be searched. This example demonstrates a method for multi-target search and pursuit of unmanned swarms in high sea state environments; the specific process is as follows: Figure 2 As shown, the specific implementation method of the unmanned swarm multi-target search and pursuit method for high sea state environments is as follows:

[0099] 1. Model the search environment using a grid method. Treat the entire environment as a planar rectangular region, and divide the region into L... x ×L y Discrete grid. (x,y) Let the x-th row and y-th column of the rectangle be represented by a raster, and Δx and Δy represent the length and width of a unit raster, respectively. The entire search environment E can be represented by a raster set as follows:

[0100] E = {Grid (x,y) |m=1,2,…,L x n = 1, 2, ..., L y}

[0101] At time t, Grid (x,y) The state is represented as:

[0102] s (x,y) (t)=[μ (x,y) ,ζ (x,y) ,η (x,y) (t),c (x,y) ]

[0103] Where μ (x,y) Represents Grid (x,y) The coordinates of the center point, ζ (x,y) ∈{0,1} represents Grid (x,y) Does there exist a target, where ζ (x,y) =1 indicates Grid (x,y) The memory contains the search target, ζ(x,y) =0 indicates Grid (x,y) There is no target inside, η (x,y) (t)∈{0,1,2,…,h} represents Grid (x,y) The number of times the object has been searched up to time t, c (x,y) For the search stimulus function, representing Grid (x,y) The level of attraction for drones.

[0104] 2. Initialize the number of devices within the unmanned swarm, including m UAVs, n communication UAVs, l pursuit UAVs, and p targets. Within the swarm, UAVs and communication UAVs form communication groups. Each communication group consists of one communication UAV and several UAVs. Based on the different communicating parties, UAV communication within a group is divided into A2S (UAV to UAV) communication and A2A (UAV to UAV) communication. In the grouped backhaul uplink, UAVs share uplink spectrum resources and reuse resource blocks within the group.

[0105] 3. Design a cooperative coverage search algorithm based on environmental stimulus functions to optimize the search path for each UAV. The specific steps are as follows:

[0106] 1) Initialize the position and state of the UAV, where the state information of UAV i at time t can be represented as:

[0107] s i (t)=[λ i (t),o i (t)]

[0108] Where λ i (t)=(x i (t),y i (t) represents the position coordinates of UAV i in environment E at time t, o i (t) represents the heading angle of UAV i at time t.

[0109] 2) Calculate the stimulus function c for each grid cell. (x,y) During the cluster search process, the stimulation function c (x,y) The following calculation method will be used for updating:

[0110]

[0111] Where c (x,y) (0) is for Grid (x,y) The initial stimulus value, α∈(0,1), is the attenuation coefficient, when Grid (x,y) The more times it is searched, the lower its search stimulus value.

[0112] The drone i will select the grid with the highest search stimulus value among the neighboring grids as the next search point, i.e.

[0113]

[0114] Under the influence of the environmental search stimulus function, drones tend to move towards grid cells with higher stimulus values, meaning they are more likely to be selected for search if a cell has not been searched before. Drones effectively avoid areas they have already searched repeatedly. This ensures high search coverage and efficiency for the cluster.

[0115] Once UAV i locates the target, it records and calculates the target's state, then sends the target state to other UAVs in the same communication group. The target state can be represented as:

[0116] s target,j (t)=[λ j (t),ν j (t),θ j,i (t)]

[0117] Where, λ j (t)=(x j (t),y j (t) represents the position coordinates of target j in environment E at time t, ν j (t) represents the velocity of target j, θ j,i (t) represents the deflection angle of target j relative to UAV i.

[0118] 4. Once the UAV locates the target, it uses ultra-wideband ranging and visual odometry to measure the relative distance between the UAV and the unmanned surface vessel in real time at each time step. and relative velocity Based on the measurement data, the relative positioning estimate between UAV i and target j at the t-th time step is given as follows:

[0119]

[0120] in, ε represents the rate of change of relative distance. t , and These are relative velocities. relative distance Relative distance change rate The measurement error at the t-th time step; T is the sampling period of the ultra-wideband sensor, γ∈R + It is a harmonic constant gain.

[0121] Furthermore, based on the status information of the pursuing unmanned surface vessel (USV) and unmanned aerial vehicle (UAV) provided within the cluster, the relative distance between the pursuing USV k and UAV i is obtained. relative speed and relative distance change rate Then, the relative distance between each pursuing unmanned surface vessel and the target was calculated. relative speed and relative distance change rate

[0122] Furthermore, the relative positioning estimates of the pursuing unmanned surface vessel k and the target j are calculated at the same time step:

[0123]

[0124] 5. In target allocation, there are l pursuit unmanned surface vessels (USVs) pursuing p targets, where l ≥ p. Set the target allocation matrix A = [a...]. ij ], when a ij When a = 1, it means that target j is assigned to the pursuing unmanned surface vessel i; when a ij When = 0, it means that target j has not been assigned to the pursuing unmanned surface vessel i. In target allocation, each target should be assigned at least one pursuing unmanned surface vessel, i.e. Furthermore, all pursuing unmanned surface vessels should ultimately be tasked with a pursuit mission, namely...

[0125] The target allocation model is established with the goal of minimizing the initial relative distance between the unmanned surface vessel and the target as follows:

[0126]

[0127]

[0128]

[0129] a ij ∈{0,1}

[0130] in, This indicates the initial relative distance between the pursuit drone and the target. Each pursuit drone calculates its matching degree with the target, and the drone with the highest matching degree is selected to carry out the pursuit mission.

[0131] 6. This invention designs a distributed multi-agent reinforcement learning method based on device communication to realize maneuver decision-making for collaborative pursuit by multiple unmanned surface vessels. The specific details are as follows:

[0132] (1) Strategic coordination mechanism

[0133] During the target pursuit process of unmanned surface vessels (USVs), each USV makes its own maneuvering decisions based on its own situation in high sea states. The pursuit of multiple USV systems can be regarded as a competitive game between USVs and targets. An USV target pursuit model is established, which is represented by tuples:

[0134]

[0135] Where S represents the current state space being pursued, which can be shared by all devices within the cluster, and A... i Let T:S×A represent the action space for pursuing the unmanned surface vessel i. l →S represents the deterministic transfer function of the environment, R i : This represents the reward function for pursuing the unmanned surface vessel.

[0136] The global reward value for the pursuit unmanned surface vessel (USV) formation is defined as the average of the reward values ​​of each USV, which can be expressed as:

[0137]

[0138] Where, r t (s,a) represents the reward value obtained by chasing the unmanned surface vessel formation at time t under state s.

[0139] The goal of pursuing unmanned surface vessel (USV) swarms is to learn a strategy that maximizes the expected value of the discount reward, i.e. Where 0 < λ ≤ 1 is the discount factor.

[0140] In summary, the following maximization strategies can be derived:

[0141]

[0142] Where, s′≡s t+1 The state at time t is determined by the state transition function T(s,a).

[0143] To reflect the role of each individual pursuit drone in the coordinated pursuit, a reward value is assigned to each pursuit drone, which can be expressed as:

[0144] r i (s,a)=r cap +r help +r step

[0145] Where, r cap This represents the capture reward (r) obtained when the distance between the pursuit drone and the target is less than the pursuit distance, i.e., when the pursuit drone catches up with the target. help This indicates that when multiple unmanned surface vessels (USVs) are pursuing the same target, an assistance reward can be obtained after the target is captured.step Represented as step-size reward, it is composed of a weighted average of multiple sub-rewards:

[0146] r step =ω1r1+ω2r2

[0147] ω1+ω2=1

[0148] The definitions of r1 and r2 are as follows:

[0149] 1. Pursuit distance bonus r1

[0150]

[0151] At each time step, the pursuit drone will receive a negative reward, r1 and the remaining pursuit distance. They have a linear relationship, and k1 is the adjustment coefficient.

[0152] 2. Collision bonus r2

[0153]

[0154] Where d min =mind i,j , represents the minimum travel distance between the pursuing unmanned vessels, r2∈(-1,0], and k2 is the adjustment coefficient.

[0155] For m individual pursuit unmanned surface vessels, there exist m Bellman equations, i.e.

[0156]

[0157] During reinforcement learning training, the allocation of reward values ​​defines the feedback for each pursuit drone in areas such as target allocation and collision avoidance. After training, the pursuit drones can achieve decision-making coordination, enabling their behaviors to reach a tacit understanding.

[0158] (2) Decision-making learning mechanism

[0159] A multi-unmanned surface vessel (USV) pursuit maneuver decision-making model is established to ensure information exchange among USVs and achieve coordinated swarm maneuvers. This model adopts an Actor-Critic structure, connecting the Actor network and Critic network of each pursuit USV through a bidirectional recurrent neural network, specifically as follows: Figure 3 As shown, the hidden layers in the policy network (Actor) and Q network (Critic) of a single pursuit unmanned surface vessel decision model are used as recursive units of a bidirectional recursive neural network, and the network is expanded according to the number of pursuit unmanned surface vessels.

[0160] The individual objective function for pursuing unmanned surface vessels can be defined as:

[0161]

[0162] in This indicates that action a is taken under the state transition function T. θ The obtained state distribution.

[0163] The objective function for pursuing the unmanned surface vessel formation is expressed as:

[0164]

[0165] According to the gradient theorem for multi-agent deterministic policies, the gradient of the policy network parameter θ is:

[0166]

[0167] Using parameterized critical function Q ξ (s,a) is used to estimate the state-action function in the above equation. The Critic algorithm is trained using a sum-of-squares loss function. ξ The gradient of (s,a) can be expressed as:

[0168]

[0169] Where ξ represents the Q-network parameters.

[0170] The Actor and Critic networks are optimized using stochastic gradient descent. During the interactive learning process, the network parameters are updated using data obtained through trial and error, thus completing the optimization of collaborative search and pursuit.

[0171] (3) Training and learning process of multi-unmanned surface vessel cooperative target pursuit decision model

[0172] a. Initialize the online network parameters for Actor and Critic, and assign these parameters to the corresponding target network parameters, i.e., θ′←θ and ξ′, where θ′ and ξ′ are the target parameters for Actor and Critic, respectively. Initialize the experience replay space. Save the data obtained during the exploration;

[0173] b. Determine the initial state of the training, and set the initial position and speed of the pursuit unmanned surface vessel formation and the target;

[0174] c. Repeat the training for multiple episodes based on the initial state, performing the following operations in each episode of the pursuit simulation:

[0175] Each pursuit drone is based on state s t And a random process ε generates an action And execute;

[0176] After all actions are performed, the state transitions to s.t+1 Calculate the reward value And will pass process variables Stored in the experience replay space During the learning process, a batch of M empirical data points is randomly selected. To calculate the target Q value for each pursuit drone, i.e.

[0177]

[0178] Calculate the gradient estimate of Critic, i.e.

[0179]

[0180] Calculate the gradient estimate of the Actor, i.e.

[0181]

[0182] Based on the obtained gradient estimates Δξ and Δθ, the online network parameters of Actor and Critic are updated. Subsequently, the target network parameters are updated.

[0183]

[0184] Where k∈(0,1).

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0190] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for multi-target search and pursuit of unmanned swarms in high sea state environments, characterized in that, Includes the following steps: S1. Discretize the sea area to be searched using a grid, and model the search environment using the grid method; S2. Treat the UAV as a particle moving on a two-dimensional plane in the air. Based on the cooperative coverage search algorithm of the environmental stimulus function, optimize the search path for each UAV. According to the UAV state information and the environmental stimulus function, optimize the next optimal waypoint of the UAV. According to the optimal waypoint, the UAV updates its motion state and moves to the corresponding position. Search the grid within its range in each time step and send the perception information to the communication UAV. S3. After the UAV finds the target, it tracks the target and calculates the relative distance, the rate of change of relative distance and the relative speed between itself and the target at each time step. It calculates the relative positioning estimate between itself and the target and further calculates the relative positioning estimate between the UAV and the target based on the positioning estimate between the UAV and the target. S4. The UAV records the target status information and transmits it to the communication UAV, constructs a target assignment matrix, and allocates UAV target tasks based on the current target status information and the existing pursuit UAV status information. S5. Based on the target task allocation for the pursuit unmanned surface vessel (USV), establish a pursuit decision model and a decision learning model for the USV; after completion, the USV executes the pursuit task of the system. The cooperative coverage search algorithm based on environmental stimulus functions in step S2 optimizes the search path for each UAV in the following specific steps: S21. Initialize the drone's position and status, where... Time Drone The state information formula is expressed as follows: ； In the formula, express Time Drone In the environment Location coordinates, express Time Drone The heading angle; S22. Calculate the stimulus function for each grid cell. During cluster search, the stimulation function The following calculation method will be used for updating: ； In the formula, for Initial stimulus value, Let be the attenuation coefficient, when The more times it is searched, the lower its search stimulus value; drones The grid cell with the highest search stimulus value among neighboring grid cells will be selected as the next search point, as expressed in the following formula: ； When drones After the target is located, the UAV records and calculates the target state, and sends the target state to the communication UAV in the same communication group. The target state formula is expressed as: ； In the formula, express Momentary Goal In the environment Location coordinates, Indicate target speed, Indicate target Compared to drones The deflection angle.

2. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 1, characterized in that, In step S1, the entire environment is considered as a planar rectangular region, and the region is divided into... Discrete grid, The first character representing the rectangle row and number Column grid, and These represent the length and width of a unit grid, respectively, and the entire search environment. The raster set formula is expressed as follows: ； exist time, The state is represented as: ； In the formula, express The coordinates of the center point express Does a target exist, among which express There is a search target in memory. express There is no target inside. express arrive The number of times it has been searched up to this point. For the search stimulus function, represents The level of attraction for drones.

3. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 1, characterized in that, In step S3, after the UAV locates the target, it uses ultra-wideband ranging and visual odometry to measure the relative distance between the UAV and the unmanned surface vessel in real time at each time step. and relative velocity Based on the measurement data, the first... At a time step, the drone and target The formula for estimating the relative positioning between them is expressed as follows: ； In the formula, Indicates the rate of change of relative distance. , and These are relative velocities. Relative distance and the rate of change of relative distance respectively in the Measurement error per time step; It is the sampling period of the ultra-wideband sensor. It is a tunable constant gain; Based on the status information of the pursuing unmanned surface vessel and drone provided within the cluster, the pursuing unmanned surface vessel is obtained. and drones relative distance Relative velocity and relative distance change rate This allows for the calculation of the relative distance between each pursuing unmanned surface vessel and the target. Relative velocity and relative distance change rate ; Calculate the pursuit of the unmanned surface vessel at the same time step. and target The formula for estimating relative positioning is expressed as follows: 。 4. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 1, characterized in that, In step S4, there are a total of A pursuit drone Several targets were pursued, among which Set the target allocation matrix ,when When, it indicates the target. Assigned to the pursuit drone ,when When, it indicates the target. Not assigned to the pursuit drone. In target allocation, each target should be assigned at least one pursuing unmanned surface vessel, i.e. Furthermore, all pursuing unmanned surface vessels should ultimately be tasked with a pursuit mission, namely... ; The target allocation model, which takes minimizing the initial relative distance between the unmanned surface vessel and the target as the allocation objective, is expressed as follows: ； In the formula, This indicates the initial relative distance between the pursuit drone and the target. Each pursuit drone calculates its matching degree with the target, and the drone with the highest matching degree is selected to carry out the pursuit mission.

5. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 1, characterized in that, In step S5, a target pursuit model for unmanned surface vessels is established. This model is represented by tuples as follows: ； In the formula, This represents the current state space of the target being pursued, and it can be shared by all devices within the cluster. Indicates pursuit of unmanned surface vessels The space of motion The deterministic transfer function representing the environment. The reward function represents the function for pursuing the unmanned surface vessel; The global reward value for the pursuit unmanned surface vessel (USV) formation is defined as the average of the reward values ​​of each USV, expressed by the following formula: ； In the formula, Indicates the state Down, The reward points gained from constantly tracking down unmanned surface vessel formations; The maximization strategy formula is expressed as follows: ； In the formula, express The state at any given moment; The reward value for each pursuit drone is set, expressed by the following formula: ； In the formula, This indicates the capture reward received when the distance between the pursuit drone and the target is less than the pursuit distance, i.e., when the pursuit drone catches up with the target. This indicates that when multiple unmanned surface vessels (USVs) are pursuing the same target, an assistance reward will be given after the target is captured. This is represented as a step-size reward. The formula is expressed as follows: ； In the formula, As a reward for pursuit distance, For collision rewards; Pursuit Distance Bonus The formula is expressed as follows: ； In the formula, The remaining pursuit distance, As a reward Adjustment coefficient; Collision Rewards The formula is expressed as follows: ； In the formula, This indicates the minimum distance traveled between the pursuing unmanned vessels. , As a reward Adjustment coefficient.

6. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 1, characterized in that, In step S5, a multi-unmanned surface vessel (USV) pursuit maneuver decision-making model is established, employing an Actor-Critic structure. The Actor and Critic networks of each pursuit USV are connected via a bidirectional recurrent neural network. The hidden layers of the Actor and Critic networks in a single pursuit USV decision-making model are used as recursive units of the bidirectional recurrent neural network, and this model is expanded according to the number of pursuit USVs. The objective function formula for pursuing the unmanned surface vessel is expressed as follows: ； In the formula, In the state transition function Take action below The obtained state distribution, For expectations; The objective function formula for pursuing the unmanned surface vessel formation is expressed as follows: ； Policy network parameters The gradient formula is expressed as follows: ； Using parameterized critical functions To estimate the state-action function of the above equation The Critic algorithm is trained using a sum-of-squares loss function. The gradient formula is expressed as follows: In the formula, These are the Q network parameters; The Actor and Critic networks are optimized using stochastic gradient descent. During the interactive learning process, the network parameters are updated using data obtained through trial and error, thus completing the optimization of collaborative search and pursuit.

7. The method for multi-target search and pursuit of unmanned swarms in high sea state environments according to claim 6, characterized in that, The training and learning process of the multi-unmanned surface vessel cooperative target pursuit decision-making model includes the following steps: S51. Initialize the online network parameters of Actor and Critic, and assign the online network parameters to the corresponding target network parameters, i.e. and ,in and These are the target parameters for the Actor and Critic, respectively, and the initialization of the experience replay space. Save the data obtained during the exploration; S52. Determine the initial state of the training, and set the initial position and speed of the pursuit unmanned surface vessel formation and the target; S53. Repeat the training for multiple sets based on the initial state, simulating the following operations: Each pursuit drone is based on state. and stochastic processes Generate an action And execute; After all actions are performed, the state transitions to... Calculate the reward value And will pass process variables Stored in the experience replay space During the learning process, a batch of students were randomly selected. 100 empirical data points The target Q value for each pursuit drone is calculated using the following formula: ； The gradient estimate of the Critic is calculated using the following formula: ； Based on the obtained gradient estimate and Update the online network parameters of Actor and Critic, and then update the target network parameters. The formula is expressed as follows: ； In the formula, .

Images