Multi-agent unmanned aerial vehicle collaborative interception method and system based on dynamic sampling

Abstract

The invention discloses a multi-agent unmanned aerial vehicle collaborative interception method and system based on dynamic sampling, and the method comprises the following steps: solving the problems of experience quality degradation and slow strategy convergence in the training process of multiple agents through introducing an independent experience cache and a dynamic sampling network, and enabling each interception unmanned aerial vehicle to carry a six-direction camera, the method is used for obtaining a target image, three-dimensional positioning of a target unmanned aerial vehicle is achieved through HSV color space segmentation, image centroid extraction, depth estimation and other methods, meanwhile, sharing of an observation value and a target state is achieved between the unmanned aerial vehicles through intra-group communication, and then distributed decision making and task collaboration are achieved. The system covers intensive reward mechanisms of multiple dimensions, including interception success, target allocation rationality, task maintenance and the like, the stability and collaboration of the agent strategy are remarkably improved, and the method has obvious advantages in the aspects of interception success rate, allocation efficiency, response speed and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and particularly to a multi-agent unmanned aerial vehicle collaborative interception method and system based on dynamic sampling. Background Art

[0002] With the development of unmanned aerial vehicle technology, its applications in the fields of public management, logistics transportation, environmental monitoring, etc. are becoming increasingly widespread. However, the extensive use of unmanned aerial vehicles may also have an impact on the public environment and facilities, making the research on unmanned aerial vehicle interception technology crucial.

[0003] In the field of unmanned aerial vehicle interception technology, algorithms are mainly divided into rule-based algorithms and learning-based algorithms. Early rule-based unmanned aerial vehicle interception algorithms achieved certain results in specific scenarios, but had obvious limitations when facing complex dynamic environments. These algorithms were difficult to adapt to the rapid changes in the environment and task requirements, resulting in a low interception rate and poor interception performance.

[0004] To overcome the defects of rule-based algorithms, learning-based interception algorithms emerged. Among them, the multi-agent deep deterministic policy gradient (MADDPG) algorithm has received extensive attention in multi-unmanned aerial vehicle collaborative tasks. (Zhang Bo, Liu Manguo, Liu Mengyan. Multi-UAV Cooperative Attack Method Based on MADDPG [J / OL]. Journal of Projectiles, Rockets, Missiles and Guidance, 1-9 [2025-04-21]). By constructing a typical adversarial scenario, the multi-unmanned aerial vehicle cooperative attack problem is modeled as a distributed partially observable Markov decision process (Dec-POMDP), a unique reward function is designed, and the MADDPG algorithm is used to train the attack strategy.

[0005] However, the traditional MADDPG algorithm still has some deficiencies in practical applications. On the one hand, the shared experience replay buffer mechanism it adopts cannot effectively distinguish the quality of experience, and a large number of low-quality experiences participating in the replay will have a negative impact on network training, resulting in low training efficiency and difficulty in converging to the optimal solution. On the other hand, the policy network of the traditional MADDPG algorithm has limited ability to handle the complex spatio-temporal relationships between multi-agents, and it is difficult to accurately capture environmental information, affecting the accuracy of task allocation and interception decision-making. In addition, in a complex dynamic environment, unmanned aerial vehicles require more accurate environmental perception capabilities, and the existing methods are not deep enough in the integration of visual detection technology and reinforcement learning, and cannot fully utilize visual information to improve the target tracking and interception performance of unmanned aerial vehicles. In such a technical background, studying how to improve the MADDPG algorithm and enhance the collaborative interception ability of multi-unmanned aerial vehicles in complex environments has become an important topic in the current field of unmanned aerial vehicle interception technology. Summary of the Invention

[0006] The present invention proposes a multi-agent UAV collaborative interception method and system based on dynamic sampling. By establishing an independent replay buffer, adopting a multi-dimensional convolutional policy network, and integrating visual detection technology with deep reinforcement learning, the collaborative interception ability of multiple UAVs in a complex dynamic environment is improved. It is especially suitable for the collaborative interception task of multiple UAVs on a target UAV in a complex dynamic environment, and can be widely applied in fields such as public management and emergency response to achieve efficient and accurate target interception and task execution.

[0007] The present invention is realized by at least one of the following technical solutions.

[0008] A multi-agent UAV collaborative interception method based on dynamic sampling includes the following steps: (1) The interceptor UAV collects target images through an on-board camera and obtains its own environmental observation value by sensing the environment through internal communication within the interceptor UAV team. (2) The improved multi-agent deep deterministic policy gradient algorithm controls the interceptor UAVs through the target images captured by the interceptor UAVs and their own environmental observation values, enabling the interceptor UAVs to track the target in a dynamic environment. The improved multi-agent deep deterministic policy gradient algorithm includes an independent replay buffer and a multi-dimensional convolutional policy network; the independent replay buffer establishes an independent experience buffer for each UAV agent, and at the same time constructs an experience buffer shared by all agents, and uses a sampling network to determine the sampling ratio of each experience buffer, and adjusts the sampling strategy according to the dynamic changes of the task.

[0009] Further, the interceptor UAV obtains information about other UAVs in its team through image color space conversion and target extraction, centroid position calculation and coordinate conversion, and uses wireless communication to send the target extraction result, the own position information, speed information of the interceptor UAV, and the image feature information obtained by the camera to other UAVs. At the same time, it receives the observation values of other UAVs in the interceptor UAV team, the own position information, speed information of the interceptor UAV, and the image feature information data obtained by the camera, and integrates the observation values from each UAV in its team through data fusion to obtain its own environmental observation value after removing noise.

[0010] Further, the interception constraint conditions of the interceptor UAV include: numerical ratio constraint , represents the maximum numerical ratio of the UAVs, is the total number of interceptor UAVs, is the total number of target UAVs; Each interceptor UAV can intercept at most one target UAV. If there exists such that , then the target UAV is regarded as a successful interception, where represents the maximum number of interception steps, represents the current time step of the UAV's interception, represents the maximum distance for the UAV to successfully intercept, represents the intercepting UAV at the step, represents the target UAV at the step.

[0011] Furthermore, the independent replay buffer constructs an independent individual optimal buffer for each agent while retaining the shared experience buffer; when a new experience tuple is generated, if the old experience tuple in the individual optimal buffer is empty and the current reward value of the intercepting UAV evaluated according to the current interception effect, target allocation situation, and cooperation strategy is greater than zero, then the new experience tuple is stored in the individual optimal buffer; if the old experience tuple is not empty, when the reward of the intercepting UAV is greater than the reward of the intercepting UAV in the old experience tuple, the old experience tuple is replaced with the new experience tuple, otherwise the old experience tuple is retained; where represents the set of observation values of the agent at the current moment, represents the set of action values of the agent at the current moment, represents the set of reward values of the agent at the current moment, represents the set of observation values of the agent at the next moment.

[0012] Furthermore, each agent adopts a sampling network, dynamically adjusts the experience ratio sampled from the individual optimal buffer and the shared experience buffer according to the current training progress and the number of iterations, calculates the sampling ratio based on the output results of the sampling network and the two target evaluation networks for the experience sampled from the shared experience buffer, and limits the ratio of private experience sampling by setting the maximum sampling ratio.

[0013] Furthermore, the multi-dimensional convolutional policy network includes an Actor network, a Critic neural network, and a sampling network; The Actor network receives the observation values of the agent, first extracts features of different scales through two 1D convolutional layers with different kernel sizes, connects the output of the convolutional layer with the observation values to form a new feature vector, then extracts features through an encoding layer using the multi-head attention mechanism, and then further extracts features and performs non-linear mapping through three fully connected multi-layer perceptron layers, and finally outputs an action vector through a softmax layer. The action vector represents the probability distribution of the agent choosing different actions in the current state; The Critic neural network includes two independent Critic networks for each agent. Both of the two independent Critic networks include three fully connected layers, which are used to evaluate the value of each action taken by the agent. The double Q-network structure is adopted to select the smaller value of the outputs of the two independent Critic networks as the final value. The sampling network includes three fully connected layers, which are used to dynamically sample possible targets or actions based on the current observation.

[0014] Further, for the target assignment and interception of the multi-dimensional convolutional policy network, the actions, observations, and reward functions of each agent are determined according to the Markov decision. Each agent selects the interception target according to the probability distribution output by the Actor network, makes decisions based on the observations to obtain environmental and task information, calculates the acceleration of the interception UAV by combining the interception target output by the Actor network with the position information of other UAVs, and realizes target interception.

[0015] Further, the reward function is: ; where represents the interception reward of the interception UAV at time step , represents the task completion reward, represents the task assignment reward, represents the interception UAV at time step of the ranking reward, represents the interception UAV at time step of the task retention reward.

[0016] A system for implementing the multi-agent UAV collaborative interception method based on dynamic sampling includes: a reinforcement learning module and a UAV motion system; The reinforcement learning module includes a reinforcement learning model, which is a model that improves the multi-agent deep deterministic policy gradient algorithm built on the Markov model according to actions, observations, and reward functions; The UAV motion system includes a motion model of the target UAV and a motion model of the interception UAV. The interception UAV is equipped with a camera for collecting target images. By processing the images captured by the UAV on-board camera and combining visual detection technology with deep reinforcement learning through team internal communication, the UAV can track the target using visual information in a complex dynamic environment.

[0017] A computer device of the present invention includes: a memory, a processor, and a computer program stored on the memory. When the computer program is executed on the processor, the described multi-agent UAV collaborative interception method based on dynamic sampling is implemented.

[0018] The present invention has the following advantages and beneficial effects compared with the prior art: 1. Optimize learning efficiency: Establish independent and shared experience buffers, use a dynamic sampling network to determine the sampling ratio, optimize the learning efficiency of the multi-agent system, and the sampling strategy can be dynamically adjusted according to the task. For example, during training, the independent experience buffer only retains the experience that has a positive impact on the agent, avoiding learning low-quality experience and improving the training stability and convergence speed.

[0019] 2. Enhance environmental perception and task allocation capabilities: Introduce a multi-dimensional convolutional structure and a multi-head attention mechanism into the policy network to capture the spatio-temporal relationships between multi-agents and extract features, enhance the agent's perception of the environment, and improve the accuracy of task allocation and interception effects. Experiments show that compared with the algorithm variant lacking this mechanism, the average reward curve of DS-MADDPG is higher in the final convergence stage, and it can more effectively allocate targets and continuously track targets.

[0020] 3. Integrate technologies and verify effectiveness: Combine visual detection technology with deep reinforcement learning to enable UAVs to use visual information to track targets in complex dynamic environments, and verify the effectiveness of the algorithm in the ROS simulation environment. Simulation experiments show that DS-MADDPG is superior to other comparison algorithms in key indicators such as interception rate, average interception distance, and average interception time, improving the UAV task allocation efficiency and collaborative interception capabilities in complex environments. Description of the Drawings

[0021] Figure 1 It is a schematic structural diagram of DS-MADDPG used in the embodiment of the present invention.

[0022] Figure 2 It is a flowchart of a multi-agent UAV collaborative interception method based on dynamic sampling in the embodiment of the present invention.

[0023] Figure 3 It is a comparison chart of evaluation indicators for the interception rate comparison experiment between DS-MADDPG and other algorithms in the embodiment of the present invention.

[0024] Figure 4 It is a comparison chart of evaluation indicators for the interception distance comparison experiment between DS-MADDPG and other algorithms in the embodiment of the present invention.

[0025] Figure 5 It is a comparison chart of evaluation indicators for the interception time comparison experiment between DS-MADDPG and other algorithms in the embodiment of the present invention. Detailed implementation manners

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the specific implementation of the present invention in conjunction with the embodiments and the accompanying drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0027] This embodiment implements a system for a multi-agent UAV collaborative interception method based on dynamic sampling, including a reinforcement learning module and a UAV motion system.

[0028] The reinforcement learning module includes a reinforcement learning model, which is a model that builds an improved multi-agent deep deterministic policy gradient algorithm (DS-MADDPG algorithm) on a Markov model according to actions, observation values, and a reward function. The DS-MADDPG algorithm includes an optimal replay module, a sampling ratio adjustment module, and an intelligent decision-making module.

[0029] The optimal replay module constructs an independent individual optimal buffer for each agent , while retaining a shared experience buffer ; when a new experience tuple is generated, where represents the set of observation values of the agent at the current moment, represents the set of action values of the agent at the current moment, represents the set of reward values of the agent at the current moment, represents the set of observation values of the agent at the next moment. If the old experience tuple in is empty and ,( represents the current reward value of the interceptor UAV evaluated according to the current interception effect, target allocation situation, and cooperation strategy ), then the new experience tuple is stored in the individual optimal buffer ; if the old experience tuple is not empty, when ( is the reward of the UAV in the experience tuple ), the new experience tuple replaces the old experience tuple , otherwise the old experience tuple is retained, so as to ensure the optimal buffer , Only the experiences valuable to the agent are retained. The collaborative strategies include active deduplication when there are conflicts in target allocation, load balancing of targets within the area, target reallocation when a task fails, or multiple agents forming a chain interception formation, etc.

[0030] The intelligent decision-making module includes an Actor network , and a Critic neural network.

[0031] The Actor network receives the observations of the agent , and the observations first extract features of different scales through two 1D convolutional layers with different kernel sizes, connect the output of the convolutional layer with the original observations to form a new feature vector, then use the multi-head attention mechanism to extract features through an encoding layer (TransformerEncoder layer), and then further extract features and perform non-linear mapping through three fully connected multi-layer perceptron (MLP) layers, and finally output a 1× action vector, representing the probability distribution of the agent choosing different actions in the current state, indicating the number of target UAVs.

[0032] The Critic neural network includes two independent Critic networks (target evaluation networks) for each agent, namely the first Critic network and the second Critic network . Each independent Critic network consists of three fully connected layers, used to evaluate the value of each action taken by the agent, adopting a double Q-network structure, and respectively output the corresponding action values and through the first Critic network and of the second Critic network, and take the smaller value of the two as the final evaluation result , to improve the stability during the training process and the conservativeness of policy estimation.

[0033] The sampling ratio adjustment module independently constructs a sampling network for each agent , and each agent independently uses a sampling network to dynamically adjust the experience ratio sampled from the individual optimal buffer and the shared experience buffer D; sample the experience from the individual optimal buffer , and sample the experience by mixing sampling from the individual optimal buffer and the shared experience buffer D. Calculate the sampling ratio At this time, the experiences sampled according to the shared experience buffer In the sampling network And two target evaluation networks (respectively The target evaluation network of And The target evaluation network of ), the output results are adjusted, and the proportion sampled from the private experience is restricted by setting the maximum sampling ratio

[0034] The sampling network Consists of three fully connected layers (MLP layers). The three fully connected layers of the sampling network Are the same as those of , The network architecture. According to the output of the sampling network And , The sampling ratio from the shared experience buffer is dynamically adjusted, reducing the volatility during network training, enhancing training stability, reducing the risk of reward overestimation, and balancing exploration and exploitation.

[0035] The Markov decision process includes the actions of each agent , observations And the reward function ; The agent selects the interception target according to the probability distribution output by the reinforcement learning model, and makes decisions based on the observations to obtain environmental and task information; The reward function consists of interception rewards, task completion rewards, task assignment rewards, task retention rewards, and ranking rewards to accelerate the learning process; According to the interception target output by the Actor network of the reinforcement learning model, combined with the position information of other drones, the acceleration of the interception drone is calculated, including the repulsive acceleration between drones and the attractive acceleration of the target to the drone, to achieve target interception.

[0036] The drone motion system includes the motion models of the target drone and the interception drone. The interception drone is equipped with a camera for collecting target images. By processing the images captured by the on-board camera of the drone and using the internal communication within the team, the visual detection technology is combined with deep reinforcement learning, enabling the drone to track the target using visual information in a complex dynamic environment.

[0037] As Figure 1 , Figure 2 Shown, a multi-agent drone cooperative interception method based on dynamic sampling in this embodiment includes the following steps: ​1. Simulation environment construction and parameter setting, including establishing the motion models of the target UAV and the interceptor UAV. Distributed decision-making and task coordination are achieved through the images captured by the on-board camera of the interceptor UAV and the internal communication within the team.

[0038] Platform selection and initialization: The ROS (Robot Operating System) platform is selected to build a multi-UAV simulation environment. The initialization settings of the simulation scenario are completed in ROS, a virtual space containing the activity areas of the interceptor UAV and the target UAV is constructed, and relevant physical parameters are set to ensure that the environment can accurately simulate the motion laws in the real scenario.

[0039] Establishing the motion model of the target UAV: Assume that the target UAV moves in a uniform straight line with random direction perturbations. For , denotes the th target UAV, denotes the number of target UAVs. The kinematic equation of the th target UAV is , where and respectively represent the position and velocity of the th target UAV at time step , is the time interval between steps.

[0040] The velocity of the target UAV , is the constant velocity of the target UAV, is the flight direction of the target UAV. Among them, represents the pitch angle of the target UAV , which is used to describe the deflection degree of its flight direction in the vertical plane; represents its yaw angle, which is used to describe the deflection angle of its flight direction in the horizontal plane. The two together determine the three-dimensional flight direction of the target UAV. and are the noise terms added to the flight direction of the target UAV at time step , respectively. They are noise terms uniformly distributed within the range of , is the parameter used to control the influence of the noise magnitude on the flight direction.

[0041] The motion model of the interceptor UAV: The position update equation of the th interceptor UAV at time step is ; where , denotes the a drone interceptor, indicating the number of drone interceptors, and respectively represent the th drone interceptor position and velocity, is the time interval between steps, is a set of real numbers of the corresponding data size; the th drone interceptor velocity , is the acceleration, which is used as the control input of the drone, and , represents the maximum acceleration that the drone interceptor can accept. Each tracking drone is equipped with six cameras in different directions for environmental perception, and its detection range is defined accordingly.

[0042] Observation and information sharing: The drone interceptor collects target images through an on-board camera and perceives the environment through internal communication within the drone interceptor team. It obtains information about other drones through image color space conversion and target extraction (such as HSV color space segmentation, image centroid extraction, and depth estimation, etc.). It uses wireless communication to send the target recognition result and the current state (such as position, velocity, observed target information, etc.) to other drones, and at the same time receives data from other drones within the team. It integrates the observations from each drone through data fusion techniques (such as Kalman filtering, multi-sensor fusion, etc.) and obtains its own environmental observation value after removing noise. .

[0043] Constraints for the interception problem: Numerical ratio constraint , represents the maximum number ratio of drones, is a constant greater than 1, is the total number of drone interceptors, is the total number of target drones; if there exists such that , then the target drone is regarded as successfully intercepted. Each drone interceptor can intercept at most one target drone, where represents the maximum number of interception steps, represents the current interception time step of the drone, , , is a small constant, represents the maximum distance for the drone to intercept successfully, represents the coordinates of the drone interceptor at the th step, represents the coordinates of the target drone at the th step.

[0044] As an embodiment, this embodiment sets up 5 interceptor drones and 5 target drones. Six depth cameras are equipped for each interceptor drone to obtain target images and enhance its environmental perception ability. Define the takeoff area of the interceptor drone as , and the generation area of the target drone as , and clarify the relevant area parameters, such as , , . Among them, represents a set of real numbers, represents the takeoff area of the interceptor drone, represents the generation area of the target drone, represents the radius of the generation area of the interceptor drone, represents the radius of the generation area of the target drone, represents the generation height of the target drone.

[0045] Parameter setting: The set parameters include the speed of the target drone, the horizontal and vertical field of view angles of the camera, the maximum observation distance of the camera, and other key parameters. Other key parameters such as the proportional factor for controlling the force between drones, the allowable target interception distance error, etc. These parameters are determined according to the actual application scenario and research requirements, and are used to accurately simulate the movement and interception process of the drones.

[0046] As an embodiment, the parameters are set as follows: the speed of the target drone , the horizontal and vertical field of view angles of the camera , , the maximum observation distance of the camera , the proportional factor for controlling the force between drones, the proportional factor for adjusting the repulsive force intensity , the proportional factor for adjusting the attractive force intensity , the allowable target interception distance error .

[0047] The interceptor drone realizes the three-dimensional precise positioning of the drone through methods such as image color space conversion and target extraction, centroid position calculation and coordinate conversion to process image data.

[0048] Image color space conversion and target extraction: First, convert the image collected by the camera to the HSV color space, and use the threshold segmentation method to segment different regions of different drones to generate a binary image.

[0049] Centroid position calculation and coordinate conversion to realize the three-dimensional precise positioning of the drone: After performing morphological opening operation on the binary image, calculate the centroid position of the target through the formulas , ​ , represents the first moment of the image in the direction, that is, the cumulative sum of the abscissa of each pixel in the target area multiplied by the pixel value. represents the first moment of the image in the direction, that is, the cumulative sum of the ordinate of each pixel in the target area multiplied by the pixel value. represents the zero - order moment of the image, that is, the cumulative sum of all pixel values in the target area. Combining depth information and the camera internal parameters (focal length , and the target centroid position ), using the formula to convert to the 3D coordinates in the camera coordinate system , and then through to convert to the world coordinate system , where represents the rotation matrix from the camera coordinate system to the world coordinate system, is the translation vector, representing the position of the origin of the world coordinate system in the camera coordinate system.

[0050] 2. Construct an improved multi - agent deep deterministic policy gradient algorithm (DS - MADDPG). The improved multi - agent deep deterministic policy gradient algorithm (DS - MADDPG) includes an independent replay buffer and a policy network with multi - dimensional convolution for target interception and target allocation.

[0051] The independent replay buffer establishes an independent experience buffer for each UAV agent, and at the same time constructs a shared experience buffer for all agents, and uses a dynamic sampling network to determine the sampling ratio of each experience buffer, and adjusts the sampling strategy according to the dynamic change of the task (for example, preferentially select experience tuples with high interception success rewards). The policy network with multi - dimensional convolution introduces a multi - dimensional convolution structure into the policy network of the traditional multi - agent deep deterministic policy gradient (MADDPG), uses convolutional neural networks (CNNs) of different dimensions to capture the spatio - temporal relationships between multiple agents, and at the same time adopts a multi - head attention mechanism to extract features from the captured relationships, enhancing the agents' perception ability of the environment and improving the accuracy of task allocation and interception.

[0052] The processes of constructing the independent replay buffer and the policy network with multi - dimensional convolution are as follows: 1). Code initialization: Integrate the code of the improved multi - agent deep deterministic policy gradient algorithm (DS - MADDPG) into the simulation environment and complete the initialization work. Initialize the individual optimal buffer and the shared experience buffer for each UAV agent , while initializing the main network: the Actor network , the Critic network , and the sampling network , and randomly assign network weights. For the target networks of each network , , , assign the same initial weights as the original network.

[0053] The architecture of the Critic network and : Each agent has two independent Critic networks, namely and . Each independent Critic network consists of three fully connected layers, which are used to evaluate the value of each action taken by the agent. The double Q-network structure is adopted to select the smaller value of the outputs of the two independent Critic networks , as the final output value , in order to reduce the risk of reward overestimation.

[0054] The architecture of the sampling network : It consists of three MLP layers, which have the same network architecture as and . It is used to dynamically adjust the sampling ratio from the shared experience buffer according to the ratio of its own output to the outputs of , , reduce the volatility during network training, enhance training stability, reduce the risk of reward overestimation, and balance exploration and exploitation.

[0055] The Actor network receives the observations of the agent . First, it extracts features of different scales through two 1D convolutional layers with different kernel sizes, connects the output of the convolutional layer with the original observations to form a new feature vector, then uses the multi-head attention mechanism to extract features through the TransformerEncoder layer, and then further extracts features and performs non-linear mapping through three fully connected multi-layer perceptron (MLP) layers. Finally, it outputs a 1×M action vector through the softmax layer, representing the probability distribution of the agent choosing different actions in the current state.

[0056] The target networks , , and are all the same as the corresponding main networks ( , , , 1) Maintain a consistent network structure. Through the delayed update mechanism, the target network ensures stability during the training process and provides reliable policies and value evaluations for the agent.

[0057] 2) Hyperparameter setting: Determine the hyperparameters required for algorithm training, such as the maximum sampling ratio , the total number of training episodes , the training frequency , the soft update rate , the exploration noise , the discount factor , the maximum number of steps for the sampling network , the batch size , the size of the independent optimal buffer , the size of the shared experience buffer . These hyperparameters have an important impact on the algorithm performance and need to be reasonably selected through multiple experiments and analyses.

[0058] 3) Reward function design and setting: Design a dense reward function with multiple dimensions to guide the UAV agent to learn effective interception strategies. The reward function consists of the following parts: Interception reward : When , indicates that the interceptor UAV successfully intercepts the target; when ( ), indicates that in addition to the interceptor UAV , other interceptor UAVs successfully intercept the target. represents the th interceptor UAV's position at time step ; represents the maximum distance for the UAV to successfully intercept.

[0059] Task completion reward : At the end of the total training episode, if the interception success rate ( ), then ; if , then ; at other times , where is the threshold for the interception success rate to judge whether the task is successfully completed. The interception success rate refers to the proportion of target UAVs successfully intercepted in the current episode.

[0060] Task allocation reward : When the target UAV index sets selected by all UAVs are equal to the current surviving target UAV index set ; Otherwise ; .

[0061] Mission retention reward : If the interceptor UAV maintains the target unchanged at adjacent time steps, i.e., , then ; Otherwise , represents the index of the target UAV assigned to the interceptor UAV at time .

[0062] Ranking reward : , where , , and are calculated based on the height and distance rankings of the target UAVs respectively, and are used to optimize the target assignment. Among them, represents the height ranking of the target UAV assigned to the interceptor UAV . The distance ranking of the target UAV assigned to the interceptor UAV .

[0063] 4), Target assignment process: Model the multi-UAV interception problem as a Markov decision process, and define the system state space, agent action space, observation space, state transition function, and reward function. The action selects the target UAV based on the probability distribution output by reinforcement learning, , represents the probability that the agent selects the th target UAV. The agent selects the target UAV with the highest probability, i.e., ; The observation value contains the position, speed, and target recognition information of various UAVs. Among them, represents the set of positions of all interceptor UAVs at time step . represents the set of positions of the target UAVs at time step . represents the set of index numbers of the target UAVs currently responsible for being intercepted by the interceptor UAVs at time step . is the velocity vector of the interceptor UAV at time step . represents the total number of intercepted drones, represents the total number of target drones. Reward , consists of interception reward , mission completion reward (reward or punishment is given according to mission completion and interception success rate), mission assignment reward (reward is given when the target assignment is reasonable), ranking reward (encouraging to select targets according to altitude and distance) and mission save reward (encouraging continuous tracking of the same target), and guides the agent's decision-making through rewards.

[0064] Target interception process: Under the reinforcement learning framework, the Actor network outputs the interception target , and the tracking drone calculates the acceleration based on this target and the position information of other drones , where is the repulsive acceleration between drones. When , , is the threshold for judging whether two intercepted drones are too close, is the proportionality factor for adjusting the repulsive force intensity; is the attractive acceleration of the target to the drone. When , , is the proportionality factor for adjusting the attractive force intensity to achieve the interception control of the target. represents the intercepted drone The current position of the target drone assigned to the current mission . represents the intercepted drone The flight position at the current moment.

[0065] Network training and optimization: During the entire simulation process, network training and update are the key links to improve the performance of the DS-MADDPG algorithm and achieve efficient cooperative interception of drones. The specific operations are as follows: 1). Data collection and storage: In each current training step of the training, each intercepted drone selects a target drone to perform an interception operation according to the action probability distribution output by the Actor network . After that, the drone observes the change of the environmental state, obtains the next state , reward and the new observation value . Subsequently, when generating a new experience tuple , where represents the set of the agent's current observation values, Represents the set of action values of the agent at the current moment, Represents the set of reward values of the agent at the current moment, Represents the set of observation values of the agent at the next moment. These stored experience data will provide learning materials for subsequent network updates.

[0066] 2). Network update, including: Sampling operation: According to the DS-MADDPG algorithm process, during training, according to the current (current training step size) and the specific requirements of the algorithm, sample the data.

[0067] When ( is the maximum training step size), randomly draw samples from the shared experience buffer D and input them into the target network 、 and the sampling network to determine the sampling ratio through the calculations of these networks. The calculation method is: . Then, draw a mini-batch of samples from the shared experience buffer D and the independent optimal buffer in the ratio of , and when , directly sample from the shared experience buffer samples . Represents a set of experience tuples, Represents another set of experience tuples. Is an operation of the function used to limit the value within a specific range. Is the value output by the sampling network . And Are the values output by two different target networks 、 .

[0068] Calculate the target value: For different network update requirements, calculate the corresponding target values respectively. When updating the critic network and , calculate the target value . When sampling from a mixture of two buffers, ; when sampling only from the shared experience buffer, . When updating the sampling network , calculate the target value 。The calculation of these target values provides an important reference basis for subsequent network weight updates. Among them, is the current reward value for intercepting the drone , is the discount factor, which is used to adjust the impact of future rewards on the current decision. is the value function in the target network, which is calculated based on the current observation value and the target policy and is used to evaluate the long-term rewards that the agent may obtain at the next moment.

[0069] Update of each network weight: The weights of each network are updated by minimizing the loss function. For the critic network and , the loss function is represents the number of samples (or the number of trajectories), and by adjusting its weight , the critic network can more accurately evaluate the action value. For the sampling network , the loss function is , and its weights are updated accordingly to optimize the adjustment strategy of the sampling ratio. For the Actor network , the policy gradient (when mixed sampling) or (when only sampling from the shared experience buffer) is used to update the weights , so that the Actor network can output a better action decision. Among them represents the objective function for taking the gradient of the parameters of the agent , that is, the gradient of the loss function for the agent , represents the number of samples (or the number of trajectories), that is, the number of samples used to calculate the gradient, is a function of the observation value and the action of the agent and , is a function of the observation value and the action of the agent and represents the gradient of the action of the agent with respect to a certain objective function , represents the gradient of the action of the agent with respect to a certain objective function , ​​​​Represents according to the agent's observation value agent 's policy with respect to the parameter gradient, given the agent's observation value , Represents according to the agent's observation value agent 's policy with respect to the parameter gradient, Represents the agent's policy function, which outputs the probability distribution of the agent executing a certain action when given the observation value , Represents the agent's policy function output given the observation value when the agent executes a certain action probability distribution.

[0070] Target network soft update: To make the network training more stable, the target network is soft updated. The soft update formula , , , is adopted, where is the soft update rate. This soft update method enables the target network to gradually track the changes of the main network and avoids training instability caused by too fast network update.

[0071] Network training metric judgment: During the training process, continuously monitor training metrics such as average reward value, interception success rate, etc. These metrics are important bases for measuring the training effect of the algorithm. When the training result reaches the preset training metrics, stop training. The setting of the preset metrics needs to comprehensively consider the actual application requirements and the expected algorithm performance. For example, it can be set that the average reward value reaches a certain level, or the interception success rate exceeds a specific threshold, etc. as the condition for training completion.

[0072] Simulation result evaluation and analysis, including: Comparison experiment setting: To verify the performance of the DS-MADDPG algorithm, algorithms such as MADDPG, MAAC, MAPPO, MATD3, etc. are selected for comparison. Under the same simulation environment and parameter settings, each algorithm is trained and tested separately to ensure the consistency of the experimental conditions.

[0073] Performance metric evaluation: Test the trained policy networks of each algorithm in the ROS simulation environment, and each algorithm is tested independently 500 times. Such as Figure 3 , Figure 4 , Figure 5, record and analyze key performance indicators such as average interception rate, average interception distance, and average interception time. By comparing the performance of different algorithms in these metrics, evaluate the superiority and effectiveness of the DS-MADDPG algorithm.

[0074] Result analysis and summary: According to the simulation results, analyze the advantages and disadvantages of the DS-MADDPG algorithm in aspects such as target allocation and interception effect. Study the adaptability and stability of the algorithm in different scenarios, providing a basis for further improving the algorithm and optimizing the UAV cooperative interception strategy.

[0075] The present invention combines visual detection technology with deep reinforcement learning by processing images captured by the UAV's on-board camera and leveraging internal team communication, enabling the UAV to track targets using visual information in complex dynamic environments; integrating the algorithm into the Robot Operating System (ROS) simulation environment to verify the effectiveness of the method in actual tasks.

[0076] The preferred embodiments of the present invention disclosed above are only used to assist in explaining the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, enabling those skilled in the relevant technical field to understand and utilize the present invention well.

Claims

1. A multi-agent UAV cooperative interception method based on dynamic sampling, characterized in that It includes the following steps: (1) The interceptor UAV captures target images through an on-board camera and obtains its own environmental observation values through internal communication within the interceptor UAV team to sense the environment; (2) The improved multi-agent deep deterministic policy gradient algorithm controls the interceptor UAV based on the target images captured by the interceptor UAV and its own environmental observation values, enabling the interceptor UAV to track the target in a dynamic environment; The improved multi-agent deep deterministic policy gradient algorithm includes an independent replay buffer and a policy network with multi-dimensional convolution; each UAV agent in the independent replay buffer establishes an independent experience buffer, and at the same time constructs an experience buffer shared by all agents, and uses a sampling network to determine the sampling ratio of each experience buffer, and adjusts the sampling strategy according to the dynamic changes of the task.

2. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 1, wherein, The interceptor UAV obtains information about other UAVs in the interceptor UAV team through image color space conversion, target extraction, centroid position calculation, and coordinate conversion, and uses wireless communication to send the target extraction results, the own position information, speed information of the interceptor UAV, and the image feature information obtained by the camera to other UAVs. At the same time, it receives the observation values of other UAVs in the interceptor UAV team, the own position information, speed information of the interceptor UAV, and the image feature information data obtained by the camera, and integrates the observation values from each UAV in the team through data fusion, and obtains its own environmental observation values after removing noise.

3. The multi-agent UAV collaborative interception method based on dynamic sampling according to claim 1, characterized in that The interception constraint conditions for intercepting drones include: numerical ratio constraint , represents the maximum numerical ratio of the drones, is the total number of drones to be intercepted, is the total number of target drones; Each intercepting drone can intercept at most one target drone. If there exists , such that , then the target drone is considered to be successfully intercepted, where represents the maximum number of interception steps, represents the current time step of the drone's interception, represents the maximum distance for the drone to successfully intercept, represents the intercepting drone at the step coordinates, represents the target drone at the step coordinates.

4. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 1, characterized in that The independent replay buffer constructs an independent individual optimal buffer for each agent while retaining a shared experience buffer; when a new experience tuple is generated, if the old experience tuple in the individual optimal buffer is empty and the current reward value of the intercepting UAV evaluated according to the current interception effect, target allocation situation and cooperation strategy is greater than zero, the new experience tuple is stored in the individual optimal buffer; if the old experience tuple is not empty, when the reward of the intercepting UAV is greater than the reward of the intercepting UAV in the old experience tuple , the old experience tuple is replaced with the new experience tuple, otherwise the old experience tuple is retained; where represents the set of observation values of the agent at the current moment, represents the set of action values of the agent at the current moment, represents the set of reward values of the agent at the current moment, represents the set of observation values of the agent at the next moment.

5. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 4, wherein, Each agent uses a sampling network to dynamically adjust the experience ratio sampled from the individual optimal buffer and the shared experience buffer according to the current training progress and the number of iterations. The experience sampled from the shared experience buffer is used to calculate the sampling ratio based on the output results of the sampling network and the two target evaluation networks, and the ratio of private experience sampling is limited by setting the maximum sampling ratio.

6. The multi-agent UAV collaborative interception method based on dynamic sampling according to claim 1, characterized in that, The policy network with multi-dimensional convolution includes an Actor network, a Critic neural network, and a sampling network; The Actor network receives the observation value of the agent, first extracts features of different scales through two 1D convolutional layers with different kernel sizes, connects the output of the convolutional layer with the observation value to form a new feature vector, then uses the multi-head attention mechanism to extract features through the encoding layer, and then further extracts features and performs non-linear mapping through three fully connected multi-layer perceptron layers. Finally, it outputs an action vector through the softmax layer, and the action vector represents the probability distribution of the agent choosing different actions in the current state; The Critic neural network includes two independent Critic networks for each agent. Both of the two independent Critic networks include three fully connected layers, which are used to evaluate the value of each action taken by the agent. The double Q-network structure is adopted to select the smaller value of the outputs of the two independent Critic networks as the final value; the sampling network includes three fully connected layers, which are used to dynamically sample possible targets or actions according to the current observation value.

7. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 6, characterized in that, The target allocation and interception of the multi-dimensional convolutional strategy network are based on Markov decision to determine the actions, observations and reward functions of each agent. Each agent selects the interception target according to the probability distribution output by the Actor network, obtains the environment and task information based on the observations to make decisions, and calculates the acceleration of the intercepted drone based on the interception target output by the Actor network and the position information of other drones to achieve target interception.

8. The multi-agent UAV collaborative interception method based on dynamic sampling according to claim 7, characterized in that, The reward function is :[[]]END]] ； Among them represents the interception of the UAV at the time step interception reward represents the mission completion reward represents the mission assignment reward represents the interception of the UAV at the time step ranking reward represents the interception of the UAV at the time step mission retention reward 9. A system for implementing the multi-agent UAV cooperative interception method based on dynamic sampling described in claim 1, characterized in that, include: Reinforcement learning module and drone motion system; The reinforcement learning module includes a reinforcement learning model, which is a model of an improved multi-agent deep deterministic policy gradient algorithm built on a Markov model based on actions, observations and reward functions; The drone motion system includes the motion model of the target drone and the motion model of the intercepting drone. The intercepting drone is equipped with a camera for collecting target images. By processing the images captured by the drone's onboard camera and relying on internal team communication, it combines visual detection technology with deep reinforcement learning, enabling the drone to use visual information to track targets in complex dynamic environments.

10. A computer device, characterized in that, include: A memory and a processor and a computer program stored in the memory, when the computer program is executed on the processor, implements a multi-agent UAV collaborative interception method based on dynamic sampling as described in any one of claims 1 to 8.

Images