A multi-agent unmanned aerial vehicle cooperative interception method and system based on dynamic sampling

Abstract

The application discloses a kind of multi-agent unmanned aerial vehicle cooperative interception method and system based on dynamic sampling, comprising the following steps, by introducing independent experience cache and dynamic sampling network, the experience quality degradation and policy convergence slow problem of multiple agents in training process is solved, each interception unmanned aerial vehicle is equipped with six-direction camera, for obtaining target image, by HSV color space segmentation, image centroid extraction and depth estimation method, three-dimensional positioning of target unmanned aerial vehicle is realized, meanwhile, the sharing of observation and target state is realized by means of in-group communication between unmanned aerial vehicles, and then distributed decision and task cooperation are achieved, in addition, the system covers multiple dimensions of intensive reward mechanism, including interception success, target distribution rationality, task maintenance and the like, which significantly improves the stability and cooperativity of agent policy, and the application has obvious advantages in interception success rate, allocation efficiency and response speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicles, in particular to a multi-agent unmanned aerial vehicle cooperative interception method and system based on dynamic sampling. BACKGROUND

[0002] With the development of unmanned aerial vehicle technology, its application in public management, logistics transportation, environmental monitoring and other fields is becoming more and more widespread. However, the widespread use of unmanned aerial vehicles may have an impact on public environment and facilities, making the research of unmanned aerial vehicle interception technology crucial.

[0003] In the field of unmanned aerial vehicle interception technology, algorithms are mainly divided into rule-based algorithms and learning-based algorithms. Early rule-based unmanned aerial vehicle interception algorithms have achieved certain results in specific scenarios, but have obvious limitations in the face of complex dynamic environments. These algorithms are difficult to adapt to the rapid changes of environment and task requirements, resulting in low interception rate and poor interception performance.

[0004] To overcome the shortcomings of rule-based algorithms, learning-based interception algorithms have emerged. Among them, the multi-agent deep deterministic policy gradient (MADDPG) algorithm has attracted widespread attention in multi-unmanned aerial vehicle cooperative tasks. (Zhang Bo, Liu Man Guo, Liu Meng Yan. Multi-unmanned aerial vehicle cooperative attack method based on MADDPG [J / OL]. Journal of Missiles and Guidance, 1-9 [2025-04-21]). By constructing a typical confrontation scene, the multi-unmanned aerial vehicle cooperative attack problem is modeled as a distributed partially observable Markov decision process (Dec-POMDP), a unique reward function is designed, and the MADDPG algorithm is used to train the attack strategy.

[0005] However, the traditional MADDPG algorithm still has some shortcomings in practical application. On the one hand, the shared experience replay buffer mechanism used by it cannot effectively distinguish the quality of experience, and a large amount of low-quality experience participating in replay will have a negative impact on network training, resulting in low training efficiency and difficulty in converging to the optimal solution. On the other hand, the strategy network of the traditional MADDPG algorithm has limited ability to handle complex spatio-temporal relationships between multiple agents, making it difficult to accurately capture environmental information and affecting the accuracy of task allocation and interception decisions. In addition, in complex dynamic environments, unmanned aerial vehicles need more accurate environmental perception capabilities, but existing methods are not deep enough in the integration of visual detection technology and reinforcement learning, and cannot fully utilize visual information to improve the target tracking and interception performance of unmanned aerial vehicles. Under such technical background, how to improve the MADDPG algorithm and improve the cooperative interception capability of multi-unmanned aerial vehicles in complex environments has become an important issue in the field of unmanned aerial vehicle interception technology. SUMMARY

[0006] The application provides a multi-agent unmanned aerial vehicle cooperative interception method and system based on dynamic sampling, which improves the cooperative interception capability of the multi-unmanned aerial vehicle in a complex dynamic environment by establishing an independent playback buffer, adopting a multi-dimensional convolution strategy network, and fusing visual detection technology and deep reinforcement learning, and is especially suitable for the cooperative interception task of the multi-unmanned aerial vehicle on the target unmanned aerial vehicle in a complex dynamic environment, and can be widely applied in the fields of public management and emergency response to realize efficient and accurate target interception and task execution.

[0007] The application is implemented by at least one of the following technical solutions.

[0008] A multi-agent unmanned aerial vehicle cooperative interception method based on dynamic sampling comprises the following steps.

[0009] (1) The interception unmanned aerial vehicle collects target images through an onboard camera, and obtains self-environment observation values through internal communication of the interception unmanned aerial vehicle team;

[0010] (2) An improved multi-agent deep deterministic policy gradient algorithm controls the interception unmanned aerial vehicle through the target images captured by the interception unmanned aerial vehicle and the self-environment observation values, so that the interception unmanned aerial vehicle can track the target in a dynamic environment.

[0011] The improved multi-agent deep deterministic policy gradient algorithm comprises an independent playback buffer and a multi-dimensional convolution strategy network; the independent playback buffer establishes an independent experience buffer for each unmanned aerial vehicle agent, simultaneously constructs an experience buffer shared by all agents, and determines the sampling proportion of each experience buffer by using a sampling network, and adjusts the sampling strategy according to the dynamic change of the task.

[0012] Further, the interception unmanned aerial vehicle obtains the information of other unmanned aerial vehicles in the team through image color space conversion and target extraction, centroid position calculation and coordinate conversion, sends the target extraction result and the self-position information, speed information and image feature information obtained by the camera of the interception unmanned aerial vehicle to other unmanned aerial vehicles by using wireless communication, simultaneously receives the observation values of other unmanned aerial vehicles in the interception unmanned aerial vehicle team and the self-position information, speed information and image feature information data of the camera of the interception unmanned aerial vehicle, integrates the observation values from each unmanned aerial vehicle in the team through data fusion, and obtains the self-environment observation values after removing noise.

[0013] Further, the interception constraint condition of the interception unmanned aerial vehicle comprises a numerical ratio constraint , represents the maximum numerical ratio of the unmanned aerial vehicle, is the total number of the interception unmanned aerial vehicles, is the total number of the target unmanned aerial vehicles.

[0014] Each interceptor drone intercepts at most one target drone, if there is such that the target drone is considered to be successfully intercepted, wherein denotes the maximum number of interception steps, denotes the current interception time step of the drone, denotes the maximum distance of successful interception of the drone, denotes the coordinates of the interceptor drone at the first step, denotes the coordinates of the target drone at the first step.

[0015] Further, the independent replay buffer constructs an individual optimal buffer for each agent while retaining a shared experience buffer; when a new experience tuple is generated, if the old experience tuple in the individual optimal buffer is empty and the current reward value of the interceptor drone obtained according to the current interception effect, target allocation and cooperation strategy evaluation is greater than zero, the new experience tuple is stored in the individual optimal buffer; if the old experience tuple is not empty, when the reward of the interceptor drone is greater than the reward of the interceptor drone in the old experience tuple, the old experience tuple is replaced with the new experience tuple, otherwise the old experience tuple is retained; wherein denotes the set of observation values of the agent at the current time, denotes the set of action values of the agent at the current time, denotes the set of reward values of the agent at the current time, denotes the set of observation values of the agent at the next time.

[0016] Further, each agent adopts a sampling network to dynamically adjust the experience sampling proportion from the individual optimal buffer and the shared experience buffer according to the current training progress and the number of iterations, and the sampling proportion is calculated according to the output results of the shared experience buffer sampling experience in the sampling network and the two target evaluation networks, and the proportion of private experience sampling is limited by setting the maximum sampling proportion.

[0017] Further, the multi-dimensional convolution strategy network includes an Actor network, a Critic neural network and a sampling network.

[0018] The Actor network receives the agent's observations and first extracts features of different scales through two 1D convolutional layers with different kernel sizes. The convolutional layer output is concatenated with the observations to form a new feature vector. The new feature vector is then extracted using a multi-head attention mechanism in the encoding layer. Further feature extraction and nonlinear mapping are performed through three fully connected multi-layer perceptron layers. Finally, the softmax layer outputs an action vector, which represents the probability distribution of different actions selected by the agent in the current state.

[0019] The critic neural network includes two independent critic networks for each agent. Both independent critic networks include three fully connected layers, which are used to evaluate the value of each action taken by the agent. A double Q network structure is used to select the smaller value of the output of the two independent critic networks as the final value; the sampling network includes three fully connected layers, which is used to dynamically sample possible targets or actions based on the current observation value.

[0020] Furthermore, the target allocation and interception of the multi-dimensional convolutional policy network are based on Markov decision-making to determine the actions, observations, and reward functions of each agent. Each agent selects the interception target based on the probability distribution output by the Actor network, obtains environment and task information based on the observations to make decisions, and calculates the acceleration of the intercepting drone based on the interception target output by the Actor network and the position information of other drones to achieve target interception.

[0021] Furthermore, the reward function for:

[0022] ;

[0023] in Indicates interception of drones At time step The interception reward, Indicates the reward for completing the task. represents the task assignment reward, Indicates interception of drones At time step Ranking rewards, Indicates interception of drones At time step The tasks remain rewarding.

[0024] A system for implementing the multi-agent UAV collaborative interception method based on dynamic sampling includes: a reinforcement learning module and a UAV motion system;

[0025] The reinforcement learning module comprises a reinforcement learning model, which is a model of an improved multi-agent deep deterministic policy gradient algorithm built on a Markov model according to actions, observation values and a reward function.

[0026] The unmanned aerial vehicle motion system comprises a motion model of the target unmanned aerial vehicle and a motion model of the intercepting unmanned aerial vehicle, the intercepting unmanned aerial vehicle is provided with a camera for capturing target images, the images captured by the camera onboard the unmanned aerial vehicle are processed, and the visual detection technology is combined with deep reinforcement learning by means of team internal communication, so that the unmanned aerial vehicle can track the target by using visual information in a complex dynamic environment.

[0027] The computer device comprises a memory, a processor and a computer program stored in the memory, and when the computer program is executed on the processor, the computer program implements the multi-agent unmanned aerial vehicle cooperative interception method based on dynamic sampling.

[0028] The present application has the following advantages and beneficial effects compared with the prior art:

[0029] 1. Optimizing learning efficiency: independent and shared experience buffers are established, and a dynamic sampling network is used to determine the sampling ratio, so as to optimize the learning efficiency of the multi-agent system, and the sampling strategy can be dynamically adjusted according to the task. For example, in training, the independent experience buffer only retains the experience that has a positive impact on the agent, avoids learning low-quality experience, and improves the training stability and convergence speed.

[0030] 2. Enhancing environmental perception and task allocation capability: a multi-dimensional convolution structure and a multi-head attention mechanism are introduced into the policy network to capture the spatio-temporal relationship between multi-agents and extract features, enhance the agent's perception of the environment, and improve the accuracy of task allocation and interception effect. Experiments show that, compared with the algorithm variant lacking the mechanism, the average reward curve of DS-MADDPG is higher in the final convergence stage, and the target can be more effectively allocated and tracked continuously.

[0031] 3. Fusion technology and verification of effectiveness: the visual detection technology and deep reinforcement learning are combined, so that the unmanned aerial vehicle can track the target by using visual information in a complex dynamic environment, and the effectiveness of the algorithm is verified in the ROS simulation environment. Simulation experiments show that DS-MADDPG is superior to other comparative algorithms in terms of interception rate, average interception distance and average interception time, and improves the task allocation efficiency of the unmanned aerial vehicle and the cooperative interception capability in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The structure diagram of DS-MADDPG used in the embodiments of the present application.

[0033] Figure 2A flow chart of a multi-agent unmanned aerial vehicle cooperative interception method based on dynamic sampling in an embodiment of the present application.

[0034] Figure 3 An evaluation index comparison graph for a comparison experiment of interception rates of DS-MADDPG and other algorithms in an embodiment of the present application.

[0035] Figure 4 An evaluation index comparison graph for a comparison experiment of interception distances of DS-MADDPG and other algorithms in an embodiment of the present application.

[0036] Figure 5 An evaluation index comparison graph for a comparison experiment of interception times of DS-MADDPG and other algorithms in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the specific implementation of the present application will be described below in conjunction with the embodiments and the accompanying drawings. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] The embodiment realizes a system of a multi-agent unmanned aerial vehicle cooperative interception method based on dynamic sampling, which comprises a reinforcement learning module and an unmanned aerial vehicle motion system.

[0039] The reinforcement learning module comprises a reinforcement learning model, which is a model of an improved multi-agent deep deterministic policy gradient algorithm (DS-MADDPG algorithm) built on a Markov model according to actions, observation values and reward functions. The DS-MADDPG algorithm comprises an optimal replay module, a sampling ratio adjustment module and an intelligent decision module.

[0040] The optimal replay module constructs an independent individual optimal buffer for each agent , while retaining a shared experience buffer ; when a new experience tuple is generated, wherein represents a set of observation values of the agent at the current time, represents a set of action values of the agent at the current time, represents a set of reward values of the agent at the current time, represents a set of observation values of the agent at the next time, if the old experience tuple in , ( Represents the intercepted drone obtained based on the current interception effect, target allocation and collaboration strategy evaluation The current reward value of ), then the new experience tuple Stored in individual optimal buffer ; If the old experience tuple Not empty, when hour( is the experience tuple Medium UAV reward), with the new experience tuple Replace the old experience tuple , otherwise keep the old experience tuple , thereby ensuring the optimal buffer zone Only valuable experiences are retained. Collaboration strategies include active deduplication when target allocation conflicts occur, load balancing of targets within a region, target reallocation when missions fail, or chained interception formations formed by multiple agents.

[0041] The intelligent decision-making module includes an Actor network , Critic neural network.

[0042] The Actor Network Receive observations from the agent , observed value First, two 1D convolutional layers with different kernel sizes are used to extract features of different scales. The output of the convolutional layer is connected with the original observation value to form a new feature vector. Then, the feature is extracted by the multi-head attention mechanism through the encoding layer (TransformerEncoder layer). After that, three fully connected multi-layer perceptron (MLP) layers are used for further feature extraction and nonlinear mapping. Finally, the 1× The action vector represents the probability distribution of the agent choosing different actions in the current state. Indicates the number of target drones.

[0043] The Critic neural network includes two independent Critic networks (target evaluation networks) for each agent, namely the first Critic network And the second Critic network Each independent Critic network consists of three fully connected layers, which are used to evaluate the value of each action taken by the agent. It adopts a double Q network structure and passes through the first Critic network. and the second Critic network Output corresponding action value and , and take the smaller value of the two as the final evaluation result , to improve the stability of the training process and the conservatism of the policy estimation.

[0044] The sampling ratio adjustment module independently constructs a sampling network for each agent , each agent independently uses a sampling network to dynamically adjust the experience ratio sampled from the individual optimal buffer and the shared experience buffer D according to the current training progress and the number of iterations; the experience is sampled from the individual optimal buffer , and the experience is mixed sampled from the individual optimal buffer and the shared experience buffer D . When calculating the sampling ratio , the experience sampled from the shared experience buffer is adjusted according to the output results of the sampling network and the two target evaluation networks (the target evaluation network for the other agent and the target evaluation network for the agent) , and the maximum sampling ratio is set to limit the sampling ratio from the private experience.

[0045] The sampling network is composed of three fully connected layers (MLP layers), and the three fully connected layers of the sampling network are the same as the network architecture of , The sampling ratio from the shared experience buffer is dynamically adjusted according to the ratio of the output of the sampling network to the output of , , which reduces the volatility of the network training process, enhances the training stability, reduces the risk of overestimation of rewards, and balances exploration and utilization.

[0046] The Markov decision process includes the action , observation value and reward function of each agent ; the agent selects the interception target according to the probability distribution output by the reinforcement learning model, and makes decisions according to the observation value to obtain environment and task information; the reward function is composed of interception reward, task completion reward, task allocation reward, task maintenance reward and ranking reward to speed up the learning process; according to the interception target output by the Actor network of the reinforcement learning model, the acceleration of the interception UAV is calculated combining with other UAV position information, including repulsive acceleration between UAVs and attractive acceleration of the target to the UAV, to realize target interception.

[0047] ​​The drone motion system includes the motion model of the target drone and the motion model of the intercepting drone. The intercepting drone is equipped with a camera for collecting target images. By processing the images captured by the drone's onboard camera and relying on internal team communication, it combines visual detection technology with deep reinforcement learning, enabling the drone to use visual information to track targets in complex dynamic environments.

[0048] like Figure 1 、 Figure 2 As shown, a multi-agent UAV collaborative interception method based on dynamic sampling in this embodiment includes the following steps:

[0049] 1. Simulation environment setup and parameter setting, including the creation of motion models for the target and intercepting drones. Distributed decision-making and task coordination are achieved through imagery captured by the intercepting drone's onboard camera and internal team communication.

[0050] Platform Selection and Initialization: We selected the ROS (Robot Operating System) platform to build a multi-drone simulation environment. We initialized the simulation scene in ROS, constructed a virtual space containing the activity areas of the intercepting and target drones, and set relevant physical parameters to ensure the environment accurately simulated the motion patterns of a real-world scenario.

[0051] Establish the target UAV motion model: Assume that the target UAV moves in a uniform straight line with random disturbances in direction. , Indicates the Target drones, Indicates the number of target drones, The kinematic equation of a target UAV is: ,in and Respectively represent The target UAV is in the time step The position and velocity of is the time interval between steps.

[0052] Target drone speed , is the constant speed of the target UAV, is the flight direction of the target drone, Indicates target drone The pitch angle is used to describe the degree of deflection of its flight direction in the vertical plane; It represents the yaw angle, which is used to describe the deflection angle of its flight direction in the horizontal plane. The two together determine the three-dimensional flight direction of the target UAV. and is in the time step respectively flight direction of the target UAV is a noise term uniformly distributed in , which is a parameter for controlling the influence of noise on flight direction.

[0053] Motion model of the interceptor UAVs: the position update equation of the th interceptor UAV at time step is ; where , denotes the th interceptor UAV, denotes the number of interceptor UAVs, and denote the position and velocity of the th interceptor UAV respectively, is the time interval between steps, is a real number set of corresponding data size; the velocity of the th interceptor UAV is the acceleration, which is the control input of the UAV, and , denotes the maximum acceleration that the interceptor UAV can accept. Each interceptor UAV is equipped with six cameras in different directions for environmental perception, and the detection range is defined accordingly.

[0054] Observation and information sharing: the interceptor UAVs collect target images through on-board cameras and perceive the environment through internal communication of the interceptor UAV team. Through image color space conversion and target extraction (such as HSV color space segmentation, image centroid extraction, and depth estimation methods), other UAV information is obtained. The target recognition results and current state (such as position, velocity, observed target information, etc.) are sent to other UAVs using wireless communication, while receiving data from other UAVs in the team. Through data fusion techniques (such as Kalman filtering, multi-sensor fusion, etc.), the observation values from each UAV are integrated, and after removing noise, the environmental observation values of the UAV itself are obtained .

[0055] Interception problem constraints:

[0056] numerical ratio constraint , denotes the maximum number ratio of UAVs, is a constant greater than 1, is the total number of interceptor UAVs, is the total number of target UAVs; if there exists such that , the target UAV is considered to be successfully intercepted, each interceptor UAV intercepts at most one target UAV, wherein represents the maximum number of interception steps, represents the current interception time step of the UAV, , , is a small constant, represents the maximum distance at which the UAV intercepts successfully, represents the coordinates of the interceptor UAV at the step, represents the coordinates of the target UAV at the step.

[0057] As an embodiment, five interceptor UAVs and five target UAVs are set in this embodiment. Each interceptor UAV is equipped with six depth cameras for obtaining target images, enhancing its environmental perception capability. The take-off area of the interceptor UAV is defined as , the target UAV generation area is , and the relevant area parameters are defined, such as , , . Among them represents a set of real numbers, represents the take-off area of the interceptor UAV, represents the generation area of the target UAV, represents the radius of the generation area of the interceptor UAV, represents the radius of the generation area of the target UAV, represents the generation height of the target UAV.

[0058] Parameter setting: The set parameters include target UAV speed, camera horizontal and vertical field of view, maximum camera observation distance, and other key parameters such as proportional factor of control force between UAVs, allowed target interception distance error, etc. These parameters are determined according to actual application scenarios and research needs, and are used to accurately simulate the UAV movement and interception process.

[0059] As an embodiment, the parameters are set as follows: target UAV speed , camera horizontal and vertical field of view , , maximum camera observation distance , proportional factor of control force between UAVs, proportional factor of adjusting repulsive force strength , proportional factor of adjusting attractive force strength , allowed target interception distance error .

[0060] The interception drone achieves three-dimensional precise positioning of the drone through image data processing methods such as image color space conversion and target extraction, center of mass position calculation and coordinate conversion.

[0061] Image color space conversion and target extraction: First, the image captured by the camera is converted to the HSV color space, and the threshold segmentation method is used to segment different areas of different drones to generate binary images.

[0062] The centroid position calculation and coordinate transformation realize the three-dimensional accurate positioning of the UAV: ​​After the morphological opening operation of the binary image, the formula 、 Calculate the target center of mass position , Indicates that the image is The first-order moment of the direction, that is, the cumulative sum of the horizontal coordinate of each pixel in the target area multiplied by the pixel value, Indicates that the image is The first-order moment of the direction, that is, the cumulative sum of the ordinate of each pixel in the target area multiplied by the pixel value, Represents the zero-order moment of the image, that is, the cumulative sum of all pixel values ​​in the target area, combined with depth information and camera intrinsic parameters (focal length 、 and the target center of mass position ), using the formula Convert to 3D coordinates in the camera coordinate system , and then through Convert to world coordinate system ,in represents the rotation matrix from the camera coordinate system to the world coordinate system, It is the translation vector, which represents the position of the origin of the world coordinate system in the camera coordinate system.

[0063] 2. Construct an improved multi-agent deep deterministic policy gradient algorithm (DS-MADDPG). The improved multi-agent deep deterministic policy gradient algorithm (DS-MADDPG) includes an independent replay buffer and a multi-dimensional convolutional policy network for target interception and target allocation.

[0064] The independent playback buffer establishes an independent experience buffer for each UAV agent, while constructing an experience buffer shared by all agents, and determines the sampling ratio of each experience buffer using a dynamic sampling network, and adjusts the sampling strategy according to the task dynamic change (for example, preferentially selecting experience tuples with high interception success reward). The multi-dimensional convolutional policy network is introduced into the policy network of the traditional multi-agent deep deterministic policy gradient (MADDPG), and different dimensional convolutional neural networks (CNNs) are used to capture the spatio-temporal relationship between multiple agents, while a multi-head attention mechanism is used to extract features from the captured relationship, enhance the perception ability of the agent to the environment, and improve the accuracy of task allocation and interception.

[0065] The construction process of the independent playback buffer and the multi-dimensional convolutional policy network is as follows:

[0066] 1) Code initialization: integrate the improved multi-agent deep deterministic policy gradient algorithm (DS-MADDPG) code into the simulation environment and complete the initialization work. Initialize the individual optimal buffer for each UAV agent and the shared experience buffer , initialize the main network: Actor network , Critic network , and sampling network , and randomly assign network weights. Assign the same initial weights to the target networks , , , of each network as the original network.

[0067] The architecture of Critic network and : each agent has two independent Critic networks, namely and , each independent Critic network consists of three fully connected layers, which are used to evaluate the value of each action taken by the agent, and the smaller value of the outputs , of the two independent Critic networks is selected as the final output value to reduce the risk of reward overestimation.

[0068] The architecture of the sampling network : consists of three MLP layers, which are the same as the network architecture of and , and is used to output itself according to , The output proportion dynamic adjustment reduces the volatility of the network training process, enhances the training stability, reduces the reward overestimation risk, and balances exploration and utilization from the sampling proportion of the shared experience buffer.

[0069] Actor network Receiving observation values of the agent The observation values are first extracted by two 1D convolution layers with different kernel sizes to obtain features of different scales. The output of the convolution layers is connected with the original observation values to form a new feature vector. Then, the TransformerEncoder layer is used to extract features by using the multi-head attention mechanism. After that, three fully connected multi-layer perceptron (MLP) layers are used for further feature extraction and nonlinear mapping. Finally, a softmax layer is used to output a 1xM action vector, representing the probability distribution of the agent selecting different actions in the current state.

[0070] Target network , , and have the same network structure as the corresponding main network ( , , , ). Through the delayed update mechanism, the target network ensures the stability of the training process and provides reliable policy and value evaluation for the agent.

[0071] 2) Hyperparameter setting: Determine the hyperparameters required for algorithm training, such as the maximum sampling proportion , the total number of training episodes , the training frequency , the soft update rate , the exploration noise , the discount factor , the maximum number of steps for the sampling network , the batch size , the size of the independent optimal buffer , the size of the shared experience buffer . These hyperparameters have a significant impact on algorithm performance and need to be reasonably selected through multiple trials and analysis.

[0072] 3) Reward function design and setting: Design a dense reward function with multiple dimensions to guide the UAV agent to learn effective interception strategies. The reward function is composed of the following parts:

[0073] Interception reward : When , the interception UAV successfully intercepts the target; when , the interception UAV ( ) time, represents the other interception UAVs successfully intercept the target except for the interception UAV . represents the position of the th interception UAV at time step . represents the maximum distance of interception UAVs interception success.

[0074] task completion reward : at the end of the total training episode, if the interception success rate ( ), then ; if , then ; otherwise , where is the threshold of the interception success rate to determine whether the task is successfully completed, and the interception success rate refers to the proportion of successfully intercepted target UAVs in the current episode.

[0075] task assignment reward : when the target UAV index set selected by all interception UAVs is equal to the current set of surviving target UAV indexes , then ; otherwise .

[0076] task maintenance reward : if the interception UAV maintains the same target at adjacent time steps, i.e. , then ; otherwise , represents the index of the target UAV assigned to the interception UAV at time .

[0077] ranking reward : , where , , and are calculated based on the height and distance ranking of the target UAV, respectively, to optimize target assignment. Wherein represents the height ranking of the target UAV assigned to the interception UAV . The distance ranking of the target UAV assigned to the interception UAV .

[0078] 4) Target assignment process: The multi-UAV interception problem is modeled as a Markov Decision Process, defining the state space, action space, observation space, state transition function and reward function of the system. The action Output based on reinforcement learning Probability distribution selects target UAV, , represents the agent selects the first target UAV with the highest probability, i.e. ; the observation contains the positions, velocities and target identification information of all UAVs, where represents the set of positions of all intercepting UAVs at time step . represents the set of positions of all target UAVs at time step . represents the set of index numbers of target UAVs currently responsible for interception by intercepting UAVs at time step . is the velocity vector of intercepting UAV at time step , represents the total number of intercepting UAVs, represents the total number of target UAVs. The reward is composed of an interception reward , a task completion reward (reward or punishment according to task completion and interception success rate), a task assignment reward (reward for reasonable target assignment), a ranking reward (encouraging selection of targets according to height and distance), and a task preservation reward (encouraging continuous tracking of the same target), guiding the agent's decision-making through rewards.

[0079] Target interception process: In the reinforcement learning framework, the Actor network outputs the interception target , and the tracking UAV calculates the acceleration according to this target and the positions of other UAVs, where is the repulsive acceleration between UAVs, when , , is a threshold value for determining whether two intercepting UAVs are too close, is a proportion factor for adjusting the strength of repulsive force; is the attractive acceleration of the target to the UAV, when , , A proportional factor for adjusting the strength of the attractive force, to achieve interception control of the target. representing the interception UAV the target UAV of the current task assignment the current position of the target UAV. representing the interception UAV the flight position at the current time.

[0080] Network training and optimization: throughout the simulation process, network training and updating is the key link to improve the performance of the DS-MADDPG algorithm and achieve efficient cooperative interception of UAVs. The specific operation is as follows:

[0081] 1) Data collection and storage: in each training current training step , each interception UAV selects the target UAV to perform interception operation according to the action probability distribution output by the actor network , and then the UAV observes the change of the environment state, obtains the next state , reward and new observation value . Then, when the new experience tuple is generated, wherein represents the set of observation values of the agent at the current time, represents the set of action values of the agent at the current time, represents the set of reward values of the agent at the current time, represents the set of observation values of the agent at the next time. These stored experience data will provide learning materials for subsequent network updating.

[0082] 2) Network updating, including:

[0083] Sampling operation: according to the DS-MADDPG algorithm process, during training, the data is sampled according to the current (current training step) and the specific requirements of the algorithm.

[0084] When ( is the maximum training step), randomly draw samples from the shared experience buffer D, input them into the target network , and the sampling network , determine the sampling proportion through the calculation of these networks, and the calculation method is: . Then, according to the proportion of , draw a small batch of samples from the shared experience buffer D and the independent optimal buffer . , and when When , it is directly from the shared experience buffer mid-sampling samples , represents a set of experience tuples, Represents another set of experience tuples. It is a function operation used to limit the value to a specific range. It is a sampling network The output value. and There are two different target networks 、 The output value.

[0085] Calculate target value: Calculate target value for different network update requirements. and Calculate the target value when When mixing samples from two buffers, ; when sampling only from the shared experience buffer, . In updating the sampling network Calculate the target value when The calculation of these target values ​​provides an important reference for the subsequent network weight update. It's an interceptor drone. The current reward value of is a discount factor used to adjust the impact of future rewards on current decisions, is the value function in the target network, which is based on the current observation and target strategies Calculated to evaluate the long-term reward that the agent may obtain at the next moment.

[0086] Each network weight update: Update the weight of each network by minimizing the loss function. and , the loss function is Indicates the number of samples (or trajectories, by adjusting their weights , which enables the critic network to evaluate the action value more accurately. , the loss function is , thereby renewing its rights , optimize the sampling ratio adjustment strategy. For Actor network , using policy gradient (when mixed sampling) or Update the weights (only shared experience buffer sampling time) , so that the actor network can output better action decisions. Among them represents the parameter of the agent , , that is, the gradient of the loss function for the agent , represents the number of samples (or the number of trajectories), that is, the number of samples used to calculate the gradient, is a function of the observation and action of the agent , is a function of the observation and action of the agent , represents the action of the agent , about the gradient of a certain objective function, represents the action of the agent , about the gradient of a certain objective function, represents the gradient of the agent's policy according to the agent's observation , about the parameter , given the agent's observation , represents the gradient of the agent's policy according to the agent's observation , about the parameter , represents the agent's policy function, which outputs the probability distribution of the agent performing a certain action given the observation , , represents the agent's policy function, which outputs the probability distribution of the agent performing a certain action given the observation . Target network soft update: In order to make the network training more stable, the target network is soft updated. The soft update formula is adopted

[0087] , , , , , where The soft update rate is. This soft update method enables the target network to gradually track the changes of the main network, avoiding unstable training caused by too fast network updates.

[0088] Network training indicator judgment: During the training process, continuously monitor training indicators such as average reward value, interception success rate, etc. These indicators are important basis for measuring algorithm training effect. When the training result reaches the preset training indicator, stop training. The setting of the preset indicator needs to consider the actual application demand and the algorithm performance expectation, for example, the average reward value reaching a certain level or the interception success rate exceeding a certain threshold can be set as the condition for completing training.

[0089] Simulation result evaluation and analysis, including:

[0090] Comparative experiment setting: To verify the performance of the DS-MADDPG algorithm, MADDPG, MAAC, MAPPO, MATD3 and other algorithms are selected as comparison. Under the same simulation environment and parameter settings, each algorithm is trained and tested to ensure consistency of experimental conditions.

[0091] Performance index evaluation: Test the trained strategy network of each algorithm in the ROS simulation environment, and perform 500 independent tests for each algorithm. For example, Figure 3 、 Figure 4 、 Figure 5 Record and analyze key performance indicators such as average interception rate, average interception distance and average interception time. By comparing the performance of different algorithms in these indicators, the superiority and effectiveness of the DS-MADDPG algorithm are evaluated.

[0092] Result analysis and summary: According to the simulation results, analyze the advantages and disadvantages of the DS-MADDPG algorithm in target allocation and interception effect. Research the adaptability and stability of the algorithm in different scenarios, and provide basis for further improving the algorithm and optimizing the unmanned aerial vehicle cooperative interception strategy.

[0093] The present application combines visual detection technology and deep reinforcement learning by processing images captured by the on-board camera of the unmanned aerial vehicle and through team internal communication, so that the unmanned aerial vehicle can track targets using visual information in complex dynamic environments. The algorithm is integrated into the Robot Operating System (ROS) simulation environment to verify the effectiveness of the method in actual tasks.

[0094] The preferred embodiments of the application disclosed above are only to facilitate the elucidation of the application. The preferred embodiments do not describe all the details and do not limit the application to the specific embodiments described. Obviously, many modifications and variations can be made in light of the teachings above. The description is chosen and described in order to provide the best illustration of the application principles and their practical application, to enable those skilled in the art to best understand and utilize the application.

Claims

1. A multi-agent UAV collaborative interception method based on dynamic sampling, characterized in that: The steps include: (1) The intercepting drone collects target images through the onboard camera and obtains its own environmental observation value through internal communication between the intercepting drone team and the environment. (2) The improved multi-agent deep deterministic policy gradient algorithm controls the intercepting drone by intercepting the target image captured by the drone and its own environmental observations, enabling the intercepting drone to track the target in a dynamic environment; The improved multi-agent deep deterministic policy gradient algorithm includes an independent replay buffer and a multi-dimensional convolutional policy network. Each drone agent in the independent replay buffer establishes an independent experience buffer, while simultaneously building an experience buffer shared by all agents. The sampling network is used to determine the sampling ratio of each experience buffer and adjust the sampling strategy according to the dynamic changes of the task. The independent playback buffer constructs an independent individual optimal buffer for each agent while retaining the shared experience buffer; when a new experience tuple When generating, if the old experience tuple in the individual optimal buffer is empty and the interception drone obtained based on the current interception effect, target allocation and cooperation strategy evaluation If the current reward value is greater than zero, the new experience tuple is stored in the individual optimal buffer; if the old experience tuple is not empty, when the drone is intercepted The reward is greater than the old experience tuple intercepting drones When the reward is , the old experience tuple is replaced by the new experience tuple, otherwise the old experience tuple is retained; represents the set of observation values ​​of the agent at the current moment, represents the set of action values ​​of the agent at the current moment, Represents the set of reward values ​​of the agent at the current moment, Represents the set of observations of the agent at the next moment; each agent uses a sampling network to dynamically adjust the ratio of experience sampled from the individual optimal buffer and the shared experience buffer according to the current training progress and number of iterations. The sampling ratio is calculated based on the output results of the sampling network and the two target evaluation networks based on the experience sampled from the shared experience buffer, and the ratio of private experience sampling is limited by setting a maximum sampling ratio; The multi-dimensional convolutional strategy network includes an Actor network, a Critic neural network, and a sampling network; The Actor network receives the agent's observations and first extracts features of different scales through two 1D convolutional layers with different kernel sizes. The convolutional layer output is concatenated with the observations to form a new feature vector. The new feature vector is then extracted using a multi-head attention mechanism in the encoding layer. Further feature extraction and nonlinear mapping are performed through three fully connected multi-layer perceptron layers. Finally, the softmax layer outputs an action vector, which represents the probability distribution of different actions selected by the agent in the current state. The critic neural network includes two independent critic networks for each agent. Both independent critic networks include three fully connected layers, which are used to evaluate the value of each action taken by the agent. A double Q network structure is used to select the smaller value of the output of the two independent critic networks as the final value; the sampling network includes three fully connected layers, which is used to dynamically sample possible targets or actions based on the current observation value.

2. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 1 is characterized in that: The intercepting drone obtains information about other drones in its intercepting drone team through image color space conversion and target extraction, center of mass position calculation and coordinate conversion, and uses wireless communication to send the target extraction results and the intercepting drone's own position information, speed information and image feature information obtained by the camera to other drones. At the same time, it receives the observation values ​​of other drones in the intercepting drone team and the intercepting drone's own position information, speed information and image feature information obtained by the camera. Through data fusion, it integrates the observation values ​​from each drone in the team and obtains its own environmental observation value after removing noise.

3. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 1 is characterized in that: The interception constraints for intercepting drones include: numerical ratio constraints , Indicates the maximum numerical ratio of the drone, is the total number of intercepted drones, is the total number of target drones; Each intercepting drone can intercept at most one target drone, if there is one , making , then the target drone is considered a successful interception, where represents the maximum number of intercept steps, It represents the time step of the current interception of the UAV. Indicates the maximum distance at which a drone can be successfully intercepted. Indicates interception of drones In the The coordinates of the step, Indicates target drone In the The coordinates of the step.

4. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 1 is characterized in that: The target allocation and interception of the multi-dimensional convolutional policy network are based on Markov decision-making to determine the actions, observation values ​​and reward functions of each agent. Each agent selects the interception target based on the probability distribution output by the Actor network, obtains environment and task information based on the observation value to make a decision, and calculates the acceleration of the intercepting drone based on the interception target output by the Actor network and the position information of other drones to achieve target interception.

5. The multi-agent UAV cooperative interception method based on dynamic sampling according to claim 4 is characterized in that: The reward function is : ； in Indicates interception of drones At time step The interception reward, Indicates the reward for completing the task. represents the task assignment reward, Indicates interception of drones At time step Ranking rewards, Indicates interception of drones At time step The tasks remain rewarding.

6. A system for implementing the multi-agent UAV collaborative interception method based on dynamic sampling as described in claim 1, characterized in that: include: Reinforcement learning module and drone motion system; The reinforcement learning module includes a reinforcement learning model, which is a model of an improved multi-agent deep deterministic policy gradient algorithm built on a Markov model based on actions, observations, and reward functions; The drone motion system includes the motion model of the target drone and the motion model of the intercepting drone. The intercepting drone is equipped with a camera for collecting target images. By processing the images captured by the drone's onboard camera and relying on internal team communication, it combines visual detection technology with deep reinforcement learning, enabling the drone to use visual information to track targets in complex dynamic environments.

7. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory. When the computer program is executed on the processor, a multi-agent UAV collaborative interception method based on dynamic sampling as described in any one of claims 1 to 5 is implemented.

Images