A multi-target tracking obstacle avoidance strategy network optimization method and device for a drone cluster

Abstract

The application discloses a kind of multi-target tracking obstacle avoidance strategy network optimization method and device of unmanned aerial vehicle cluster, method includes obtaining tracking obstacle avoidance network obtained by simulation training environment training;Each tracking obstacle avoidance network is respectively loaded to unmanned aerial vehicle cluster in each unmanned aerial vehicle respectively corresponding agent, and each agent respectively corresponding action strategy network is initialized;Real-time observation information of current environment is obtained by each agent;According to observation information and experience replay mechanism, the weight parameters of each action strategy network are shared and updated, to obtain the multi-target tracking obstacle avoidance strategy network of each agent respectively corresponding, so that unmanned aerial vehicle cluster can rely on the strategy network completed by training, when actual multi-target tracking task is carried out, quickly respond to dynamic environment change and the obstacle avoidance of barrier, realize individual action decision and the coordination and unity of cluster overall task target.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a network optimization method and apparatus for multi-target tracking and obstacle avoidance strategies in UAV swarms. Background Technology

[0002] With advancements in artificial intelligence, communication, and sensor technologies, drones have expanded from aerial photography for entertainment to operational fields such as agricultural plant protection, logistics delivery, and disaster relief. Compared to single drones, swarms offer advantages such as wider detection range, higher operational efficiency, and stronger fault tolerance. However, their target search and tracking tasks are naturally coupled with obstacle avoidance. Tracking requires the drone to approach the target and maintain an observation distance, while obstacle avoidance forces the drone to maintain a safe distance from obstacles. Especially in areas with dense obstacles, the optimal tracking pose may overlap with the threat area, forcing the system to frequently make trade-offs between "tracking" and "avoiding." Furthermore, historical trajectories can easily cause obstacle avoidance interruptions, disrupting the continuity of tracking.

[0003] To address this problem, existing technologies mainly fall into two categories: one is collaborative control methods based on mechanistic modeling, such as control barrier functions, model predictive control, and barrier Lyapunov functions, which design control laws based on dynamic models and constraints; the other is multi-agent methods based on deep reinforcement learning, which treat each UAV as an agent in a partially observable Markov decision process, learning end-to-end policies through trial and error to bypass the dependence on precise modeling. Both types of methods attempt to balance tracking and obstacle avoidance by optimizing the target or reward design.

[0004] However, regardless of the cost function weighting in mechanistic modeling or the reward weighting in reinforcement learning, the relative weights of tracking and obstacle avoidance are preset to constants. In actual flight, the target motion, obstacle density and relative speed change dynamically. Fixed weights are not only difficult to adjust to the global optimum, but also cannot automatically adjust the emphasis according to the current situation. As a result, when the UAV switches between scenarios such as target acceleration or sudden increase in obstacles, it is difficult to balance the performance of tracking and obstacle avoidance. Summary of the Invention

[0005] This invention provides a method and apparatus for optimizing the multi-target tracking and obstacle avoidance strategy network of a drone swarm. It solves the technical problem that, regardless of whether it is cost function weighting in mechanism modeling or reward weighting in reinforcement learning, the relative weights of tracking and obstacle avoidance are preset to constants. However, in actual flight, the target motion, obstacle density, and relative speed change dynamically. Fixed weights are not only difficult to adjust to the global optimum, but also cannot automatically adjust the emphasis according to the current situation. This makes it difficult for drones to balance the performance of tracking and obstacle avoidance when switching between scenarios such as target acceleration or sudden increase in obstacles.

[0006] This invention provides a network optimization method for multi-target tracking and obstacle avoidance strategies in unmanned aerial vehicle (UAV) swarms, comprising: Obtain the tracking and obstacle avoidance network trained in a simulation training environment; Each of the aforementioned tracking and obstacle avoidance networks is loaded into the corresponding intelligent agent of each drone in the drone cluster, and the action policy network corresponding to each of the aforementioned intelligent agents is initialized. Each of the aforementioned intelligent agents acquires real-time observation information about the current environment; Based on the observation information and experience playback mechanism, the weight parameters of each action policy network are shared and updated to obtain the multi-target tracking and obstacle avoidance policy network corresponding to each agent.

[0007] Optionally, the step of obtaining the tracking and obstacle avoidance network trained in a simulation training environment includes: Acquire hardware data of each drone in the drone swarm, first motion data of the target to be tracked, location data of the task area, and second motion data of obstacles; Based on the hardware data, the first motion data, the position data, and the second motion data, a simulation training environment corresponding to the UAV cluster is created; the simulation training environment includes a task constraint model, a UAV motion model, a target motion feature model, and an obstacle feature model; Based on the UAV motion model, a training agent corresponding to the UAV is constructed, and the initial search and tracking experience network and the initial obstacle avoidance experience network in the training agent are initialized. The training agent interacts with the simulation training environment, and the training agent is trained based on the interaction results until the training number is met, so as to obtain the search and tracking experience network and the obstacle avoidance experience network. The search and tracking experience network and the obstacle avoidance experience network are used as the tracking and obstacle avoidance network.

[0008] Optionally, the initial search and tracking experience network includes an initial search and tracking policy network and an initial search and tracking evaluation network, and the initial obstacle avoidance experience network includes an initial obstacle avoidance policy network and an initial obstacle avoidance evaluation network; the step of interacting with the training agent and the simulation training environment, and training the training agent based on the interaction results until the training iterations are satisfied, to obtain the search and tracking experience network and the obstacle avoidance experience network, includes: Initialize the positions of each training agent, the target to be tracked, the obstacles, and the total number of training rounds within the simulation training environment; The training agent acquires training observation information from the simulated training environment. The training agent outputs a search and tracking action based on the training observation information and the initial search and tracking strategy network, and evaluates the search and tracking action based on the initial search and tracking evaluation network to obtain a first evaluation result corresponding to the search and tracking action. Based on the first evaluation result and the first reward corresponding to the search and tracking action, update the parameters of the initial search and tracking strategy network and the initial search and tracking evaluation network until the total number of training rounds is reached, and obtain the search and tracking experience network; The trained agent outputs obstacle avoidance actions based on the training observation information and the initial obstacle avoidance strategy network, and evaluates the obstacle avoidance actions based on the initial obstacle avoidance evaluation network to obtain a second evaluation result corresponding to the obstacle avoidance actions. Based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, update the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network until the total number of training rounds is reached, thus obtaining the obstacle avoidance experience network.

[0009] Optionally, the step of updating the parameters of the initial search tracking strategy network and the initial search tracking evaluation network based on the first evaluation result and the first reward corresponding to the search tracking action, until the total number of training rounds is reached, to obtain the search tracking experience network, includes: Based on the training observation information, the search and tracking action, the first reward, and the training observation information at the next moment, a first experience tuple is constructed and stored in the experience replay buffer; The second experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The first policy gradient and the first temporal difference error are calculated based on the second empirical tuple; The parameters of the initial search and tracking policy network are updated according to the first policy gradient, and the parameters of the initial search and tracking evaluation network are updated according to the first temporal difference error, until the total number of training rounds is reached, and the initial search and tracking evaluation network at the current time is determined as the search and tracking experience network.

[0010] Optionally, the step of updating the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, until the total number of training rounds is reached, to obtain the obstacle avoidance experience network, includes: The training observation information, the obstacle avoidance action, the second reward, and the training observation information at the next moment are stored as a third experience tuple in the experience replay buffer. The fourth experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The second policy gradient and the second temporal difference error are calculated based on the fourth empirical tuple; The parameters of the initial obstacle avoidance policy network are updated according to the second policy gradient, and the parameters of the initial obstacle avoidance evaluation network are updated according to the second temporal difference error, until the total number of training rounds is reached, and the initial obstacle avoidance evaluation network at the current moment is determined as the obstacle avoidance experience network.

[0011] Optionally, the step of sharing and updating the weight parameters of each action policy network based on the observation information and experience replay mechanism to obtain the multi-target tracking obstacle avoidance policy network corresponding to each agent includes: The agent determines the action to be performed based on the action policy network and the observation information, respectively. Calculate the reward information corresponding to the executed action, and combine the observation information and the executed action to construct the experience tuple corresponding to the agent, and cache it in the corresponding experience replay cache area; Dynamic weights are determined based on the tracking and obstacle avoidance network and the evaluation network of the agent; Based on the experience replay mechanism, target experience tuples are sampled from the experience replay buffer, and combined with the dynamic weights, the first parameter of the action policy network and the second parameter of the evaluation network are updated. When the termination condition is met, the action policy network at the current moment is determined as the multi-target tracking and obstacle avoidance policy network of each agent.

[0012] Optionally, the evaluation network includes an agent tracking evaluation network and an agent obstacle avoidance evaluation network; the step of determining dynamic weights based on the tracking and obstacle avoidance network and the agent's evaluation network includes: Based on the tracking experience network and obstacle avoidance experience network in the tracking and obstacle avoidance network, calculate the first tracking evaluation value and the first obstacle avoidance evaluation value corresponding to the observation information respectively. Using the agent's agent tracking evaluation network and agent obstacle avoidance evaluation network, the second tracking evaluation value and the second obstacle avoidance evaluation value corresponding to the observation information are calculated respectively. Based on the first tracking evaluation value and the second tracking evaluation value, calculate the tracking correction information; Based on the first obstacle avoidance assessment value and the second obstacle avoidance assessment value, calculate obstacle avoidance correction information; Based on the tracking correction information and the obstacle avoidance correction information, determine the dynamic weights; The calculation process for the dynamic weights is as follows: ; in, The dynamic weights are the observation information at time t. To track and correct information, For obstacle avoidance correction information, e is a natural constant.

[0013] Optionally, the step of sampling target experience tuples from the experience replay buffer based on the experience replay mechanism and updating the first parameters of the action policy network and the second parameters of the evaluation network in combination with the dynamic weights includes: Based on the experience replay mechanism, the experience replay buffer is sampled in batches to obtain the target experience tuple; Based on the dynamic weights and the target experience tuples, the policy loss function of the action policy network is calculated; Based on the policy loss function, the first parameters of the action policy network and the second parameters of the evaluation network are updated using gradient descent. The calculation process of the policy loss function is as follows: ; ; ; in, Let be the policy loss function of the action policy network at time t. For the action policy network at time t, observe the information Select the action to perform. The strategy probability, Let be the target search and tracking advantage function at time t. Let be the obstacle avoidance advantage function at time t, where Temp is the temperature coefficient and γ is the discount factor. For time t+1, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t+1, the evaluation network of the agent's obstacle avoidance is evaluated based on the observed information. The evaluation value, For time t, the evaluation network of the intelligent agent's obstacle avoidance is based on the observed information. The evaluation value.

[0014] Optionally, the step of updating the first parameters of the action policy network and the second parameters of the evaluation network based on the policy loss function using gradient descent includes: ; ; ; in, This is the second parameter of the agent tracking evaluation network. This is the second parameter of the intelligent agent obstacle avoidance evaluation network. The first parameter of the action policy network, The learning rate of the network is used to evaluate the tracking performance of the agent. The learning rate of the network is used to evaluate the obstacle avoidance of the intelligent agent. The learning rate of the action policy network. These are the initial parameters for the action policy network.

[0015] The present invention also provides a network optimization device for multi-target tracking and obstacle avoidance strategies in unmanned aerial vehicle (UAV) swarms, comprising: The tracking and obstacle avoidance network acquisition module is used to acquire the tracking and obstacle avoidance network trained through a simulation training environment. The agent initialization module is used to load each of the tracking and obstacle avoidance networks into the agent corresponding to each UAV in the UAV cluster, and to initialize the action policy network corresponding to each agent. The observation information acquisition module is used to acquire observation information of the current environment in real time through each of the intelligent agents; The network optimization module is used to share and update the weight parameters of each action strategy network according to the observation information and experience playback mechanism, so as to obtain the multi-target tracking and obstacle avoidance strategy network corresponding to each agent.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention obtains a tracking and obstacle avoidance network trained in a simulation training environment; loads each tracking and obstacle avoidance network into the corresponding agent of each UAV in the UAV swarm, and initializes the action policy network corresponding to each agent; obtains observation information of the current environment in real time through each agent; and updates the weight parameters of each action policy network by sharing according to the observation information and experience playback mechanism, thereby obtaining the multi-target tracking and obstacle avoidance policy network corresponding to each agent. This enables the UAV swarm to rely on the trained policy network to quickly respond to dynamic environmental changes and obstacle position changes when actually carrying out multi-target tracking tasks, thereby achieving the coordinated unity of individual action decisions and the overall task objectives of the swarm. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1A flowchart illustrating the steps of a network optimization method for multi-target tracking and obstacle avoidance strategy in an unmanned aerial vehicle (UAV) swarm, as provided in this embodiment of the invention. Figure 2 A schematic diagram of a simulation training environment provided in an embodiment of the present invention; Figure 3 A network model architecture diagram of an action policy network provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the process of obtaining the collaborative multi-target search, tracking, and obstacle avoidance strategy for UAV swarms in an embodiment of the present invention. Figure 5 This is a structural block diagram of a network optimization device for multi-target tracking and obstacle avoidance strategy of unmanned aerial vehicle (UAV) swarms, provided in an embodiment of the present invention. Detailed Implementation

[0019] With the development of technologies such as artificial intelligence, communication, and sensors, drones have gradually transformed from early aerial photography and entertainment tools into operational equipment for multiple industries. Compared to single drones, drone swarms have advantages such as wide detection range, high operational efficiency, greater fault tolerance, and strong spatial coverage. Currently, in the military field, swarm technology can perform tasks such as coordinated reconnaissance, saturation attacks, and electronic warfare, significantly improving combat effectiveness and survivability. In the civilian field, swarm technology can be widely applied in scenarios such as coordinated search in disaster relief, large-area precision spraying in agricultural plant protection, convoy delivery in logistics transportation, and infrastructure inspection. Compared to single drones, drone swarms face multiple constraints, mainly including environmental constraints, target constraints, and constraints between drones. The target search, tracking, and obstacle avoidance tasks of drone swarms have coupled and asynchronous characteristics. On the one hand, target search and tracking requires drones to actively approach the target and maintain a relative distance to continuously acquire observation information, with the optimization goal of minimizing the relative distance to the target. On the other hand, obstacle avoidance requires drones to maintain a safe distance from static and dynamic obstacles in the environment, and in special circumstances, emergency obstacle avoidance is required, with the optimization direction being to maximize the distance between the drone and the obstacle. When a target navigates through areas with dense obstacles, the optimal tracking position may happen to be located within the obstacle threat zone. This requires the drone to make real-time decisions between "target search and tracking" and "moving away from obstacles." Furthermore, search and tracking based on historical trajectories may guide the drone closer to obstacle areas, triggering emergency obstacle avoidance and interrupting the target search and tracking process.

[0020] Unmanned aerial vehicle (UAV) swarms demonstrate strong engineering applicability in multi-target search, tracking, and obstacle avoidance. Individual UAVs suffer from limitations such as limited field of view, large blind spots, poor search and tracking stability, and limited detection range of single sensors. By employing multi-platform sensor multi-view fusion, distributed collaborative perception, and obstacle avoidance algorithms, a comprehensive, three-dimensional aerial perception network is constructed. In multi-target search and tracking, UAVs within the swarm can acquire target information from multiple angles, including elevation, distance, and pitch. Through distributed data fusion and collaborative correlation filtering, long-term, uninterrupted, and stable tracking is achieved. For obstacle avoidance, the swarm can simultaneously acquire more perception information, including obstacles such as buildings, towers, and mobile radar vehicles. Using a distributed obstacle avoidance decision algorithm, it can autonomously adjust its trajectory to avoid collisions with obstacles while ensuring continuous execution of multi-target tracking tasks.

[0021] To address this issue, this invention provides a method and apparatus for optimizing a multi-target tracking and obstacle avoidance strategy network for UAV swarms. This method solves the problem that, regardless of whether it is cost function weighting in mechanistic modeling or reward weighting in reinforcement learning, the relative weights of tracking and obstacle avoidance are preset to constants. However, in actual flight, target motion, obstacle density, and relative speed change dynamically. Fixed weights are difficult to adjust to the global optimum and cannot automatically adjust the emphasis according to the current situation. This results in the UAV struggling to balance tracking and obstacle avoidance performance when switching between scenarios such as target acceleration or sudden increase in obstacles.

[0022] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a network optimization method for multi-target tracking and obstacle avoidance strategy in a drone swarm, as provided in this embodiment of the invention.

[0024] This invention provides a network optimization method for multi-target tracking and obstacle avoidance strategies in unmanned aerial vehicle (UAV) swarms, comprising: Step 101: Obtain the tracking and obstacle avoidance network trained in the simulation training environment; An obstacle avoidance tracking network refers to a network model pre-trained in a simulation training environment to guide a swarm of drones in collaboratively completing an integrated task of target search and tracking and autonomous obstacle avoidance. This network includes at least a search and tracking experience network and an obstacle avoidance experience network. The simulation training environment is a virtual simulation space or digital simulation platform constructed from a task constraint model, a drone motion model, a target motion feature model, and an obstacle feature model. It simulates the interaction between the drone swarm and the target and obstacles to be tracked, providing the training agent with near-realistic task scenarios and interactive feedback, enabling the policy network to iteratively optimize under safe and controllable conditions.

[0025] In this embodiment, a simulation training environment is created by acquiring hardware data of each UAV within the UAV swarm, first motion data of the target to be tracked, position data of the task area, and second motion data of obstacles. Simultaneously, the UAVs are abstracted as training agents, and interactive training is performed on these agents based on this simulation training environment. Model training is then conducted based on the observation information of the training agents within the simulation training environment to obtain a trained tracking and obstacle avoidance network. This allows for efficient learning and convergence to an optimal strategy that balances tracking continuity and obstacle avoidance safety without the risk of real flight.

[0026] It should be noted that, in terms of hardware computing platform, the multi-target tracking obstacle avoidance strategy network optimization method provided in this embodiment preferably uses a high-performance computing device equipped with high-speed, large-capacity memory and a multi-core processor. For example, a workstation configured with an NVIDIA RTX 3090 or higher graphics processing unit (GPU) can be used. Sufficient computing power and video memory resources can ensure the efficiency of the model training and inference phases, shortening the algorithm's computation time. Regarding the software runtime environment, this solution is based on mainstream deep learning frameworks such as PyTorch and TensorFlow for algorithm development and implementation. These frameworks integrate complete application programming interfaces (APIs) and underlying performance optimization toolsets, which can efficiently support network model building, algorithm debugging, and effect verification through multiple sets of control experiments. A simulation environment is developed based on ROS2 (Robot Operating System 2), and the UAV is equipped with cameras and LiDAR sensing devices.

[0027] In practical implementation, the problem of multi-target search, tracking, and obstacle avoidance in a swarm of UAVs with bounded perception range can be described as follows: The mission area includes a swarm of drones. Target set and dynamic obstacle set drone swarm Include A drone is deployed, and the drone's target detection range radius is [missing information]. The detection range radius for obstacles is The drone swarm system is unaware of the number of targets, their movement patterns, and the distribution and movement patterns of dynamic obstacles within the mission environment. For safety, emergency obstacle avoidance actions take precedence over other strategic actions in the drone's flight control system. The drone makes action decisions based on its own information, information about friendly drones, target information within its perception range, obstacle information within its detection range, and its flight control strategy, aiming to perceive as many targets as possible while minimizing the frequency of emergency obstacle avoidance actions.

[0028] In this embodiment of the invention, the size of the task environment area is set to , These are the boundaries in the X, Y, and Z directions, respectively. Obstacles. Equivalent to a cylinder , and This represents the coordinates of the center of the base of the cylinder. Let and represent the height and radius of the cylinder, respectively. Its motion model is as follows: ; in, The center of the circle representing the obstacle's movement. Indicates the radius of the obstacle's movement. This indicates the angular velocity of the obstacle.

[0029] Target The movement pattern follows the following rules: (1) When encountering obstacles or area boundaries, mirror reflection will be used to change the direction angle of movement. (2) Maintain motion state when there are no obstacles: If there are no obstacles or area boundaries in the target's path, then the current motion state remains unchanged; (3) Randomly update the motion direction after the number of iterations reaches the target: When the number of motion iterations reaches a preset specific value, a new motion direction is randomly selected for the target regardless of whether there is an obstacle in front.

[0030] drones The movement pattern is described as follows: ; ; ; in, For the first drone At any moment The three-dimensional spatial coordinates, The unit step length for a single flight of a drone. This is a sign function; if the input is greater than 0, it outputs 1; if the input is less than 0, it outputs -1; and if the input is equal to 0, it outputs 0. and drones exist Positive and negative displacement control commands on the axis; and drones exist Positive and negative displacement control commands on the axis; and drones exist Positive and negative displacement control commands on the axis; a and b The drone was given control commands for both positive and negative displacement. The height of the drone relative to the horizontal plane. and These are the minimum and maximum ground clearances that the drone is allowed to fly relative to the horizontal plane.

[0031] drones Perception range of the target It can be described as: ; in, This represents the maximum angle at which the UAV can detect ground targets.

[0032] Obstacles relative to drones Emergency obstacle avoidance will occur when the following conditions are met: ; The problem of cooperative target search, tracking, and obstacle avoidance in UAV swarms can be described as a multi-index optimization problem: ; ; ; ; ; st ; ; ; ; in, For target tracking flag function, This is a flag function for emergency obstacle avoidance. Representative target At any moment The position at that time Obstacles With drones The horizontal distance between them. In the model constraints of the above problem, the first and second terms represent restrictions on the movement range of the target, while the third and fourth terms represent restrictions on the movement range of the obstacle. For the goal To drone swarm at time distance, To track the total revenue of the target search, The total cost or number of emergency obstacle avoidance attempts is represented by k, where k represents the total number of obstacles and m represents the total number of targets.

[0033] In one example of this application, step 101 may include the following sub-steps S11-S15: S11. Acquire the hardware data of each drone in the drone cluster, the first motion data of the target to be tracked, the position data of the task area, and the second motion data of the obstacles; S12. Based on hardware data, first motion data, position data, and second motion data, create a simulation training environment corresponding to the UAV cluster; the simulation training environment includes a task constraint model, a UAV motion model, a target motion feature model, and an obstacle feature model. S13. Construct a training agent corresponding to the UAV based on the UAV motion model, and initialize the initial search and tracking experience network and the initial obstacle avoidance experience network in the training agent. S14. By interacting with the simulation training environment, the training agent is trained based on the interaction results until the number of training times is met, and the search and tracking experience network and obstacle avoidance experience network are obtained. S15. A search tracking experience network and an obstacle avoidance experience network are used as the tracking and obstacle avoidance network.

[0034] Hardware data refers to data related to the UAV platform's own flight capabilities, sensing device configuration, kinematic constraints, etc., which are used to limit the UAV's motion and detection boundaries in the simulation.

[0035] The first motion data refers to the state information of the target under tracking, such as its position, velocity, heading, and motion pattern, which changes over time within the mission area. This information is used to construct a target motion characteristic model so that the simulation environment can accurately reflect the dynamic behavior of the target.

[0036] The second type of motion data refers to the dynamic changes in the position, speed, trajectory, and dimensions of obstacles within the task area. This data is used to construct obstacle feature models, enabling the simulation environment to accurately simulate the motion patterns and collision risks of obstacles.

[0037] Location data refers to spatial information such as the scope, boundary conditions, and key landmarks of the mission area, which is used to construct a mission constraint model to limit the feasible flight area and spatial boundary conditions of the drone swarm when performing the mission.

[0038] In this step, after acquiring the hardware data of each drone in the drone swarm, the first motion data of the target to be tracked, the location data of the task area, and the second motion data of obstacles, a visual simulation environment for collaborative multi-target search, tracking, and obstacle avoidance of the drone swarm can be developed using Python. This involves defining the task constraint model, the drone motion model, the target motion feature model, and the obstacle feature model. Specifically, the constructed simulation training environment is as follows: Figure 2 As shown, the task constraint model is configured with parameters, for example, defining the task environment as a three-dimensional space with dimensions of 5000×5000×1000. The obstacle feature model includes the obstacle's motion pattern and emergency obstacle avoidance conditions. Obstacles are represented as cylinders; when the horizontal distance between the obstacle and the center of the cylinder is less than the cylinder's radius, the UAV will perform an emergency obstacle avoidance maneuver. The target motion feature model includes the target's motion rules and the tracked mode. When the target is at time... When a target is within the perception range of any drone in the swarm, it is considered to have been tracked and observed by the swarm at that moment. The drone motion model includes a perception model, action space, and policy priorities. The drone's perception information includes its own information, friendly drone information, target information within its perception range, and obstacle information within its detection range. Local information It can be represented as: ; drones Compared to friendly machines Information It can be described as: ; in, and They represent the times respectively. drones Relative to friendly machines within effective communication range exist and Relative position on the axis Indicates at time drones Heyou Machine The distance between them.

[0039] drones Regarding targets within the perception range Observational information for: ; in, and They represent the times respectively. drones Relative to the target exist and Relative position on the axis Indicates drone and target At any moment The distance between time, and They represent the times respectively. Time target In and The speed on.

[0040] drones Regarding obstacles within the detection range Observational information for: ; in, and They represent the times respectively. drones Relative to obstacles exist and Relative position on the axis Indicates drone and obstacles At any moment The distance between time, and They represent the times respectively. Time Obstacles In and The speed on.

[0041] In addition, the drone model also includes a reward function for drone swarms, which consists of two parts: one part is the target search and tracking reward. One part is drone obstacle avoidance rewards. .

[0042] For drones Its target search tracking reward is This portion of the reward is used to guide the drone to fly towards the target area within the mission area, and it includes environmental rewards. Friendly aircraft rewards and target search rewards .

[0043] Environmental awards Used to estimate the flight of the UAV within the mission area while satisfying altitude constraints: ; Friendly machine rewards Used to increase the probability of target detection by expanding the sensing range of drone swarms: ; ; in, For drones With friendly machines Related collaborative perception rewards.

[0044] Target search rewards The reward is used to estimate that the drone adjusts its position in time, follows the target, and moves in the direction of the target; ; ; ; in, For drones With the target T to be tracked k Related search tracking rewards.

[0045] For drones The obstacle avoidance reward is This portion of the reward is used to guide the drone in obstacle avoidance flight within the mission area. It is further divided into environmental rewards. Obstacle avoidance rewards The reward is based on the assumption that the drone flies within the mission area while meeting altitude constraints; ; Obstacle Avoidance Rewards The reward is used to guide the drone away from obstacles: ; ; ; After constructing and integrating the task constraint model, UAV, UAV motion model, target motion feature model, and obstacle feature model into a simulation training environment, a training agent is built based on the UAV motion model. An initial search-tracking experience network and an initial obstacle avoidance experience network are initialized for the training agent. The training agent interacts with the simulation training environment, selecting actions, rewards, and next-moment states based on observation information. The interaction results are used to update the parameters of the initial search-tracking experience network and the initial obstacle avoidance experience network. When the training iterations are reached or the corresponding stopping conditions are met, the search-tracking experience network and the obstacle avoidance experience network are obtained and used as the tracking and obstacle avoidance network for subsequent loading by the UAV swarm. For example, the search-tracking experience network learns to approach and continuously observe the target, while the obstacle avoidance experience network learns to move away from obstacles. This provides a priori networks for collaborative multi-target search, tracking, and obstacle avoidance, reducing the difficulty of exploring strategies from scratch and improving training efficiency and task adaptability.

[0046] Optionally, the initial search and tracking experience network includes an initial search and tracking strategy network and an initial search and tracking evaluation network, and the initial obstacle avoidance experience network includes an initial obstacle avoidance strategy network and an initial obstacle avoidance evaluation network; S14 may include the following sub-steps S141-S146: S141. Initialize the positions of each training agent, the target to be tracked, the obstacles, and the total number of training rounds in the simulation training environment. S142. Train the intelligent agent to obtain training observation information from the simulation training environment; S143. The trained agent outputs search and track actions based on the training observation information and the initial search and track strategy network, and evaluates the search and track actions based on the initial search and track evaluation network to obtain the first evaluation result corresponding to the search and track actions. S144. Based on the first evaluation result and the first reward corresponding to the search and tracking action, update the parameters of the initial search and tracking strategy network and the initial search and tracking evaluation network until the total number of training rounds is reached, and obtain the search and tracking experience network. Optionally, S144 may include the following sub-steps: Based on training observation information, search and tracking actions, the first reward, and the training observation information of the next time step, a first experience tuple is constructed and stored in the experience replay buffer; The second experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The first policy gradient and the first temporal difference error are calculated based on the second empirical tuple; The parameters of the initial search and tracking policy network are updated according to the first policy gradient, and the parameters of the initial search and tracking evaluation network are updated according to the first temporal difference error, until the total number of training rounds is reached. The initial search and tracking evaluation network at the current time is then determined as the search and tracking experience network.

[0047] The total number of training rounds refers to the pre-set upper limit of training rounds, used to determine whether the training of the search tracking network has reached the stopping condition.

[0048] Training observation information refers to the current state representation obtained by the training agent in the simulation training model, which is used as input to the initial search and tracking policy network and the initial search and tracking evaluation network.

[0049] The initial search and tracking strategy network refers to a network that has not yet been trained and is used to output search and tracking actions based on training observation information. It can be represented as an actor network (Actor).

[0050] The initial search tracking evaluation network refers to a network that has not yet been trained and is used to evaluate the value of search tracking actions. It can be represented as a critic network (Critic).

[0051] An experience tuple refers to an experience record consisting of observation information, search and tracking actions, rewards and next-moment training observation information, transition probabilities, and discount factors, including the first experience tuple, the second experience tuple, etc.

[0052] In this embodiment, each drone in the drone swarm is abstracted as a training agent. Each training agent is trained using the dominant actor-critic algorithm with a fusion-priority experience replay mechanism. This training abstraction is a quintuple. Markov decision process, in which The state space for training the agent. This represents the action space of the training agent. It is the reward obtained by training the intelligent agent. It is the state transition probability. This represents the discount factor. The training agent acquires training observation information from the simulated training environment, such as the local state, relative positions of friendly aircraft, relative positions and velocities of targets, and relative positions and velocities of obstacles. This training observation information is encoded and fed into the initial search and tracking policy network and the initial search and tracking evaluation network, providing input for action selection and action evaluation. This ensures that each round of training is conducted under controllable initial conditions, guarantees the repeatability of the training process, and reduces the interference of random initial states on the learning of the search and tracking policy.

[0053] After initialization and input of training observation information, the training agent calls the initial search and tracking policy network to output search and tracking actions based on the training observation information. The initial search and tracking evaluation network evaluates the actions to obtain the first evaluation result. Based on the above-mentioned calculation formula for target search and tracking reward, the first reward is calculated, and the training observation information for the next time step is obtained.

[0054] Based on training observation information, search and tracking actions, the first reward, and the training observation information at the next time step, a first experience tuple is constructed and stored in the experience replay buffer. Specifically, this can be achieved through the aforementioned five-tuple. The first experience tuple is constructed and cached using a specific method, and the cumulative reward of the trained agent is calculated and updated. Based on the priority experience replay method, batch samples are sampled from the agent's experience replay buffer, the policy gradient and temporal difference error are calculated, and gradient descent is used to refine the policy network. parameters Update the evaluation network based on the temporal difference error. parameters Determine whether the maximum number of steps for each training round has been reached. If so, further determine whether the total number of training rounds has been reached. If so, then determine the initial search-tracking evaluation network at the current moment as the search-tracking experience network.

[0055] If the maximum number of training rounds has not been reached, proceed to step S142. If the total number of training rounds has not been reached, proceed to step S141 until the total number of training rounds is reached.

[0056] S145. The training agent outputs obstacle avoidance actions based on training observation information and the initial obstacle avoidance strategy network, and evaluates the obstacle avoidance actions based on the initial obstacle avoidance evaluation network to obtain the second evaluation result corresponding to the obstacle avoidance actions. S146. Based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, update the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network until the total number of training rounds is reached, and obtain the obstacle avoidance experience network.

[0057] Optionally, S146 may include the following sub-steps: The training observation information, obstacle avoidance actions, second reward, and training observation information at the next moment are stored as a third experience tuple in the experience replay buffer. The fourth experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The second policy gradient and the second temporal difference error are calculated based on the fourth empirical tuple; The parameters of the initial obstacle avoidance policy network are updated according to the second policy gradient, and the parameters of the initial obstacle avoidance evaluation network are updated according to the second temporal difference error, until the total number of training rounds is reached. The initial obstacle avoidance evaluation network at the current moment is then determined as the obstacle avoidance experience network.

[0058] In this embodiment, the training process of the obstacle avoidance experience network is similar to that of the search and tracking experience network, and the specific process will not be described in detail.

[0059] In practical implementation, the obstacle avoidance experience network can be trained in the following ways: The drones within the cluster are abstracted as agents. Each agent is trained using the dominant actor-critic algorithm with a fusion-priority experience replay mechanism. This training is abstracted into a quintuple. Markov decision process, in which Let be the state space of the intelligent agent. Represents the action space of an intelligent agent. It is the reward obtained by the intelligent agent. It is the state transition probability. Represents the discount factor; Step S31: Configure the drone obstacle avoidance strategy network. parameters and drone obstacle avoidance evaluation network parameters Perform random initialization, setting the total number of training iterations (Episode), the learning rate, and the maximum number of steps per iteration; Step S32: Start the UAV obstacle avoidance simulation training environment and randomly initialize the positions of the cluster of UAVs and all dynamic obstacles; Step S33: Each training agent obtains training observation information; Step S34: The training agent obtains actions based on the current obstacle avoidance strategy and training observation information, and simultaneously receives a reward. And new state observation information; Step S35: Transfer the current second empirical tuple The accumulated reward is calculated and updated in the agent's experience replay buffer. Step S36: Sample batches of samples from the agent's experience replay buffer using the priority experience replay method, calculate the policy gradient and temporal difference error, and apply gradient descent to the policy network. parameters The parameters ω of the evaluation network ObsCritic are updated based on the temporal difference error. Step S36: Has the maximum number of steps for each round been reached? If yes, proceed to step S37; otherwise, proceed to step S33. Step S37: Has the total number of training rounds been reached? If not, proceed to step S32. If yes, obtain the initial obstacle avoidance strategy network. and initial obstacle avoidance evaluation network At this point, the initial obstacle avoidance evaluation network This is the required obstacle avoidance experience network.

[0060] Step 102: Load each tracking and obstacle avoidance network into the corresponding agent of each drone in the drone cluster, and initialize the action policy network corresponding to each agent. An intelligent agent refers to an abstract autonomous decision-making entity of each drone in a drone swarm during non-training processes, capable of selecting actions based on observation information and action policy networks.

[0061] Action policy network refers to a network that outputs action probability distribution or action commands based on observation information. Its parameters are shared and updated through an experience playback mechanism to coordinate the decision-making behavior of each UAV in the cluster.

[0062] In this embodiment, the tracking and obstacle avoidance network is configured to the corresponding agents of each drone in the drone swarm, and the action policy network corresponding to each agent is initialized. This allows each drone to encode and make decisions based on the observed information through the action policy network, and then output the action selection result, such as the probability of outputting forward, lateral, and altitude displacement control commands. This enables each drone to have both independent decision-making and collaborative capabilities in swarm collaborative tasks. Simultaneously, multiple agents share and update the weight parameters of each action policy network through an experience replay mechanism combined with parameter sharing. This allows the experience gained by any agent in the swarm to be indirectly used by other agents, thereby effectively improving the overall learning efficiency and policy generalization ability of the swarm and avoiding the problem of insufficient training caused by sparse experience from individual drones.

[0063] The network model architecture diagram of the action policy network is as follows: Figure 3 As shown, the action policy network comprises an input layer, an encoding layer, a concatenation layer, a neural network module, and an output layer. The input layer is divided into four paths: local information, friendly drone information, target information, and obstacle information. The encoding layer sets an encoder for each path of information, mapping raw observation data with different dimensions and physical meanings into a feature vector of a unified dimension. The encoder can employ fully connected layers, convolutional layers, recurrent networks, or graph networks depending on the information type. For example, friendly drone information can use a Graph Attention Network (GAT) to aggregate features from neighboring drones, while target and obstacle information can use convolutional layers or fully connected layers to extract spatial relationships. The concatenation layer concatenates the four encoded features by channel or dimension to form a joint feature vector. The neural network module consists of several fully connected layers, activation functions, or residual structures, performing nonlinear transformations on the joint features. The output layer outputs action distribution parameters or deterministic action quantities based on the action space type, such as positive and negative displacement control commands for the drone along the x, y, and z axes.

[0064] Specifically, after acquiring observation information, the agent groups it into four categories: local, friendly, target, and obstacle, and performs normalization or masking processing. Each type of information is input into its corresponding encoder to obtain local, friendly, target, and obstacle features. These four types of features are fused into joint features through a concatenation layer. The joint features are then input into a neural network module, which outputs actions after multiple layers of nonlinear mapping. During the training phase, actions can be sampled based on the output distribution; during the deployment phase, the mean or maximum probability action can be selected, and height, speed, and emergency obstacle avoidance priority coverage can be applied. This architecture preserves the characteristics of each information source through independent encoding, achieves cross-source fusion through concatenation, and learns the mapping from observation to action through the neural network module, leading to improvements in collaborative perception capabilities, real-time decision-making, and policy generalization capabilities.

[0065] Step 103: Obtain observation information of the current environment in real time through each intelligent agent; Observational information refers to the raw observational data acquired by the agent in the environment through sensors or communication links, including its own position, velocity, remaining energy, and the relative positions and motion states of surrounding obstacles or targets. This observational information is encoded and used as input to the action policy network to generate the agent's state representation at the current moment, thereby supporting the action policy network in outputting appropriate control commands.

[0066] After initializing the agents and the action policy network, each agent periodically collects its own position, distance from the previous action, speed, remaining energy, and the relative positions and motion states of surrounding obstacles and targets, forming the current moment's observation information. Subsequently, each agent preprocesses the observation information and inputs it into the action policy network. The network then performs forward inference to output the probability distribution of each candidate action, and determines the action to be executed based on probability sampling or a greedy strategy. This process provides environmental support for collaborative decision-making, enabling agents to simultaneously consider target search and tracking as well as obstacle avoidance requirements.

[0067] Step 104: Based on the observation information and experience playback mechanism, the weight parameters of each action strategy network are shared and updated to obtain the multi-target tracking and obstacle avoidance strategy network corresponding to each agent.

[0068] The experience replay mechanism refers to storing experience tuples generated during the interaction between the agent and the environment in a replay buffer, and sampling from these tuples according to certain rules during training to update network parameters. In this embodiment, the experience replay mechanism incorporates a priority sampling strategy, that is, assigning priority weights to experience tuples so that high-value samples (such as target loss, near-field obstacle avoidance, and other critical events) are sampled more frequently, thereby accelerating the policy network's learning and adaptation capabilities to key scenarios. Simultaneously, to balance the impact of new and old experiences on network updates, importance sampling weight correction is introduced during sampling to reduce distribution bias caused by priority sampling and ensure the stability of the training process.

[0069] The multi-target tracking and obstacle avoidance strategy network refers to a strategy network used by UAV swarms to collaboratively perform multi-target search, tracking, and obstacle avoidance tasks. It consists of an action policy network, a target search and tracking evaluation network, a UAV obstacle avoidance evaluation network, and their corresponding mirror networks. This network architecture is based on the Actor-Critic framework. The action policy network generates control commands based on current observation information, while the target search and tracking evaluation network and the UAV obstacle avoidance evaluation network evaluate actions from the dimensions of task achievement and safety constraints, respectively, thus providing multi-dimensional value feedback for policy updates. The two mirror evaluation networks gradually approximate the parameters of their respective evaluation networks through soft updates, improving the stability of value estimation during training and reducing the risk of policy oscillations caused by rapid changes in evaluation network parameters.

[0070] In this embodiment, after acquiring observation information, each agent inputs it into the action policy network to obtain the probability distribution of each candidate action, and samples based on this distribution to determine the current action to be executed. Simultaneously, the target search and tracking reward and obstacle avoidance reward corresponding to this action are calculated by the respective evaluation network and summarized into a total reward signal fed back to the agent. The agent stores the current experience tuple (state, action, reward, next state) in the experience replay buffer and samples it according to priority weights during subsequent training to strengthen the learning of key experiences. Subsequently, the action policy network and evaluation network update their parameters through policy gradients and temporal difference errors, respectively, while the mirror network synchronizes parameters through soft updates to maintain the stability of value estimation. During training, each agent makes independent decisions and acts collaboratively. Multiple agents share parameters, allowing the same policy network to be updated jointly based on the experience of different UAVs. Simultaneously, the relative weights of target tracking and obstacle avoidance in the loss can be dynamically adjusted according to the tracking and obstacle avoidance network; for example, increasing the obstacle avoidance weight when the obstacle avoidance evaluation is insufficient, and increasing the tracking weight when the tracking evaluation is insufficient. This leads to a multi-target tracking and obstacle avoidance strategy network, which improves the balance and generalization of collaborative search tracking and obstacle avoidance.

[0071] In one example of this application, step 104 may include the following sub-steps S21-S25: S21. The agent determines the action to be performed based on the action policy network and observation information, respectively. S22. Calculate the reward information corresponding to the action, and combine the observation information and the action to construct the experience tuple corresponding to the agent, and cache it in the corresponding experience replay cache area. Action policy network refers to a network module used to map the current observation information of an agent into action distribution or deterministic action quantities. Its data processing process includes feature encoding, action generation and action constraint processing; for example, inputting friendly aircraft information, target information and obstacle information, and outputting UAV displacement control commands.

[0072] Execution refers to the control commands output by the action policy network and, after being constrained by feasibility, actually applied to the simulation environment or a real drone platform.

[0073] The experience replay buffer refers to a storage structure used to store experience tuples and support batch sampling, which can further support priority sampling, segmented storage and timestamp management.

[0074] In this embodiment, the intelligent agent, abstracted from a drone, reads observation information at the decision-making moment and inputs it into the action policy network. After feature encoding, action distribution generation or deterministic mapping, and action constraint processing, it outputs the action to be executed. The environment returns the corresponding reward information according to the executed action. Based on the observation information, executed action, reward information, state transition probability, and discount factor, the intelligent agent creates an experience tuple and writes it to the experience replay buffer.

[0075] It should be noted that the calculation of reward information is similar to the calculation process for target search and tracking rewards and obstacle avoidance rewards described above, and the cumulative reward is also calculated. .

[0076] S23. Determine the dynamic weights based on the tracking and obstacle avoidance network and the agent's evaluation network; Optionally, the evaluation network includes an agent tracking evaluation network and an agent obstacle avoidance evaluation network; S23 may include the following sub-steps: Based on the tracking experience network and obstacle avoidance experience network in the tracking and obstacle avoidance network, calculate the first tracking evaluation value and the first obstacle avoidance evaluation value corresponding to the observation information respectively; An agent tracking evaluation network and an agent obstacle avoidance evaluation network are used to calculate the second tracking evaluation value and the second obstacle avoidance evaluation value corresponding to the observation information, respectively. Based on the first tracking evaluation value and the second tracking evaluation value, calculate the tracking correction information; Based on the first obstacle avoidance assessment value and the second obstacle avoidance assessment value, calculate the obstacle avoidance correction information; Dynamic weights are determined based on tracking correction information and obstacle avoidance correction information; The calculation process for the dynamic weights is as follows: ; in, The dynamic weights are the observation information at time t. To track and correct information, For obstacle avoidance correction information, e is a natural constant.

[0077] In this embodiment, the tracking and obstacle avoidance network outputs target search and tracking evaluation values ​​and obstacle avoidance evaluation values, respectively. The evaluation network outputs corresponding evaluation signals. The two types of evaluation signals are compared with expert experience evaluation signals to obtain the difference, which is then normalized and mapped to generate dynamic weights. These weights are used for subsequent loss weighting of the strategy network. This process ensures that sampling, evaluation, and weight determination are completed within the same interaction cycle. The weights are adjusted according to the evaluation differences, which allows for the derivation of an adaptive balance between tracking gains and obstacle avoidance safety in the strategy. This reduces the task bias caused by fixed weights and improves convergence stability and the responsiveness of obstacle avoidance priorities.

[0078] Specifically, the calculation process for the tracking correction information and obstacle avoidance correction information is as follows: ; ; S24. Based on the experience replay mechanism, sample the target experience tuples from the experience replay buffer, and combine them with dynamic weights to update the first parameter of the action policy network and the second parameter of the evaluation network. Furthermore, S24 may include the following sub-steps S241-S243: S241. Based on the experience replay mechanism, perform batch sampling of the experience replay buffer to obtain the target experience tuple; S242. Based on dynamic weights and target experience tuples, calculate the policy loss function of the action policy network; S243. Based on the policy loss function, the gradient descent method is used to update the first parameter of the action policy network and the second parameter of the evaluation network. The calculation process for the policy loss function is as follows: ; ; ; in, Let be the policy loss function of the action policy network at time t. For the action policy network at time t, observe the information Select the action to perform. The strategy probability, Let be the target search and tracking advantage function at time t. Let be the obstacle avoidance advantage function at time t. Tem p is the temperature coefficient. γ As a discount factor, For time t+1, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t+1, the evaluation network of the agent's obstacle avoidance is evaluated based on the observed information. The evaluation value, For time t, the evaluation network of the intelligent agent's obstacle avoidance is based on the observed information. The evaluation value.

[0079] Alternatively, S243 can be implemented through the following process: ; ; ; in, This is the second parameter of the agent tracking evaluation network. This is the second parameter of the intelligent agent obstacle avoidance evaluation network. The first parameter of the action policy network, The learning rate of the network is used to evaluate the tracking performance of the agent. The learning rate of the network is used to evaluate the obstacle avoidance of the intelligent agent. The learning rate of the action policy network. These are the initial parameters for the action policy network.

[0080] In this embodiment, a batch of target experience tuples is extracted from the experience replay buffer according to the experience replay mechanism. The extraction rule can adopt uniform or priority methods, and importance sampling weights can be attached. The observation information and execution actions in the target experience tuples are input into the evaluation network. The evaluation network outputs tracking evaluation values ​​and obstacle avoidance evaluation values. The two types of evaluation signals are weighted by dynamic weights to construct a policy loss function. The gradient of the policy loss function is calculated, and the first parameter of the action policy network is updated using the gradient descent method. At the same time, the second parameter of the evaluation network is updated according to the temporal difference error between the output of the evaluation network and the reward information. This process reduces the update correlation by using batch samples, adjusts the policy between tracking and obstacle avoidance according to the evaluation difference through dynamic weights, and forms a closed loop through the alternating update of policy and evaluation. It can be deduced that the sample utilization efficiency, convergence stability and task adaptability are improved.

[0081] In this scheme, the target search and tracking evaluation network (TCritic) and the drone obstacle avoidance evaluation network (OCritic) need to use their respective mirror networks to estimate the value at the next time step when calculating the temporal difference error. If the parameters of the main evaluation network are directly copied to the mirror network for each update, i.e., a hard update is used, the temporal difference target will change rapidly with the main evaluation network, causing the evaluation signal to fluctuate drastically, which in turn makes the policy gradient direction unstable and prone to training oscillations or even divergence. To solve the "moving target" problem during the bootstrapping update of the evaluation network in reinforcement learning, and to provide stable support for the dual evaluation network and dynamic weight structure in this scheme, a mirror target evaluation network mirrored with the agent tracking evaluation network and a mirror obstacle avoidance evaluation network mirrored with the agent obstacle avoidance evaluation network can be set. This scheme has both a tracking evaluation network and an obstacle avoidance evaluation network. The two evaluation outputs will jointly participate in the dynamic weight generation. If the evaluation network changes abruptly, the dynamic weights will also change abruptly, causing the policy loss to switch drastically between the tracking and obstacle avoidance terms. Soft updates can smooth the two evaluation outputs and make the dynamic weight transition more continuous.

[0082] For mirror target evaluation network parameter and mirror obstacle avoidance evaluation network parameters The parameter updates can be achieved through the following update process: ; ; in, This is the preset soft update coefficient.

[0083] S25. When the termination condition is met, the action policy network at the current moment is determined as the multi-target tracking and obstacle avoidance policy network of each agent.

[0084] In this embodiment, the indicators required for termination judgment are continuously collected during the training loop. When the termination condition is met, sampling and parameter updates are stopped. The parameters of the action policy network at the current moment are determined as the multi-target tracking and obstacle avoidance policy network for each agent, and this network is loaded into the same policy network shared by each agent or with shared parameters. During deployment, the agents generate actions only based on observation information and this policy network. The evaluation network, expert experience network, and dynamic weight calculation can exit the process. This process solidifies the tracking and obstacle avoidance coordination capabilities obtained in the training phase into an executable model, avoiding overfitting or policy drift introduced by continued training. It can be deduced that the computational load in the deployment phase is reduced, the decision response is accelerated, and the policy consistency is improved.

[0085] The termination condition can be set as the maximum number of training rounds. Each training round has a corresponding training step size. During the update process of the action policy network, the parameters are updated cyclically step by step until the maximum number of training steps is reached. The environmental conditions are then updated and the parameters are updated cyclically again until the maximum number of training rounds is reached. The action policy network at this moment is determined as the multi-target tracking and obstacle avoidance policy network of each agent.

[0086] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the process of obtaining the collaborative multi-target search, tracking, and obstacle avoidance strategy for UAV clusters in an embodiment of the present invention.

[0087] In this embodiment, at the start of training, the parameters of the action policy network, target evaluation network TCritic, obstacle avoidance evaluation network OCritic, and their corresponding mirror networks are initialized. Then, each agent loads the target search and tracking expert policy and the UAV obstacle avoidance expert policy as prior guidance. In response to a training environment reset command, the system reinitializes the states of the UAVs, targets, and dynamic obstacles for the UAV swarm target search and obstacle avoidance task scenario. The agents execute corresponding actions based on real-time observation information and the current policy, obtain new observation information, and calculate the reward Re based on the search and tracking reward function. t Calculate the reward Re based on the obstacle avoidance reward function. o The process then updates the relative weights of tracking and obstacle avoidance in the loss function based on the target search and tracking expert strategy and the obstacle avoidance expert strategy, respectively. The experience tuple, consisting of the current observation information, actions, rewards, and new observation information, is stored in the agent's experience replay buffer. Subsequently, batch samples are sampled from the buffer, and the weight parameters of the action policy network (Actor), target evaluation network (TCritic), obstacle avoidance evaluation network (OCritic), and the corresponding mirror network are updated using policy loss and temporal difference error. If the current episode's end condition is not met, the process returns to the real-time observation information and action execution stage for further iteration. If the condition is met, it further determines whether the agent's training end condition has been met. If not, it returns to the environment reset stage to start a new episode. If the condition is met, the trained multi-target tracking and obstacle avoidance policy network is output, and the process ends. This single-stage process, guided by expert strategies and dynamically adjusting weights, achieves progressive policy optimization under multi-task coupling of tracking and obstacle avoidance. The soft update of the mirror network ensures the smooth transition of evaluation signals and weights, thereby deriving an overall improvement in policy convergence stability, task adaptability, and deployment availability.

[0088] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0089] The following describes the network optimization device for multi-target tracking and obstacle avoidance strategy of UAV swarm provided in the embodiments of this application. The network optimization device for multi-target tracking and obstacle avoidance strategy of UAV swarm described below and the network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm described above can be referred to in correspondence with each other.

[0090] Please see Figure 5 The present invention also provides a network optimization device for multi-target tracking and obstacle avoidance strategies in unmanned aerial vehicle (UAV) swarms, comprising: The tracking and obstacle avoidance network acquisition module 501 is used to acquire the tracking and obstacle avoidance network trained through a simulation training environment. The agent initialization module 502 is used to load each tracking and obstacle avoidance network into the agent corresponding to each drone in the drone cluster, and initialize the action policy network corresponding to each agent. The observation information acquisition module 503 is used to acquire observation information of the current environment in real time through each intelligent agent; The network optimization module 504 is used to share and update the weight parameters of each action strategy network based on observation information and experience playback mechanism, so as to obtain the multi-target tracking and obstacle avoidance strategy network corresponding to each agent.

[0091] Optionally, the tracking obstacle avoidance network acquisition module 501 includes: The data acquisition submodule is used to acquire the hardware data of each drone in the drone cluster, the first motion data of the target to be tracked, the location data of the task area, and the second motion data of obstacles. The simulation training environment creation submodule is used to create a simulation training environment for the UAV cluster based on hardware data, first motion data, position data, and second motion data. The simulation training environment includes a task constraint model, a UAV motion model, a target motion feature model, and an obstacle feature model. The experience network initialization submodule is used to construct the corresponding training agent for the UAV based on the UAV motion model, and to initialize the initial search and tracking experience network and the initial obstacle avoidance experience network in the training agent. The experience network training submodule is used to train the training agent by interacting with the simulation training environment and training the training agent based on the interaction results until the training number is met, so as to obtain the search and tracking experience network and the obstacle avoidance experience network. The tracking and obstacle avoidance network generation submodule is used to employ a search tracking experience network and an obstacle avoidance experience network as the tracking and obstacle avoidance network.

[0092] Optionally, the initial search and tracking experience network includes an initial search and tracking policy network and an initial search and tracking evaluation network, and the initial obstacle avoidance experience network includes an initial obstacle avoidance policy network and an initial obstacle avoidance evaluation network; the experience network training submodule includes: The environment initialization unit is used to initialize the positions of each training agent, the position of the target to be tracked, the position of obstacles, and the total number of training rounds in the simulation training environment. The training observation information acquisition unit is used to acquire training observation information from the simulation training environment through the training agent; The first evaluation unit is used to output search and track actions by the trained agent based on training observation information and the initial search and track strategy network, and to evaluate the search and track actions based on the initial search and track evaluation network to obtain the first evaluation result corresponding to the search and track actions. The first training unit is used to update the parameters of the initial search and tracking strategy network and the initial search and tracking evaluation network based on the first evaluation result and the first reward corresponding to the search and tracking action, until the total number of training rounds is reached, and the search and tracking experience network is obtained. The second evaluation unit is used to output obstacle avoidance actions by the trained agent based on training observation information and the initial obstacle avoidance strategy network, and to evaluate the obstacle avoidance actions based on the initial obstacle avoidance evaluation network to obtain the second evaluation result corresponding to the obstacle avoidance actions. The second training unit is used to update the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, until the total number of training rounds is reached, and the obstacle avoidance experience network is obtained.

[0093] Optionally, the first training unit is specifically used for: Based on training observation information, search and tracking actions, the first reward, and the training observation information of the next time step, a first experience tuple is constructed and stored in the experience replay buffer; The second experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The first policy gradient and the first temporal difference error are calculated based on the second empirical tuple; The parameters of the initial search and tracking policy network are updated according to the first policy gradient, and the parameters of the initial search and tracking evaluation network are updated according to the first temporal difference error, until the total number of training rounds is reached. The initial search and tracking evaluation network at the current time is then determined as the search and tracking experience network.

[0094] Optionally, the second training unit is specifically used for: The training observation information, obstacle avoidance actions, second reward, and training observation information at the next moment are stored as a third experience tuple in the experience replay buffer. The fourth experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The second policy gradient and the second temporal difference error are calculated based on the fourth empirical tuple; The parameters of the initial obstacle avoidance policy network are updated according to the second policy gradient, and the parameters of the initial obstacle avoidance evaluation network are updated according to the second temporal difference error, until the total number of training rounds is reached. The initial obstacle avoidance evaluation network at the current moment is then determined as the obstacle avoidance experience network.

[0095] Optionally, the network optimization module 504 includes: The action determination submodule is used to determine the action to be performed by the agent based on the action policy network and observation information, respectively. The experience tuple caching submodule is used to calculate the reward information corresponding to the executed action, and combine the observation information and the executed action to construct the experience tuple corresponding to the agent, and cache it in the corresponding experience replay cache area; The dynamic weight determination submodule is used to determine dynamic weights based on the tracking and obstacle avoidance network and the agent's evaluation network; The parameter update submodule is used to sample target experience tuples from the experience replay buffer based on the experience replay mechanism, and update the first parameter of the action policy network and the second parameter of the evaluation network in combination with dynamic weights. The loop termination submodule is used to determine the action policy network at the current moment as the multi-target tracking and obstacle avoidance policy network for each agent when the termination condition is met.

[0096] Optionally, the evaluation network includes an agent tracking evaluation network and an agent obstacle avoidance evaluation network; the dynamic weight determination submodule is specifically used for: Based on the tracking experience network and obstacle avoidance experience network in the tracking and obstacle avoidance network, calculate the first tracking evaluation value and the first obstacle avoidance evaluation value corresponding to the observation information respectively; An agent tracking evaluation network and an agent obstacle avoidance evaluation network are used to calculate the second tracking evaluation value and the second obstacle avoidance evaluation value corresponding to the observation information, respectively. Based on the first tracking evaluation value and the second tracking evaluation value, calculate the tracking correction information; Based on the first obstacle avoidance assessment value and the second obstacle avoidance assessment value, calculate the obstacle avoidance correction information; Dynamic weights are determined based on tracking correction information and obstacle avoidance correction information; The calculation process for the dynamic weights is as follows: ; in, The dynamic weights are the observation information at time t. To track and correct information, For obstacle avoidance correction information, e is a natural constant.

[0097] Optionally, the parameter update submodule specifically includes: The batch sampling unit is used to sample the experience replay buffer in batches based on the experience replay mechanism to obtain the target experience tuple; The policy loss function calculation unit is used to calculate the policy loss function of the action policy network based on dynamic weights and target experience tuples. The parameter update unit is used to update the first parameter of the action policy network and the second parameter of the evaluation network based on the policy loss function and using gradient descent. The calculation process for the policy loss function is as follows: ; ; ; in, Let be the policy loss function of the action policy network at time t. For the action policy network at time t, observe the information Select the action to perform. The strategy probability, Let be the target search and tracking advantage function at time t. Let be the obstacle avoidance advantage function at time t, where Temp is the temperature coefficient and γ is the discount factor. For time t+1, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t+1, the evaluation network of the agent's obstacle avoidance is evaluated based on the observed information. The evaluation value, For time t, the evaluation network of the intelligent agent's obstacle avoidance is based on the observed information. The evaluation value.

[0098] Optionally, the parameter update unit is specifically used for: ; ; ; in, This is the second parameter of the agent tracking evaluation network. This is the second parameter of the intelligent agent obstacle avoidance evaluation network. The first parameter of the action policy network, The learning rate of the network is used to evaluate the tracking performance of the agent. The learning rate of the network is used to evaluate the obstacle avoidance of the intelligent agent. The learning rate of the action policy network. These are the initial parameters for the action policy network.

[0099] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0101] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0103] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] In the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0105] In the description of the embodiments of the present invention, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the present invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A network optimization method for multi-target tracking and obstacle avoidance strategies in a drone swarm, characterized in that, include: Obtain the tracking and obstacle avoidance network trained in a simulation training environment; Each of the aforementioned tracking and obstacle avoidance networks is loaded into the corresponding intelligent agent of each drone in the drone cluster, and the action policy network corresponding to each of the aforementioned intelligent agents is initialized. Each of the aforementioned intelligent agents acquires real-time observation information about the current environment; Based on the observation information and experience playback mechanism, the weight parameters of each action policy network are shared and updated to obtain the multi-target tracking and obstacle avoidance policy network corresponding to each agent.

2. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 1, characterized in that, The step of obtaining the tracking and obstacle avoidance network trained in a simulation training environment includes: Acquire hardware data of each drone in the drone swarm, first motion data of the target to be tracked, location data of the task area, and second motion data of obstacles; Based on the hardware data, the first motion data, the position data, and the second motion data, a simulation training environment corresponding to the UAV cluster is created; the simulation training environment includes a task constraint model, a UAV motion model, a target motion feature model, and an obstacle feature model; Based on the UAV motion model, a training agent corresponding to the UAV is constructed, and the initial search and tracking experience network and the initial obstacle avoidance experience network in the training agent are initialized. The training agent interacts with the simulation training environment, and the training agent is trained based on the interaction results until the training number is met, so as to obtain the search and tracking experience network and the obstacle avoidance experience network. The search and tracking experience network and the obstacle avoidance experience network are used as the tracking and obstacle avoidance network.

3. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 2, characterized in that, The initial search and tracking experience network includes an initial search and tracking strategy network and an initial search and tracking evaluation network; the initial obstacle avoidance experience network includes an initial obstacle avoidance strategy network and an initial obstacle avoidance evaluation network. The step of interacting with the training agent and the simulation training environment, and training the training agent based on the interaction results until the training count is met, to obtain the search and tracking experience network and the obstacle avoidance experience network, includes: Initialize the positions of each training agent, the target to be tracked, the obstacles, and the total number of training rounds within the simulation training environment; The training agent acquires training observation information from the simulated training environment. The training agent outputs a search and tracking action based on the training observation information and the initial search and tracking strategy network, and evaluates the search and tracking action based on the initial search and tracking evaluation network to obtain a first evaluation result corresponding to the search and tracking action. Based on the first evaluation result and the first reward corresponding to the search and tracking action, update the parameters of the initial search and tracking strategy network and the initial search and tracking evaluation network until the total number of training rounds is reached, and obtain the search and tracking experience network; The trained agent outputs obstacle avoidance actions based on the training observation information and the initial obstacle avoidance strategy network, and evaluates the obstacle avoidance actions based on the initial obstacle avoidance evaluation network to obtain a second evaluation result corresponding to the obstacle avoidance actions. Based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, update the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network until the total number of training rounds is reached, thus obtaining the obstacle avoidance experience network.

4. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 3, characterized in that, The step of updating the parameters of the initial search tracking strategy network and the initial search tracking evaluation network based on the first evaluation result and the first reward corresponding to the search tracking action, until the total number of training rounds is reached, to obtain the search tracking experience network, includes: Based on the training observation information, the search and tracking action, the first reward, and the training observation information at the next moment, a first experience tuple is constructed and stored in the experience replay buffer; The second experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The first policy gradient and the first temporal difference error are calculated based on the second empirical tuple; The parameters of the initial search and tracking policy network are updated according to the first policy gradient, and the parameters of the initial search and tracking evaluation network are updated according to the first temporal difference error, until the total number of training rounds is reached, and the initial search and tracking evaluation network at the current time is determined as the search and tracking experience network.

5. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 3, characterized in that, The step of updating the parameters of the initial obstacle avoidance strategy network and the initial obstacle avoidance evaluation network based on the second evaluation result and the second reward corresponding to the obstacle avoidance action, until the total number of training rounds is reached, to obtain the obstacle avoidance experience network, includes: The training observation information, the obstacle avoidance action, the second reward, and the training observation information at the next moment are stored as a third experience tuple in the experience replay buffer. The fourth experience tuple is obtained in batches from the experience replay buffer based on the priority experience replay mechanism; The second policy gradient and the second temporal difference error are calculated based on the fourth empirical tuple; The parameters of the initial obstacle avoidance policy network are updated according to the second policy gradient, and the parameters of the initial obstacle avoidance evaluation network are updated according to the second temporal difference error, until the total number of training rounds is reached, and the initial obstacle avoidance evaluation network at the current moment is determined as the obstacle avoidance experience network.

6. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 1, characterized in that, The step of sharing and updating the weight parameters of each action policy network based on the observation information and experience playback mechanism to obtain the multi-target tracking obstacle avoidance policy network corresponding to each agent includes: The agent determines the action to be performed based on the action policy network and the observation information, respectively. Calculate the reward information corresponding to the executed action, and combine the observation information and the executed action to construct the experience tuple corresponding to the agent, and cache it in the corresponding experience replay cache area; Dynamic weights are determined based on the tracking and obstacle avoidance network and the evaluation network of the agent; Based on the experience replay mechanism, target experience tuples are sampled from the experience replay buffer, and combined with the dynamic weights, the first parameter of the action policy network and the second parameter of the evaluation network are updated. When the termination condition is met, the action policy network at the current moment is determined as the multi-target tracking and obstacle avoidance policy network of each agent.

7. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 6, characterized in that, The evaluation network includes an agent tracking evaluation network and an agent obstacle avoidance evaluation network; the step of determining dynamic weights based on the tracking and obstacle avoidance network and the agent's evaluation network includes: Based on the tracking experience network and obstacle avoidance experience network in the tracking and obstacle avoidance network, calculate the first tracking evaluation value and the first obstacle avoidance evaluation value corresponding to the observation information respectively. Using the agent's agent tracking evaluation network and agent obstacle avoidance evaluation network, the second tracking evaluation value and the second obstacle avoidance evaluation value corresponding to the observation information are calculated respectively. Based on the first tracking evaluation value and the second tracking evaluation value, calculate the tracking correction information; Based on the first obstacle avoidance assessment value and the second obstacle avoidance assessment value, calculate obstacle avoidance correction information; Based on the tracking correction information and the obstacle avoidance correction information, determine the dynamic weights; The calculation process for the dynamic weights is as follows: ； in, The dynamic weights are the observation information at time t. To track and correct information, For obstacle avoidance correction information, e is a natural constant.

8. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 6, characterized in that, The step of sampling target experience tuples from the experience replay buffer based on the experience replay mechanism, and updating the first parameters of the action policy network and the second parameters of the evaluation network in combination with the dynamic weights, includes: Based on the experience replay mechanism, the experience replay buffer is sampled in batches to obtain the target experience tuple; Based on the dynamic weights and the target experience tuples, the policy loss function of the action policy network is calculated; Based on the policy loss function, the first parameters of the action policy network and the second parameters of the evaluation network are updated using gradient descent. The calculation process of the policy loss function is as follows: ； ； ； in, Let be the policy loss function of the action policy network at time t. For the action policy network at time t, observe the information Select the action to perform. The strategy probability, Let be the target search and tracking advantage function at time t. Let be the obstacle avoidance advantage function at time t. Tem p is the temperature coefficient. γ As a discount factor, For time t+1, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t, the agent tracks and evaluates the network's response to observed information. The evaluation value, For time t+1, the evaluation network of the agent's obstacle avoidance is evaluated based on the observed information. The evaluation value, For time t, the evaluation network of the intelligent agent's obstacle avoidance is based on the observed information. The evaluation value.

9. The network optimization method for multi-target tracking and obstacle avoidance strategy of UAV swarm according to claim 8, characterized in that, The steps of updating the first parameters of the action policy network and the second parameters of the evaluation network based on the policy loss function using gradient descent include: ； ； ； in, This is the second parameter of the agent tracking evaluation network. This is the second parameter of the intelligent agent obstacle avoidance evaluation network. The first parameter of the action policy network, The learning rate of the network is used to evaluate the tracking performance of the agent. The learning rate of the network is used to evaluate the obstacle avoidance of the intelligent agent. The learning rate of the action policy network. These are the initial parameters for the action policy network.

10. A network optimization device for multi-target tracking and obstacle avoidance strategies in a drone swarm, characterized in that, include: The tracking and obstacle avoidance network acquisition module is used to acquire the tracking and obstacle avoidance network trained through a simulation training environment. The agent initialization module is used to load each of the tracking and obstacle avoidance networks into the agent corresponding to each UAV in the UAV cluster, and to initialize the action policy network corresponding to each agent. The observation information acquisition module is used to acquire observation information of the current environment in real time through each of the intelligent agents; The network optimization module is used to share and update the weight parameters of each action strategy network according to the observation information and experience playback mechanism, so as to obtain the multi-target tracking and obstacle avoidance strategy network corresponding to each agent.

Images