A switching multi-agent cooperative tracking control method based on deep reinforcement learning

Abstract

This invention relates to the field of multi-agent cooperative control technology, and in particular to a switching multi-agent cooperative tracking control method based on deep reinforcement learning. The method includes building a two-layer independent architecture for each independent agent, constructing a multi-level grouping architecture for the overall multi-agent cluster, proactively adjusting the internal structure of the multi-level grouping architecture based on prediction results, and using causal analysis technology to trace and determine the root causes of sudden changes and select an execution strategy. This invention adopts a two-layer independent architecture scheme, configuring a leader layer and an executor layer for each agent. Through an asynchronous decision-making mechanism with unidirectional instruction transmission, it achieves complete decoupling between global planning and local execution, ensuring the global optimality of the cooperative tracking task, meeting the real-time control requirements in dynamic environments, eliminating the single-point failure risk of centralized decision-making, and improving the cluster decision-making efficiency and operational stability.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control technology, and in particular to a switching multi-agent cooperative tracking control method based on deep reinforcement learning. Background Technology

[0002] With the rapid iterative development of unmanned systems, artificial intelligence, and the Internet of Things, swarm systems composed of multiple mobile intelligent agents (drones, unmanned vehicles, mobile robots, etc.) have been widely applied in various core scenarios such as urban security, emergency rescue, target capture, border patrol, and maritime supervision, thanks to their core advantages of distributed deployment, redundancy and fault tolerance, and strong collaborative operation capabilities. Among these, multi-agent collaborative tracking and control is the core foundational technology for swarm systems to complete various operational tasks. Its core objective is to enable multiple intelligent agents to continuously and without loss lock-on, collaboratively capture, and stably track non-cooperative moving targets in complex dynamic environments, while ensuring the operational safety and collaborative consistency of the swarm itself.

[0003] In recent years, deep reinforcement learning has become the mainstream technology for solving multi-agent cooperative tracking problems in complex environments due to its strong nonlinear fitting ability, end-to-end decision-making capability, and adaptive learning capability in unknown dynamic environments. Related research and engineering applications have made some progress. However, existing deep reinforcement learning-based multi-agent cooperative tracking technologies still face the following unavoidable problems in practical engineering applications:

[0004] First, multi-agent systems generally adopt a single-layer integrated decision-making and execution architecture, with global task planning and local real-time control being highly coupled and unable to achieve effective decoupling. If a centralized decision-making architecture is adopted, all state information of the entire cluster needs to be aggregated to a single decision center, which can easily lead to an explosion of decision dimensions and an exponential increase in computing power requirements. At the same time, there is a risk that a single point of failure will cause the entire cluster to fail collaboratively. If a fully distributed decision-making architecture is adopted, each agent makes autonomous decisions based on local information, which can easily get trapped in local optima, making it impossible to guarantee the optimality of the global tracking task and easily leading to problems such as target loss and collaborative chaos.

[0005] Second, multi-agent clusters often employ fixed topology architectures or only support passive topology adjustment mechanisms after a failure. In scenarios involving urban building obstruction, strong electromagnetic interference, and dynamic changes in communication links caused by agent maneuvering, communication interruptions, packet loss, and command transmission failures are highly likely to occur. Passive topology adjustments suffer from adjustment lag, which can easily lead to the interruption of collaborative tracking tasks and target loss. The robustness and connectivity of the cluster operation in complex interference environments cannot be effectively guaranteed.

[0006] Therefore, there is an urgent need to propose a technical solution that can solve the above problems. Summary of the Invention

[0007] To overcome the problems mentioned in the background art, this invention proposes a switching multi-agent cooperative tracking control method based on deep reinforcement learning.

[0008] The technical solution of this invention is: a switching multi-agent cooperative tracking control method based on deep reinforcement learning, applied to a cluster cooperative tracking system containing multiple mobile agents. The system includes an agent perception module, a communication module, a motion control module, and a global decision-making module. The specific steps are as follows:

[0009] S1: Build a two-layer independent architecture for each independent agent in the multi-agent cluster, including a leader layer and an executor layer. Relying on the two-layer independent architecture, global overall planning and local execution control are decoupled in layers.

[0010] S2: Construct a multi-level grouping architecture for the overall multi-agent cluster, dynamically group the agents based on their actual locations and hardware resources, hierarchically set group leaders and global leaders, integrate high-precision maps, decision data, and environmental data to predict communication anomalies, and proactively adjust the internal structure of the multi-level grouping architecture based on the prediction results.

[0011] S3: Real-time online monitoring and tracking of various sudden changes in the entire operation process, and using causal analysis technology to trace and determine the root cause of the sudden changes, and distinguish the event type based on the traceability results, and select to switch the execution strategy or delay to keep the original tracking strategy running until the cluster operation status returns to normal.

[0012] S4: The entire process integrates deep reinforcement learning algorithms to drive a two-layer architecture decision output and multi-level cluster collaborative scheduling, combined with dynamic topology adjustment and event-differentiated strategy control, until all job objectives are successfully tracked collaboratively.

[0013] Preferably, step S1 specifically includes:

[0014] S11: Define the first layer of architecture for each intelligent agent as the leader layer, with global information reception and overall planning as the core functions. Collect group global state, target state, and environmental state data in a unified manner, and generate overall planning instructions and task allocation instructions for the global tracking task.

[0015] S12: Define the second layer architecture of each agent as the executor layer, which unidirectionally receives planning instructions from its leader layer, and combines its own local perception information and the state information of neighboring agents to autonomously plan and generate a refined and real-time executable specific tracking and control scheme.

[0016] As a preferred option, the multi-agent, multi-layer architecture construction and grouping in step S2 are specifically as follows:

[0017] S21: Using the real-time spatial location distance, computing resources, communication resources, and battery life of intelligent agents as dividing indicators, complete the automatic division of multi-agent clusters and form several independent cooperative groups;

[0018] S22: Within each independent group, a separate group leader is configured to be responsible for instruction forwarding, status aggregation, and small-scale collaborative management within the group;

[0019] S23: A single overall leader is selected across all groups to coordinate the operation of all groups, cross-group collaborative scheduling, and global architecture management.

[0020] Preferably, the steps of communication risk prediction and topology adaptive adjustment in step S2 include:

[0021] S24: Real-time collection of high-precision map geographic information, cluster historical decision-making and operation data, external environment perception data, and communication quality data of each node, and integration to construct a multi-source fusion dataset;

[0022] S25: Utilize multi-source fusion datasets to predict in real time the probability and timing of abnormal operating conditions such as communication interruption, signal blockage, and link failure;

[0023] S26: Based on the communication anomaly prediction results, the group reorganization, leader replacement, communication link reconstruction and node position adaptation adjustment are completed in advance before the abnormal working conditions occur, so as to realize the proactive pre-adjustment of the multi-layer architecture topology.

[0024] As a preferred option, the process for real-time detection and causal attribution of sudden changes in step S3 is as follows:

[0025] S31: Continuously collect and track target motion parameters, agent operation parameters, environmental interference parameters, and cluster communication parameters, and capture various sudden abnormal changes in multiple dimensions in real time;

[0026] S32: By using causal analysis techniques to build a causal relationship model, sort out the relationship between various influencing factors and sudden changes, and accurately determine the true causes and scope of impact of sudden changes.

[0027] As a preferred option, the strategy control logic based on the causal analysis results in step S3 is as follows:

[0028] S33: If the sudden change is determined to be a short-term, self-recoverable transient disturbance event, the strategy switching instruction will not be executed, and the existing tracking strategy will continue to run for a preset fixed duration.

[0029] S34: If the sudden change is determined to be an irreversible structural mutation event with long-term impact, immediately initiate the strategy switching process and replace it with a new collaborative tracking control strategy adapted to the current operating conditions.

[0030] As a preferred option, the dual-layer independent architecture is equipped with a deep reinforcement learning model throughout. The leader layer relies on deep reinforcement learning to solve the global optimal decision, while the executor layer relies on deep reinforcement learning to output local dynamic control quantities, thereby achieving hierarchical intelligent decision-making.

[0031] As a preferred option, both the overall leader and the group leaders adopt a dynamic selection mechanism, which can be rotated in real time according to cluster resource consumption, communication status, and task load to ensure the long-term stable operation of the multi-layer architecture.

[0032] As a preferred approach, the entire topology adjustment process adopts a smooth transition mode, ensuring that the tracking task execution is not interrupted during the adjustment process, thus guaranteeing the continuity of multi-agent tracking operations.

[0033] As a preferred approach, after the preset holding time expires, the cluster and environment operating status are checked a second time. If the status recovers, the original strategy is used indefinitely. If the status continues to be abnormal, a second linkage causal analysis is performed to complete the closed-loop adaptive collaborative tracking control for the entire scenario.

[0034] The beneficial effects of this invention are:

[0035] 1. Compared with the existing multi-agent architecture scheme that adopts a single-layer decision-making and execution integration, which has the inherent defects of high coupling between global planning and local control, centralized architecture is prone to decision dimension explosion and single point of failure risk, while distributed architecture is prone to getting trapped in local optima and cannot take into account both global task optimality and local control response efficiency. This invention adopts a two-layer independent architecture scheme that configures a leader layer and an executor layer for each agent. Through the asynchronous decision-making mechanism of unidirectional instruction transmission, it achieves complete decoupling between global planning and local execution. This not only ensures the global optimality of collaborative tracking tasks, but also meets the real-time control requirements in dynamic environments. At the same time, it eliminates the single point of failure risk of centralized decision-making and improves the cluster decision-making efficiency and operational stability.

[0036] 2. Compared with existing cluster management solutions that use fixed topology architecture or passive topology adjustment after a failure, which have inherent drawbacks such as strong adjustment lag and are prone to communication interruption, cluster disconnection, and task interruption in complex interference environments, this invention adopts a three-level, multi-level cluster topology architecture scheme, combined with dynamic grouping and a rotating leader selection mechanism. At the same time, it achieves advanced prediction of communication anomalies through multi-source data fusion, and proactively completes topology pre-adjustment before a failure occurs, changing passive remediation to proactive prevention and control. This completely eliminates the lag of traditional adjustment methods and greatly improves the communication connectivity and long-term operational robustness of the cluster in complex environments.

[0037] 3. Compared to existing technologies that use threshold-triggered instantaneous policy switching schemes, which lack causal tracing and essential differentiation of sudden changes and have inherent drawbacks such as inability to identify the intrinsic differences of sudden events, this invention adopts a causal analysis-based sudden change root cause tracing and differentiated policy control scheme. It completes the root cause location and classification of sudden changes through causal reasoning and adopts differentiated control logic for different types of events. This not only completely avoids the policy oscillation problem caused by instantaneous disturbances and ensures task continuity, but also enables rapid and accurate policy adaptation for structural changes, greatly improving the cluster's environmental adaptability and anti-interference capability.

[0038] 4. Compared to existing technologies that use a single deep reinforcement learning model to simultaneously complete global planning and local control, or that fragment deep reinforcement learning is applied to a single stage, which have inherent drawbacks such as poor model-scene adaptability, difficulty in model convergence, and insufficient decision-making accuracy, failing to fully leverage the adaptive decision-making advantages of deep reinforcement learning, this invention adopts a layered and adapted deep reinforcement learning full-closed-loop fusion-driven scheme. This scheme uses a two-layer architecture with different levels of matching and scene-highly adapted reinforcement learning models. At the same time, reinforcement learning-driven processes are integrated throughout the entire collaborative tracking process, forming a complete adaptive closed-loop control system. This significantly improves model convergence speed and decision-making accuracy, while meeting the differentiated needs of global planning and local control, and achieving adaptive optimization of the entire collaborative tracking process. Attached Figure Description

[0039] Figure 1 The diagram shown is a first three-dimensional structural schematic of the switching multi-agent cooperative tracking control method based on deep reinforcement learning of the present invention.

[0040] Figure 2 The diagram shown is a second three-dimensional structural schematic of the switching multi-agent cooperative tracking control method based on deep reinforcement learning according to the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] Please see Figure 1-2This invention provides an embodiment: a switching multi-agent cooperative tracking control method based on deep reinforcement learning, applied to a cluster cooperative tracking system containing multiple mobile agents. In this embodiment, the mobile agents are specifically 12 industrial-grade quadcopter drones, the cooperative tracking target is a dynamically moving unmanned vehicle target on the ground, and the operation scenario is continuous tracking of dynamic targets in a complex environment of urban built-up areas; the cluster cooperative tracking system is equipped with an agent perception module, a communication module, and a motion control module for each drone, and a global decision-making module is configured at the cluster ground control terminal.

[0043] System hardware and software basic configuration

[0044] Intelligent agent perception module: Each drone is equipped with an airborne binocular vision camera (1920×1080 resolution, 30fps), a 16-line lidar (detection range 100m, angular resolution 0.2°), a GPS / BeiDou dual-mode positioning unit (positioning accuracy ±2cm), an IMU inertial measurement unit (update frequency 200Hz), and a power monitoring sensor. Among them, the binocular vision camera and lidar are used to collect local environmental data and track the local motion data of the target. GPS / BeiDou and IMU are used to collect the drone's own position, speed, attitude, acceleration and other operating parameters. The power monitoring sensor is used to collect the drone's remaining flight time data. The timestamp synchronization accuracy of all perception data is ≤10ms.

[0045] Communication module: Each drone is equipped with a 5G-DTU data radio and a Mesh self-organizing network communication unit, supporting point-to-point communication between drones, intra-group broadcast communication, and full-cluster global communication, with a maximum communication distance of 1500m. At the same time, it can collect communication quality data such as signal-to-noise ratio, packet loss rate, and end-to-end latency of the communication link in real time, and realize 10Hz frequency real-time monitoring of the communication status of multiple nodes and full-duplex data interaction.

[0046] Motion control module: Each UAV is equipped with an industrial-grade airborne flight control unit, which supports receiving continuous control commands such as speed, position, attitude, and waypoints, and completes real-time motion control and trajectory tracking of the UAV. The underlying control frequency is not lower than 50Hz, and the trajectory tracking error is ≤0.5m, ensuring the real-time performance and stability of tracking control.

[0047] Global Decision Module: A high-performance industrial control computer deployed at the ground control terminal, equipped with an 8-core CPU and a 24GB GPU computing unit, running the Ubuntu 22.04 operating system and the ROS2 robot operating system. It is used for the aggregation of global data across the entire cluster, centralized training and online inference of deep reinforcement learning models, global topology scheduling, and global task planning. It also supports low-latency data interaction with the onboard decision units of each UAV, enabling coordinated linkage between global and local decision-making.

[0048] Software and Algorithm Infrastructure: All deep reinforcement learning models, time series prediction models, and causal analysis models are developed based on the Python 3.10 and PyTorch 2.0 frameworks. Clustering algorithms and data preprocessing algorithms are implemented based on the Scikit-learn library. All algorithms support offline pre-training and online incremental fine-tuning.

[0049] Complete steps explained in detail

[0050] The collaborative tracking control method in this embodiment follows a complete closed-loop process from task initialization to the completion of tracking all targets. The specific process is as follows:

[0051] Step S1: Construction and Layered Decoupling of the Two-Layer Independent Architecture of the Intelligent Agent

[0052] This step establishes the decision-making and execution infrastructure for a multi-agent cluster. Each drone is independently configured with a two-layer architecture consisting of a leader layer and an executor layer. This two-layer architecture features hardware decoupling, independent software operation, and unidirectional command transmission, achieving complete layered decoupling between global planning and local execution control. Specifically, this includes:

[0053] S11: Leadership Layer Architecture Setup and Function Implementation

[0054] The first layer of architecture for each agent is defined as the leader layer, whose core functions are global information reception and overall planning. In this embodiment:

[0055] Deployment rules: The leader layer master instance corresponding to the global overall leader is deployed on the global decision-making module on the ground. The leader layer instances corresponding to each group leader are deployed on the airborne decision-making unit of the corresponding UAV. The leader layer of ordinary intelligent agents serves as a hot backup decision-making unit, which is only activated when the leader switches and is normally in standby mode to reduce computing power consumption.

[0056] Algorithm selection: The leader layer is equipped with a deep reinforcement learning model throughout, specifically using the proximal policy optimization (PPO) algorithm. This algorithm has strong training stability and high robustness, is suitable for the discrete-continuous mixed action space of multi-agent global planning, and can achieve the global optimal decision solution.

[0057] Input state space: 256 dimensions, specifically including: global state data of position, speed, attitude, and remaining battery power of all 12 drones in the cluster; global motion state data of latitude, longitude, speed, heading, and acceleration of the tracked target; global high-precision raster map feature encoding data of the urban environment; communication link status data of the entire cluster; and historical decision command data that has been executed.

[0058] Output action space: 32-dimensional, specifically including: global sub-target position allocation for global tracking tasks, tracking task weight allocation for each UAV, safe flight constraint boundaries for UAV swarms, timing synchronization instructions for group collaboration, and scheduling instructions for global topology adjustment.

[0059] Reward function: A weighted summation is used, and the specific formula is as follows:

[0060] ;

[0061] in, The reward is a negative reward for the target tracking error; the smaller the tracking error, the higher the reward value. The higher the cluster's formation maintenance and communication connectivity, the higher the reward value for cluster coordination and consistency. The shorter the target lock time, the higher the reward value; For safety and obstacle avoidance rewards, positive rewards are given for avoiding collisions and not entering no-fly zones.

[0062] Decision-making mechanism: The decision-making frequency of the leader layer is set to 5Hz, that is, a global planning update is completed every 200ms. Emergency updates are only triggered when a structural change occurs in the global state, so as to avoid the waste of computing power and instruction oscillation caused by high-frequency decision-making.

[0063] S12: Executor Layer Architecture Setup and Function Implementation

[0064] The second layer of architecture for each agent is defined as the executor layer, with its core function being local fine-grained tracking control. In this embodiment:

[0065] Deployment rules: Each drone's onboard decision-making unit is independently deployed with its own dedicated executor layer architecture, completely decoupled from the leader layer on the same drone. It only receives planning instructions from its own leader layer in one direction and has no reverse control authority, ensuring the authority of the overall planning and the flexibility of local execution.

[0066] Algorithm selection: The executor layer is equipped with a deep reinforcement learning model throughout, specifically adopting the dual-delay deep deterministic policy gradient (TD3) algorithm. This algorithm is suitable for high-precision real-time control in continuous action spaces, can effectively solve the overestimation problem of traditional algorithms, and is suitable for real-time motion control scenarios of UAVs to complete the output of local dynamic control quantities.

[0067] Input state space: 128 dimensions, specifically including: planning instructions issued by the leader, the drone's real-time position / speed / attitude / battery operation data, local perception of environmental obstacles and target motion data, state data of 3 neighboring drones, and current communication link quality data.

[0068] Output motion space: 4-dimensional, specifically including the drone's roll angle, pitch angle, yaw angle, and throttle control values, which can be directly input to the motion control module to achieve real-time flight control and local obstacle avoidance of the drone.

[0069] Reward function: A weighted summation is used, and the specific formula is as follows:

[0070] ;

[0071] in, The reward is a negative one for instruction tracking errors; the higher the tracking accuracy of the leader's instructions, the higher the reward value. The reward is for local obstacle avoidance safety; the farther away from the obstacle, the higher the reward value. The reward is for trajectory smoothness; the smoother the change in control quantity, the higher the reward value, thus avoiding violent maneuvers by the drone.

[0072] Execution mechanism: The executor layer control frequency is set to 50Hz, that is, the control command is updated once every 20ms. It forms an asynchronous training and synchronous execution cooperation mechanism with the leader layer. The low-frequency update of global planning and the high-frequency response of local control are completely decoupled, which ensures both the optimality of the global tracking task and meets the accuracy requirements of real-time control.

[0073] Step S2: Building a multi-level grouping architecture for the cluster and proactively adjusting the topology.

[0074] This step involves building a collaborative management and control architecture for a multi-agent cluster. It establishes a three-tiered, multi-level grouping structure consisting of a general leader, group leaders, and ordinary agents. Simultaneously, it enables proactive prediction of communication risks and active pre-adjustment of the topology. Specifically, this includes:

[0075] S21: Dynamic Grouping of Multi-Agent Clusters

[0076] Using the real-time spatial location distance of intelligent agents, computing resources, communication resources, and battery life as dividing indicators, the multi-agent cluster is automatically divided into several independent cooperative groups. In this embodiment:

[0077] Quantification of indicators:

[0078] Spatial location distance: The real-time straight-line distance between drones is the core constraint. The maximum straight-line distance between drones in the group does not exceed 300m to ensure the stability of communication within the group;

[0079] Resource endowment score: Each drone is scored from 0 to 100 points for computing power resources, communication resources, and battery life resources. The computing power resources are scored based on the proportion of onboard CPU / GPU computing power, the communication resources are scored based on the maximum communication distance and communication bandwidth, and the battery life resources are scored based on the remaining battery percentage.

[0080] Grouping Algorithm: The K-means++ clustering algorithm is adopted, with spatial location coordinates as the core clustering feature and resource endowment score as the balance constraint. The 12 drones are automatically divided into 3 independent cooperative groups, with the number of drones in each group controlled between 3 and 5, to ensure that the resource endowment of each group is balanced and there are no obvious weaknesses.

[0081] Grouping update mechanism: Dynamic groups are updated every 10 seconds. Group reorganization is only triggered when there are significant changes in the spatial location of the drone or significant differences in resource endowment. Under normal circumstances, the grouping architecture is kept stable to avoid collaborative oscillations caused by frequent grouping.

[0082] S22: Group Leader Configuration and Selection

[0083] Within each independent group, a separate group leader is configured, responsible for instruction forwarding, status aggregation, and small-scale collaborative management within the group. In this embodiment:

[0084] Selection rules: A weighted voting method will be used, with each drone in the group being comprehensively scored based on the following indicators: communication quality within the group (weight 50%), perception capability (weight 30%), and relative position (weight 20%). The drone with the highest score will be selected as the group leader.

[0085] Core responsibilities: Receive global planning instructions from the overall leader and forward them to ordinary intelligent agents within the group; summarize the status and perception data of all UAVs within the group in real time and upload them to the overall leader; complete the local collaborative scheduling, obstacle avoidance collaboration, and secondary task allocation of UAVs within the group; and act as a communication relay node within the group to ensure the connectivity of communication links within the group.

[0086] S23: Selection and Configuration of the Overall Leader

[0087] A single overarching leader is selected across all groups to coordinate the operation of all groups, cross-group collaborative scheduling, and global architecture management. In this embodiment:

[0088] Selection rules: A weighted voting method is adopted, with global communication capability (weight 40%), computing resources (weight 30%), and remaining battery life (weight 30%) as scoring indicators to comprehensively score all drones in the entire cluster. The drone with the highest score is elected as the global leader.

[0089] Core responsibilities: Complete global task planning, global topology scheduling, and group management for the entire cluster; issue global planning instructions to the leaders of each group; aggregate global status data of the entire cluster, complete communication anomaly prediction and global decision-making for emergencies; trigger global policy switching and topology adjustment;

[0090] Dynamic selection mechanism: Both the overall leader and group leaders adopt a dynamic rotation mechanism, and a scoring review is conducted every 30 seconds. When a leader has less than 20% battery remaining, communication quality is consistently below the threshold, or hardware failure occurs, a leader re-election is immediately triggered to complete the handover of authority and data synchronization in advance, ensuring the long-term stable operation of the multi-layer architecture and eliminating the risk of decision interruption.

[0091] S24: Construction of Multi-Source Fusion Dataset

[0092] In this embodiment, high-precision map geographic information, historical decision-making and operational data of the cluster, external environment perception data, and communication quality data of each node are collected in real time and integrated to construct a multi-source fusion dataset.

[0093] Data source details:

[0094] High-precision map geographic information: Centimeter-level high-precision urban environment map, including geographic feature data such as building outlines, terrain elevation, obstructed areas, and no-fly zones;

[0095] Cluster historical decision-making and operational data: the movement trajectory, decision-making commands, and task execution status data of the entire cluster of drones in the past 30 seconds;

[0096] External environment perception data: obstacle data, environmental occlusion data, and electromagnetic interference monitoring data collected by the lidar and vision cameras of each UAV;

[0097] Communication quality data for each node: signal-to-noise ratio, packet loss rate, end-to-end latency, and signal strength data for all communication links in the entire cluster over the past 10 seconds;

[0098] Dataset preprocessing: Timestamp synchronization, normalization, missing value completion, and outlier filtering are performed on all data to construct a time series dataset of length 100 for subsequent communication anomaly prediction.

[0099] S25: Advance prediction of abnormal communication conditions

[0100] In this embodiment, the probability and timing of abnormal operating conditions such as communication interruption, signal blockage, and link failure are predicted in real time using a multi-source fusion dataset.

[0101] Prediction Model: A TCN-LSTM fusion time series prediction model is adopted, in which 3 TCN layers extract spatial geographic and topological features, 2 LSTM layers extract time series features, and the output layer uses the Sigmoid activation function to output the probability of communication interruption of each communication link in the entire cluster within the next 5-second time window, with a prediction step size of 100ms.

[0102] Model training: The training mode of offline pre-training + online fine-tuning is adopted. In the offline stage, 1 million rounds of pre-training are completed based on the urban environment simulation dataset, covering various communication interruption scenarios caused by building obstruction, electromagnetic interference, and drone maneuvers. In the online execution stage, incremental fine-tuning is performed based on the real-time collected dataset, and the model parameters are updated every 1 second to improve prediction accuracy.

[0103] Predictive output: The model outputs the probability of interruption, the expected time of interruption, and the expected duration of interruption for each communication link in real time, providing a basis for topology pre-adjustment.

[0104] S26: Active pre-adjustment of multi-layered architecture topology

[0105] Based on the communication anomaly prediction results, packet reassembly, leader replacement, communication link reconstruction, and node position adaptation adjustments are completed in advance before the occurrence of abnormal operating conditions, realizing proactive pre-adjustment of the multi-layer architecture topology. In this embodiment:

[0106] Trigger threshold: The preset communication interruption probability threshold is 0.85. When the predicted interruption probability of a certain communication link within the next 5 seconds is ≥0.85, the topology pre-adjustment process is triggered immediately.

[0107] Differentiation adjustment strategy:

[0108] If the probability of communication link interruption between the group leader and the general leader is predicted to exceed the limit, a new group leader will be elected immediately in the corresponding group to complete data synchronization and authority handover, and the group leader replacement will be completed in advance.

[0109] If the probability of communication link interruption among multiple nodes in the group exceeds the standard, group reorganization is immediately triggered, and the corresponding node is adjusted to a neighboring group with better communication quality, thus completing the splitting and merging of groups.

[0110] If the probability of the core communication link of the overall leader being interrupted exceeds the standard, the overall leader pre-switch is immediately triggered, the overall leader is re-elected, all global planning data and task status are synchronized, and the authority is smoothly transferred.

[0111] If it is predicted that communication will be interrupted at local nodes due to building obstruction, the flight position of the corresponding drones will be adjusted in advance to avoid the obstructed areas and ensure the connectivity of the communication link.

[0112] Smooth transition mechanism: The entire topology adjustment adopts a smooth transition mode, and the tracking task is not interrupted during the adjustment process. All instruction handover and data synchronization are completed during the task operation. There are no abrupt changes in control variables, ensuring the continuity and stability of multi-agent tracking operations.

[0113] Step S3: Monitoring of sudden changes, causal tracing, and differentiated strategy management

[0114] This step involves dynamic adaptive control of the cluster tracing process, enabling real-time detection of sudden changes, root cause analysis, and differentiated strategy management to avoid frequent strategy oscillations and ensure the continuity of the tracing task. Specifically, it includes:

[0115] S31: Real-time online monitoring of sudden changes

[0116] Continuously collect and track target motion parameters, agent operation parameters, environmental interference parameters, and cluster communication parameters to capture various sudden and abnormal changes in multiple dimensions in real time. In this embodiment:

[0117] Monitoring dimensions and mutation thresholds:

[0118] Track target motion parameters: velocity change threshold ≥ 2.5 m / s, heading change threshold ≥ 45° / s, acceleration change threshold ≥ 4 m / s²;

[0119] Intelligent agent operating parameters: UAV position change threshold ≥6m, attitude change threshold ≥30°, remaining power drop threshold ≥10% / s, hardware fault alarm;

[0120] Environmental interference parameters: sudden obstacle distance threshold ≤15m, sudden change in environmental obstruction area, and sudden change in electromagnetic interference intensity threshold ≥20dBm;

[0121] Cluster communication parameters: Communication signal-to-noise ratio mutation threshold ≥25dB, packet loss rate mutation threshold ≥50%, link interruption alarm;

[0122] Monitoring mechanism: The monitoring frequency is synchronized with the executor-level control frequency at 50Hz. The collected data is compared with the preset mutation threshold in real time. When the change in any dimension of the parameter exceeds the corresponding threshold, an alarm for sudden change event is immediately triggered, and the causal tracing analysis process is initiated.

[0123] S32: Causal Analysis of Sudden Change Events

[0124] A causal relationship model is built using causal analysis techniques to analyze the correlation between various influencing factors and sudden changes, accurately determining the true triggering cause and scope of impact of the sudden changes. In this embodiment:

[0125] Causal analysis method: The structural causal model (SCM) combined with Do calculus causal reasoning techniques is used to locate the root cause of sudden changes;

[0126] Model building:

[0127] Outcome variable: Sudden changes that trigger alarms (such as sudden changes in target trajectory, communication interruption, abnormal drone status, etc.);

[0128] Candidate causal variables are divided into 5 categories: target motion (target speed, heading, acceleration), agent itself (drone position, attitude, battery level, hardware status), environmental interference (obstacles, occlusion, electromagnetic interference, terrain), communication status (signal-to-noise ratio, packet loss rate, link connectivity), and swarm collaboration (group topology, leader status, task allocation).

[0129] Construct a causal directed acyclic graph (DAG) to clarify the causal relationship paths between variables and eliminate spurious correlations;

[0130] Root cause identification: The average causal effect (ACE) of each candidate cause variable on the outcome variable is calculated through Do calculus. The larger the ACE value, the higher the causal contribution of the variable to the sudden change. The variable with the largest ACE value is the root cause variable of the sudden change. At the same time, the scope, duration and recoverability of the impact of the sudden change are determined based on the causal model.

[0131] S33: Policy control for instantaneously recoverable disturbances

[0132] If the sudden change is determined to be a short-term, self-recoverable transient disturbance event, the strategy switching instruction is not executed, and the existing tracking strategy continues to run for a preset fixed duration. In this embodiment:

[0133] Criteria for determining instantaneously recoverable disturbances: Based on causal analysis, the root cause is a brief communication blockage (expected duration ≤ 2s), a temporary target maneuver (expected fluctuation duration ≤ 3s), transient electromagnetic interference, or brief sensor noise, etc., without any permanent structural changes, and the disturbance can recover to its normal state on its own.

[0134] Control logic: Without triggering a global policy switch, the current tracking policy is maintained for a preset fixed duration of 2 seconds, which is based on the average recovery time of urban environmental disturbances. Within the maintenance duration, the cluster and environmental status are continuously monitored. If the status returns to the normal range, the original policy is continued to be used.

[0135] Secondary detection mechanism: After the preset retention period ends, a secondary state detection is performed. If the state has returned to normal, the original tracking strategy will be used indefinitely. If the state continues to be abnormal and has not returned to the normal range, the causal analysis process will be triggered again to determine the root cause and make decisions again.

[0136] S34: Strategy Switching Management for Structural Mutation Events

[0137] If the sudden change is determined to be an irreversible structural mutation event with long-term effects, the strategy switching process is immediately initiated to replace it with a new collaborative tracking control strategy adapted to the current operating conditions. In this embodiment:

[0138] Criteria for identifying structural mutation events: Based on causal analysis, the root cause is determined to be a permanent structural change that cannot be recovered on its own, such as a drone hardware failure, the appearance of a permanent obstacle, a permanent change in the target trajectory (expected duration ≥5s), a permanent interruption of the communication link, a failure of the leader node, or a permanent change in the operating environment.

[0139] Strategy switching process:

[0140] The PPO deep reinforcement learning model at the top leadership level immediately re-completes the planning update of the global tracking task based on the global state after the mutation, and generates a new global planning instruction adapted to the current working condition.

[0141] The new overall planning instructions are issued to the leaders of each group, who then redistribute tasks and forward instructions within their groups.

[0142] The TD3 model of each UAV's executor layer, based on the new planning instructions and the current local state, immediately updates the local control strategy and generates adapted real-time control instructions.

[0143] The strategy switching process adopts a smooth transition, and the control quantity is updated gradually using a linear interpolation method without abrupt changes, thus avoiding the drone from going out of control or oscillating and ensuring the continuity of the tracking mission.

[0144] Step S4: Full-process closed-loop collaborative tracking control and task completion

[0145] The entire process integrates deep reinforcement learning algorithms to drive a two-layer architecture for decision output and multi-level cluster collaborative scheduling, combined with dynamic topology adjustment and event-differentiated strategy control, until all job objectives are successfully collaboratively tracked. In this embodiment:

[0146] The entire process is closed-loop: starting from the task initialization and startup, steps S1-S3 form a complete closed loop, which is continuously executed in a loop, and the two-layer architecture decision, cluster topology and tracking control strategy are updated in real time to achieve adaptive collaborative tracking in complex dynamic environments.

[0147] Criteria for determining completion of tracking tasks: Meeting any one of the following criteria constitutes completion of all task objectives:

[0148] The target tracking duration is ≥30s, the tracking error is always ≤3m, and the target is not lost;

[0149] Track the target into the pre-set capture area and stop moving to complete the target capture;

[0150] If the target being tracked exceeds the preset work area, or if a task termination command is triggered, the tracking task will be completed.

[0151] Mission termination mechanism: After all operational objectives are completed, the overall leadership layer generates a return-to-home planning instruction, and each executor layer updates its return-to-home control strategy simultaneously to control the entire cluster of drones to safely return to the takeoff point, completing the entire closed loop of the collaborative tracking mission. Specific Implementation

[0152] This embodiment demonstrates the practical application of the above method in a park security scenario, specifically in the collaborative tracking of a cluster of unmanned ground vehicles. It verifies the technical effectiveness of this method on low-speed ground mobile platforms and in complex, enclosed park environments. Details are as follows:

[0153] 1. Application Scenarios and System Hardware Configuration

[0154] Application scenario: Closed industrial parks, including complex environments such as municipal roads, green belts, buildings, underground parking garage entrances and exits, and temporary construction areas, with an operating range of 2km×2km; the target being tracked is an illegally intruding mobile four-wheeled vehicle with a maximum speed of 40km / h; the core task is to continuously lock onto and track the target without losing track of it, and finally complete the encirclement and interception in the designated interception area of ​​the park.

[0155] Cluster configuration: A total of 8 four-wheel differential speed unmanned patrol vehicles, each of which is an independent mobile intelligent agent, equipped with a complete cluster collaborative tracking system. The core module configuration is as follows:

[0156] Intelligent agent perception module: Each unmanned vehicle is equipped with a 32-line lidar (detection range 200m, horizontal angular resolution 0.1°), an industrial-grade global shutter camera, a Beidou differential RTK positioning unit (positioning accuracy ±1cm), a wheel speed odometer, and an automotive-grade IMU inertial measurement unit, which can collect its own motion status, local environmental obstacles, and the position and motion parameters of the target in real time. The multi-sensor data timestamp synchronization accuracy is ≤5ms.

[0157] Communication module: Each unmanned vehicle is equipped with an industrial-grade Wi-Fi 6 wireless communication unit and a LoRa long-range backup communication unit, which supports Mesh self-organizing network, with a maximum line-of-sight communication distance of 800m. It can collect the signal-to-noise ratio, packet loss rate, and end-to-end latency data of the communication link in real time, and the communication status update frequency is 10Hz.

[0158] Motion control module: Each autonomous vehicle is equipped with an automotive-grade motion controller that supports closed-loop control of throttle, brake, and steering. The underlying control frequency is 100Hz, with a lateral error of ≤0.2m for straight path tracking and ≤0.5m for turning path tracking, enabling precise path tracking and emergency obstacle avoidance control.

[0159] Global Decision Module: A rack-mounted server deployed in the park's security control center, equipped with a 16-core industrial-grade CPU and a 32GB GPU computing unit, used for global data aggregation across the entire cluster, centralized inference of deep reinforcement learning models, global topology scheduling, and full-process task control, with end-to-end communication latency with unmanned vehicles ≤20ms.

[0160] Algorithm infrastructure: All algorithm models are developed based on the ROS2Humble robot operating system, deep reinforcement learning models are implemented based on the PyTorch2.2 framework, and offline pre-training is completed based on the Gazebo simulation environment, covering more than 95% of typical tracking scenarios in the park.

[0161] 2. Complete Process

[0162] S1: Construction of a dual-layer independent architecture for autonomous vehicle intelligent agents

[0163] Each of the eight autonomous vehicles is independently built with a two-layer architecture consisting of a leader layer and an executor layer. The two-layer architecture adopts a one-way command transmission mechanism, with hardware decoupling and software running independently, achieving complete layered decoupling between global overall planning and local execution control. The specific configuration is as follows:

[0164] Leader Layer: Employs a Proximal Policy Optimization (PPO) deep reinforcement learning model with a decision frequency of 2Hz (global planning updated every 500ms). The master instance is deployed in the park's global decision module, while leader layer instances for each group are deployed on the onboard industrial control computer of the corresponding unmanned vehicle. The leader layers of ordinary unmanned vehicles serve as hot backup units, remaining in standby mode. The input state space has 192 dimensions, specifically including: the global position, speed, heading, remaining battery power, and hardware health status of the 8 unmanned vehicles; the real-time position, speed, heading, and historical trajectory of the tracked target; the feature encoding of the park's global high-precision map; and the status data of the entire cluster's communication links. The output action space has 24 dimensions, specifically including: the global tracking sub-target point allocation, the tracking task weights of each unmanned vehicle, the planning of encirclement and interception points, safe driving constraints, and multi-vehicle collaborative timing instructions. The reward function uses a weighted summation form, with the formula:

[0165] ;

[0166] in, The target tracking error is given a negative reward. The closer the target is to the preset interception area, the higher the reward value. As a reward for safe driving, Rewards for cluster collaboration and consistency.

[0167] The executor layer employs a dual-delay deep deterministic policy gradient (TD3) deep reinforcement learning model, with a control frequency set to 50Hz (control commands updated every 20ms). Each autonomous vehicle's onboard industrial control computer is deployed as a dedicated instance. The input state space is 96-dimensional, including: planning commands issued by the leader layer, its own position / velocity / heading / wheel speed data, obstacle and target data perceived by local LiDAR, and real-time state data from two neighboring autonomous vehicles. The output action space is 3-dimensional, including: the autonomous vehicle's steering angle, throttle opening, and brake opening continuous control values, which can be directly input to the motion control module for real-time driving control. The reward function formula is:

[0168] ;

[0169] in, As a reward for instruction tracking accuracy, As a reward for local obstacle avoidance safety, To improve driving smoothness and prevent sudden braking, sharp turns, or other violent maneuvers.

[0170] Operating mechanism: The leader layer completes the overall planning of global capture and tracking, while the executor layer completes real-time path tracking and dynamic obstacle avoidance control. Global low-frequency planning and local high-frequency control work asynchronously and are completely decoupled, thus completely avoiding the decision-making dimension explosion and local optimal dilemma of a single-layer architecture.

[0171] Step S2: Multi-level grouping architecture construction and active topology pre-adjustment

[0172] The specific implementation is as follows:

[0173] Dynamic grouping implementation: Taking the real-time spatial location distance of the unmanned vehicles (maximum straight-line distance within the group ≤ 200m) as the core constraint, and combining the standardized scores of computing power, communication, and endurance resources of each unmanned vehicle, the K-means++ clustering algorithm is used to automatically divide the 8 unmanned vehicles into 2 independent collaborative groups, with 4 unmanned vehicles in each group, to ensure that the resource endowment within the group is balanced and there are no communication or computing power shortcomings. The group is reviewed every 10 seconds, and reorganization is triggered only when there are significant differences in spatial location and resources.

[0174] Group leader selection: Within each group, one autonomous vehicle is elected as the group leader through a weighted voting method, based on the scoring indicators of intra-group communication quality (weight 55%), perception range (weight 30%), and relative center position (weight 15%). The leader is responsible for forwarding instructions within the group, summarizing status data, and coordinating local scheduling within the group.

[0175] Global Leader Selection: Across the entire cluster, one autonomous vehicle is elected as the global leader through a weighted voting method, based on scoring indicators including global communication coverage (45% weight), onboard computing resources (30% weight), and remaining battery life (25% weight). This leader coordinates all group operations, cross-group collaborative scheduling, global architecture management, and global decision-making. Both the global leader and group leaders employ a dynamic, rotating mechanism, with a scoring review every 20 seconds. If a leader experiences issues such as remaining battery power below 20%, communication quality consistently below a threshold, or hardware alarms, a re-election is immediately triggered. This ensures data synchronization and smooth handover of permissions, guaranteeing the long-term stable operation of the multi-layered architecture.

[0176] Construction of multi-source fusion dataset: Real-time collection of building occlusion information from high-precision maps of the park, historical motion trajectory and decision data of unmanned vehicles, environmental perception data collected by LiDAR, and real-time communication quality data of each node. After completing timestamp synchronization and normalization preprocessing, a time series multi-source fusion dataset with a length of 80 is constructed.

[0177] Communication anomaly prediction: The TCN-LSTM fusion time series prediction model is adopted. The input is a multi-source fusion dataset, and the output is the probability of communication interruption of each communication link in the entire cluster within the next 4-second time window. The prediction step size is 100ms. The offline pre-training of the model covers all communication interruption scenarios caused by building obstruction, electromagnetic interference, and vehicle movement in the park. In the online execution phase, incremental fine-tuning is completed every 1 second, and the prediction accuracy is ≥95%.

[0178] Active pre-adjustment of topology: The preset communication interruption probability trigger threshold is 0.8. When it is predicted that the communication link between a group leader and the overall leader will be interrupted due to building obstruction within the next 4 seconds with a probability of 0.92, a new group leader will be elected immediately within that group to complete the handover of authority and full data synchronization in advance. When it is predicted that an unmanned vehicle will enter a permanently blocked signal area, its driving path will be adjusted in advance to avoid the blocked area. The entire topology adjustment adopts a smooth transition mode, and the tracking task is not interrupted during the adjustment process. There are no abrupt changes in the control variables, ensuring the continuity of the tracking operation.

[0179] Step S3: Causal Analysis of Sudden Changes and Differentiated Strategy Management

[0180] This step fully executes the entire process of emergency monitoring, causal tracing, and differentiated strategy management as defined in the claims, and is implemented as follows:

[0181] Real-time monitoring of sudden changes: Continuously collect and track target motion parameters, unmanned vehicle operation parameters, environmental interference parameters, and cluster communication parameters at a frequency of 50Hz. Preset sudden change thresholds: target speed change ≥3m / s, heading change ≥60° / s, sudden static obstacle distance ≤10m, communication packet loss rate change ≥40%. When the change of any parameter exceeds the corresponding threshold, a sudden change alarm is immediately triggered, and the cause-and-effect tracing process is initiated.

[0182] Cause-and-effect analysis of sudden changes: A causal relationship model based on structural causal model (SCM) and Do calculus is constructed. The sudden change that triggers the alarm is the outcome variable, and the target motion, the autonomous vehicle's own state, environmental interference, communication state, and cluster coordination state are the five categories of candidate cause variables. A causal directed acyclic graph is constructed, and the average causal effect (ACE) of each candidate variable is calculated through Do calculus. The variable with the largest ACE value is the root cause of the sudden change. At the same time, the recoverability and scope of impact of the event are determined.

[0183] Strategy control for transient disturbances: When a target is detected to make a temporary turn to avoid other vehicles at an intersection, the causal analysis determines that it is a transient and recoverable disturbance. The root cause is the target's temporary maneuver, and the expected duration is ≤2s. Therefore, the strategy switch is not triggered, and the original tracking strategy is maintained for a preset fixed duration of 2s. After the preset duration ends, a second state check is performed. After 1.8s, the target returns to its original driving route, and the state returns to normal. Therefore, the original strategy is used for a long time, which completely avoids the frequent oscillations of the strategy caused by transient disturbances.

[0184] Structural mutation strategy switching control: When a sudden temporary construction roadblock is detected on the main road of the park, and the cause-effect analysis determines that it is an irreversible structural mutation that cannot be recovered on its own, the strategy switching process is immediately initiated: The PPO model at the overall leader level immediately replans the global tracking and capture path based on the global environment after the mutation, and generates a brand-new global planning instruction; the leaders of each group complete the instruction forwarding and secondary allocation; the TD3 models at each unmanned vehicle executor level synchronously update the local control strategy and generate real-time control quantities adapted to the new path; the entire strategy switching process adopts a smooth transition using linear interpolation, with no step changes in control quantities and no interruption of the tracking task.

[0185] Step S4: Closed-loop collaborative tracking and task completion throughout the entire process

[0186] The entire process integrates deep reinforcement learning algorithms to drive a two-layer architecture for decision output and multi-level cluster collaborative scheduling. Combined with dynamic topology adjustment and event-differentiated strategy control, it continuously executes the entire closed loop of S1-S3, ultimately achieving: continuous locking and tracking of intruding targets for 42 seconds, with a tracking error always ≤1.5m, and no target loss throughout the process; target capture completed within the pre-set interception area of ​​the park, target stops moving, and the collaborative tracking of all targets is deemed complete; subsequently, the overall leadership layer generates a return planning command to control the unmanned vehicle cluster to return to the pre-set guard position, completing the entire closed loop of the mission.

[0187] 3. Implementation effect verification

[0188] This embodiment is compared with the existing traditional multi-agent tracking method in the park in the same scenario, and achieves the following core technical effects:

[0189] In complex, obstructed environments within the park, the cluster communication connectivity rate increased from 68% to 99%, with no issues of coordination failure or target loss caused by communication interruptions.

[0190] When faced with unexpected road conditions or temporary target maneuvers, the strategy oscillation rate decreased from 76% to less than 3%, and the tracking task continuity reached 100%.

[0191] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A switching multi-agent cooperative tracking control method based on deep reinforcement learning, characterized in that, An application is made to a cluster-based collaborative tracking system comprising multiple mobile intelligent agents. The system includes an agent perception module, a communication module, a motion control module, and a global decision-making module. The specific steps are as follows: S1: Build a two-layer independent architecture for each independent agent in the multi-agent cluster, including a leader layer and an executor layer. Relying on the two-layer independent architecture, global overall planning and local execution control are decoupled in layers. S2: Construct a multi-level grouping architecture for the overall multi-agent cluster, dynamically group the agents based on their actual locations and hardware resources, hierarchically set group leaders and global leaders, integrate high-precision maps, decision data, and environmental data to predict communication anomalies, and proactively adjust the internal structure of the multi-level grouping architecture based on the prediction results. S3: Real-time online monitoring and tracking of various sudden changes in the entire operation process, and using causal analysis technology to trace and determine the root cause of the sudden changes, and distinguish the event type based on the traceability results, and select to switch the execution strategy or delay to keep the original tracking strategy running until the cluster operation status returns to normal. S4: The entire process integrates deep reinforcement learning algorithms to drive a two-layer architecture decision output and multi-level cluster collaborative scheduling, combined with dynamic topology adjustment and event-differentiated strategy control, until all job objectives are successfully tracked collaboratively.

2. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 1, characterized in that, Step S1 specifically includes: S11: Define the first layer of architecture for each intelligent agent as the leader layer, with global information reception and overall planning as the core functions. Collect group global state, target state, and environmental state data in a unified manner, and generate overall planning instructions and task allocation instructions for the global tracking task. S12: Define the second layer architecture of each agent as the executor layer, which unidirectionally receives planning instructions from its leader layer, and combines its own local perception information and the state information of neighboring agents to autonomously plan and generate a refined and real-time executable specific tracking and control scheme.

3. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 2, characterized in that, The specific steps for building and grouping the multi-agent, multi-layer architecture in step S2 are as follows: S21: Using the real-time spatial location distance, computing resources, communication resources, and battery life of intelligent agents as dividing indicators, complete the automatic division of multi-agent clusters and form several independent cooperative groups; S22: Within each independent group, a separate group leader is configured to be responsible for instruction forwarding, status aggregation, and small-scale collaborative management within the group; S23: A single overall leader is selected across all groups to coordinate the operation of all groups, cross-group collaborative scheduling, and global architecture management.

4. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 3, characterized in that, The steps in step S2, namely communication risk prediction and topology adaptive adjustment, include: S24: Real-time collection of high-precision map geographic information, cluster historical decision-making and operation data, external environment perception data, and communication quality data of each node, and integration to construct a multi-source fusion dataset; S25: Utilize multi-source fusion datasets to predict in real time the probability and timing of abnormal operating conditions such as communication interruption, signal blockage, and link failure; S26: Based on the communication anomaly prediction results, the group reorganization, leader replacement, communication link reconstruction and node position adaptation adjustment are completed in advance before the abnormal working conditions occur, so as to realize the proactive pre-adjustment of the multi-layer architecture topology.

5. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 4, characterized in that, The process for real-time detection and causal attribution of sudden changes in step S3 is as follows: S31: Continuously collect and track target motion parameters, agent operation parameters, environmental interference parameters, and cluster communication parameters, and capture various sudden abnormal changes in multiple dimensions in real time; S32: By using causal analysis techniques to build a causal relationship model, sort out the relationship between various influencing factors and sudden changes, and accurately determine the true causes and scope of impact of sudden changes.

6. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 5, characterized in that, The strategy control logic based on the causal analysis results in step S3 is as follows: S33: If the sudden change is determined to be a short-term, self-recoverable transient disturbance event, the strategy switching instruction will not be executed, and the existing tracking strategy will continue to run for a preset fixed duration. S34: If the sudden change is determined to be an irreversible structural mutation event with long-term impact, immediately initiate the strategy switching process and replace it with a new collaborative tracking control strategy adapted to the current operating conditions.

7. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 6, characterized in that, The dual-layer independent architecture is equipped with a deep reinforcement learning model throughout. The leader layer relies on deep reinforcement learning to solve the global optimal decision, while the executor layer relies on deep reinforcement learning to output local dynamic control quantities, thus realizing hierarchical intelligent decision-making.

8. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 7, characterized in that, Both the overall leader and the group leaders adopt a dynamic selection mechanism, which can be rotated in real time according to cluster resource consumption, communication status, and task load to ensure the long-term stable operation of the multi-layer architecture.

9. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 8, characterized in that, The entire topology adjustment process adopts a smooth transition mode, ensuring that the tracking task is not interrupted during the adjustment process, thus guaranteeing the continuity of multi-agent tracking operations.

10. The switching multi-agent cooperative tracking control method based on deep reinforcement learning according to claim 9, characterized in that, After the preset holding time expires, the cluster and environment operation status are checked a second time. If the status recovers, the original strategy will be used in the long term. If the status continues to be abnormal, a second linkage causal analysis will be performed to complete the closed loop of full-scenario adaptive collaborative tracking control.

Images