Space-air-ground integrated multi-agent collaborative large model online training method and system

Abstract

The application discloses a space-air-ground integrated multi-agent collaborative large model online training method and system. Based on the dynamic generation of a distributed mixed precision quantization configuration table for the heterogeneity of multi-node resources, the application allocates appropriate parameter precision for different nodes; adopts a cross-node gradient residual distribution and local compensation strategy, only transmits the compressed gradient residual between nodes and accumulates the error locally, and regularly performs high-precision compensation update; supports the elastic division and dynamic scheduling of model parameters and learning tasks among multiple nodes, adjusts the task allocation and model segmentation in real time according to the performance and network status of each node; when a node is lost or the computing load is uneven, the system has fault tolerance and adaptive scheduling capability, and can automatically redistribute the model fragments or adjust the training process. The application makes full use of the computing power resources of the heterogeneous multi-agent, and realizes efficient collaborative training of a super large-scale model in a bandwidth limited and dynamically changing node scene.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, distributed machine learning and edge computing, and in particular to an online training method and system for a large-scale collaborative model integrating air, space, and ground. Background Technology

[0002] With the exponential growth of deep learning model parameters and data volume, training models in resource-constrained edge environments has become extremely difficult. Existing technologies, such as federated learning, allow multiple devices to train models independently and periodically aggregate parameters, but they typically assume that participating devices use the same precision and that the training process is synchronized, resulting in low efficiency in heterogeneous computing power and unstable network environments. Traditional data-parallel or model-parallel distributed training is primarily geared towards data centers or homogeneous clusters, requiring high-bandwidth interconnection between nodes, and is not optimized for the dynamic nature of space-air-ground distributed intelligent agent networks.

[0003] In integrated space-air-ground intelligent agent collaborative computing scenarios, multiple types of nodes often collaborate to execute intelligent tasks, such as satellites, drones, ground vehicles, and edge servers jointly training large-scale AI models. These nodes have varying computing capabilities, storage capacities, and communication bandwidths (i.e., significant heterogeneity), and network connections exhibit dynamic topologies and intermittency: for example, satellites can only communicate when passing overhead, drones may experience link interruptions due to flight distance, and ground nodes may experience performance bottlenecks due to load fluctuations. These challenges make traditional centralized training difficult to apply directly.

[0004] Enabling multi-agent systems to efficiently collaborate in training large models in such heterogeneous distributed environments has become a pressing issue. On the one hand, large models have a massive number of parameters and are computationally intensive, making it difficult for a single node to handle the entire training process. On the other hand, frequently transmitting complete gradients or model parameters between nodes consumes significant bandwidth and can even slow down the training process. Furthermore, if some nodes fail or become disconnected, or if their computational speed is significantly lower than that of other nodes, the overall training may be forced to stop or significantly slow down if fault tolerance and scheduling mechanisms are lacking. Therefore, current technologies lack a solution that can optimize mixed-precision training for heterogeneous resources while reducing communication overhead and adapting to dynamic changes in nodes. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings of existing technologies in distributed multi-agent collaborative training, and to provide a large model learning method and system that supports multi-node collaborative decision-making and flexible precision training scheduling. This invention aims to leverage the collaborative effect of distributed heterogeneous nodes to achieve efficient training of ultra-large models with lower communication costs, while ensuring model accuracy and robustness of the training process.

[0006] To achieve the above objectives, this invention provides an online training method for a large-scale collaborative model integrating air, space, and ground systems, comprising the following steps:

[0007] 1. Heterogeneous Resource Awareness and Configuration Table Generation: Obtain the resource status of each computing node in the distributed environment, including computing power (processor type, peak computing power), available memory capacity, network bandwidth and latency, as well as the sensitivity indicators of each layer's parameters to model accuracy. The specific calculation method is as follows:

[0008] S i,j =ω i *H j

[0009] Where S i,j H is used to quantify the degree of adaptation of node j to the i-th layer task of the model. j The hardware capability score is calculated as follows:

[0010]

[0011] Where P j The quantization value representing the processor type (e.g., GPU = 1, TPU = 0.8, CPU = 0.5, embedded = 0.3), A j / A max M represents the ratio of the current computing power to the system's maximum computing power. j / M max B represents the ratio of the total memory capacity to the maximum system memory. j / B max This represents the ratio of the bandwidth to the system's maximum bandwidth, with a weighting coefficient α+β+γ+δ=1.

[0012] ω i For model sensitivity.

[0013] Optionally, the model layer sensitivity is obtained by statistically analyzing the average norm of the gradients of each layer during historical training, and it is calculated as follows:

[0014]

[0015] in ω represents the gradient with respect to the parameters of the i-th layer under the k-th batch of data, where N is the number of batches. i To quantify the impact of the i-th layer on training, ω i The larger the value, the greater the impact of that layer on training performance.

[0016] Based on the above information, a "distributed hybrid precision quantization configuration table" is dynamically generated. This table defines the representation precision strategy for each part of the model on each node. For example, higher precision parameters (FP32 or FP16) are assigned to nodes with strong computing power, while lower precision parameters (such as INT8) are assigned to resource-constrained nodes. At the same time, higher precision is still used overall for critical model layers to maintain model performance. The configuration table can be updated in real time during training according to a preset period or event-triggered mechanism to adapt to changes in node performance or task requirements.

[0017] 2. Model partitioning and elastic task scheduling: Based on the configuration table and the real-time status of each node, the model parameters and training tasks are elastically partitioned and allocated among multiple nodes.

[0018] Specifically, the partitioning strategy can assign different layers or modules to different nodes based on the model structure (model parallelism), or distribute training data in batches to different nodes for parallel computation (data parallelism), or a combination of both; during the scheduling process, the current computing load and network status of each node are considered to ensure that the task allocation load is balanced.

[0019] The node load is calculated as follows:

[0020]

[0021] in This represents the computing power utilization rate of a node. Indicates memory utilization, B j (t) represents network latency, B base This represents the delay baseline value.

[0022] When a node has strong computing power or is idle, its allocated task share or model sharding can be dynamically increased; conversely, the task load or number of layers can be reduced to fully utilize the overall computing power without creating performance bottlenecks for individual nodes. This elastic scheduling mechanism supports adjustments to the model sharding scheme during operation, allowing nodes to join or leave collaborative computing as needed.

[0023] 3. Cross-node gradient residual distribution and local compensation update: In multi-node collaborative training iterations, each node independently performs forward and backward propagation calculations based on the aforementioned partition to obtain local gradient or parameter update values. For gradient information that needs to be synchronized between nodes, gradient data with full precision is not directly transmitted. Instead, it is first compressed according to the aforementioned quantization configuration to reduce precision (e.g., quantization by FP16 or INT8, or extraction of important gradients for sparsification), and the compression error (gradient residual) is calculated.

[0024] Optional, specifically quantified according to the following formula:

[0025]

[0026] in This represents the gradient (ideal value) theoretically calculated with full precision (e.g., FP32), where Δ is the quantization step size, calculated as follows:

[0027]

[0028] Where max(g) t ) and min(g t ) represent the maximum and minimum values ​​in the gradient vector, respectively. The quantization step size determines the resolution during mapping; a smaller step size results in narrower intervals between discrete levels, and the quantized value approximates the original value more precisely, but may require higher computational precision. Through the above quantization operation, the full-precision gradient g is obtained. t It maps to a discrete set of values, thereby using lower numerical precision (INT8) for updates and transmission, thus reducing computational and communication overhead.

[0029] Each node only sends the compressed gradient or model update increment to the relevant node or the central aggregation server, while storing the unsent gradient residuals in its local residual buffer. The receiving node updates its parameters based on the gradient increments from other nodes.

[0030] In addition, the system sets a gradient residual compensation trigger strategy: when the accumulated residual of a node exceeds a preset threshold or reaches a set synchronization period, a high-precision parameter correction step is triggered. This step can be achieved by each node sending the accumulated residual or resynchronizing key gradients with higher precision, injecting previously discarded subtle gradient information into the global model to eliminate error accumulation caused by low-precision transmission. Optionally, the compensation update obtains the compensation gradient G by summing the errors in the residual buffer in a decaying form. comp :

[0031]

[0032] Where T is the number of training steps within the compensation period, γ∈(0,1] is the decay factor, which assigns higher weights to the most recent errors, while older errors are considered in a decaying manner to control the weights of historical errors in the compensation update; the compensation gradient G comp This represents the weighted sum of the accumulated errors from all low-precision updates within the current period. Subsequently, full precision is applied to G... comp The parameters are injected into the model to correct for biases caused by low-precision quantization. After compensation, the residual buffer G is cleared. res This restarts the process of accumulating errors for the next cycle.

[0033] By combining gradient compression transmission with error compensation, the amount of cross-node communication data is significantly reduced while ensuring the convergence accuracy of distributed training.

[0034] 4. Fault Tolerance and Adaptive Cooperative Scheduling: During training, the online status and operational performance of each node are monitored in real time. When a node is detected to be disconnected, experiencing communication abnormalities, or exhibiting a significant decline in computational performance, a fault tolerance mechanism is activated: the faulty node is temporarily isolated, and the scheduling module is notified to adjust task allocation.

[0035] Specifically: for nodes that are temporarily disconnected, the model parameter updates they are responsible for can be compensated by other nodes based on the latest global parameters, or they can be synchronized after the node recovers; for nodes that are unavailable for a long time, the model slices and unfinished tasks they hold are redistributed to other available nodes for execution. If necessary, the accuracy requirements of the redistributed parts are reduced to adapt to the resource limitations of the receiving nodes, so as to ensure that training continues without interruption.

[0036] Similarly, when a node is found to be overloaded and become a training bottleneck, the scheduling module can, depending on the situation, transfer some tasks from that node to idle nodes, or dynamically adjust the quantization configuration to reduce the precision of that node to accelerate its computation speed and alleviate overall imbalance. Simultaneously, this invention supports the dynamic addition of new nodes: when a new computing node joins the distributed system, the configuration table is updated and the model splitting and task allocation are adjusted accordingly, allowing the new node to immediately engage in collaborative training to further accelerate convergence. The above adaptive collaborative scheduling strategy improves the system's adaptability and robustness to uncertain environments, enabling it to stably complete training tasks even with changes in node size or capabilities.

[0037] The present invention also provides a training system for implementing the above method. The system consists of multiple heterogeneous computing nodes, each node having a local training unit, and the following component functions are implemented by a central coordinator or distributed control logic:

[0038] Resource monitoring and quantization configuration module: This module collects real-time resource information and model layer sensitivity indicators from each node, runs a predetermined algorithm to generate the distributed hybrid precision quantization configuration table, and distributes the configuration to each node for execution. This module supports dynamic updates to the configuration table, ensuring that the precision allocation strategy always matches the node capabilities and task requirements.

[0039] The task partitioning and scheduling module is used to partition and allocate model parameters and training data tasks among nodes according to the configuration table, and continuously monitor the node status during training to adaptively adjust the scheduling scheme. This module can re-plan the distribution of the model and the allocation of computing tasks in real time based on the addition, removal, or performance changes of nodes, ensuring load balancing and efficient utilization of multi-node collaboration.

[0040] The gradient communication and error compensation module controls gradient exchange and merging across nodes. In each training iteration, this module receives gradient increments (after quantization and compression) uploaded by each node, performs necessary aggregation calculations to update the global model parameters, and broadcasts or sends the update results to relevant nodes. Simultaneously, it maintains the gradient residual buffers of each node, triggers high-precision compensation update operations according to a preset strategy, and guides each node to perform residual correction on its local model, thereby improving the accuracy of distributed training.

[0041] The fault-tolerance control module coordinates adjustments to the training process when a node fails, goes offline, or experiences performance anomalies. This module detects node timeouts or error events, promptly notifying the task scheduling module to reassign the model and data tasks assigned to that node. When a failed node regains connectivity, it can decide whether and how to reinstate it for training. For example, when a UAV node temporarily loses connection, the fault-tolerance module instructs other nodes to temporarily take over its computation; once it recovers, the corresponding tasks are returned or the latest model parameters are synchronized to catch up with the global progress. The fault-tolerance control module also records the point of failure and the training status so that in extreme cases (such as coordinating a server restart), training can resume from the most recent checkpoint, ensuring system robustness.

[0042] The above-described methods, steps, and system modules work together to form a large-scale collaborative training scheme suitable for heterogeneous multi-agent environments. This invention can be effectively deployed in an integrated air-space-ground agent network: for example, in the training of a satellite-UAV-ground collaborative target detection model, each node performs its specific function according to its capabilities and efficiently shares learning results through the method of this invention, thereby significantly shortening model convergence time and improving overall performance.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Fully utilize heterogeneous computing resources to improve training efficiency: By dynamically generating a distributed mixed-precision quantization configuration table, this invention can allocate appropriate computational precision and model fragments based on the hardware characteristics of different nodes, enabling each node to work efficiently within its capabilities. For example, nodes with strong GPU computing power undertake the main computations and can use higher precision, while embedded devices participate in some model computations with lower precision, thereby maximizing overall performance. Compared to methods of isolated training on a single edge device or fixed-precision training on each node, this invention significantly improves the computing power utilization of multi-agent clusters and shortens training time.

[0045] 2. Reduced communication overhead and efficient collaboration: By adopting a strategy combining gradient residual distribution and local compensation, the amount of data redundancy in cross-node communication is significantly reduced. Each node only needs to transmit compressed, important gradient information, retaining subtle gradient changes for local accumulation, thus avoiding frequent transmission of massive parameter updates.

[0046] 3. Ensuring Model Accuracy and Stable Convergence: Despite employing low-precision computation and compressed communication, this invention avoids error accumulation caused by reduced accuracy through local residual compensation and periodic high-precision correction, enabling model training to achieve convergence accuracy approximately equivalent to full-precision training. Simultaneously, dynamic quantization allocates precision based on the sensitivity of each part of the model, ensuring that the computational precision of key layers is not excessively weakened. Therefore, it maintains stable and reliable model performance even under heterogeneous node co-training.

[0047] 4. Flexible fault tolerance enhances system robustness: This invention supports dynamic node addition, removal, and fault recovery. When a node fails or becomes disconnected, the system can quickly adjust the allocation of models and tasks, ensuring uninterrupted or only briefly paused training without completely halting it. Compared to traditional distributed training lacking fault tolerance mechanisms, this invention can adapt to the frequent node fluctuations in space-air-ground multi-agent networks, guaranteeing the continuous progress of training tasks during long-term operation.

[0048] 5. Expanding Application Scenarios and Achieving Collaborative Intelligence: This invention is particularly suitable for scenarios such as integrated air-space-ground systems and edge computing clusters. It can be deployed in multi-agent collaborative environments such as UAV swarms, vehicle-to-everything (V2X) systems, and remote sensing satellites. Through this solution, multiple agents can jointly train and share AI models, such as jointly learning tasks like target recognition and environmental perception. The models can be updated in real time on-site to adapt to environmental changes, significantly improving the autonomous learning capability and collaborative decision-making level of distributed intelligent systems, and has significant industrial application value.

[0049] In summary, the multi-agent collaborative large-scale model training method and system provided by this invention enables efficient and reliable training of ultra-large-scale models in heterogeneous distributed environments, filling a gap in the existing technology in this field. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the multi-agent collaborative large model training system architecture according to an embodiment of the present invention. It shows a distributed training system composed of heterogeneous nodes such as satellites, drones, and ground vehicles, as well as a central coordination unit. The interaction relationship between the modules is shown in the figure.

[0051] Figure 2 This is a flowchart illustrating the training task scheduling process of an embodiment of the present invention, showing the steps of resource awareness, configuration generation, model partitioning, gradient communication, and fault-tolerant scheduling.

[0052] Figure 3 This is a schematic diagram of a typical collaborative scenario in an embodiment of the present invention. Taking the collaborative training of a target recognition model by satellites, drones and ground vehicles as an example, it illustrates the workflow and data interaction of multi-agent collaborative training. Detailed Implementation

[0053] To better understand the present invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be emphasized that the following embodiments are intended to illustrate the technical principles and preferred implementations of the present invention, and are not intended to limit the scope of protection of the present invention. Based on the ideas of the present invention, those skilled in the art can make various modifications and substitutions in specific applications, and all such modifications and substitutions, as long as they do not deviate from the principles of the present invention, should fall within the scope of protection of the present invention.

[0054] Example: Training of a multi-agent target recognition model for air-space-ground collaborative operation

[0055] This embodiment combines Figure 1 and Figure 3 The diagram illustrates a typical application scenario for collaborative training of a large-scale target recognition model using satellites, drones, and ground vehicles. Assuming the model is a deep neural network with billions of parameters (e.g., an improved Transformer model), incremental training is required in a distributed multi-agent environment to identify newly emerging target types. The system hardware environment includes: one low-Earth orbit remote sensing satellite operating at high altitude, carrying an embedded AI module with limited computing power; multiple drones equipped with small GPU accelerators for edge computing; several ground vehicles or edge servers with strong GPU / CPU computing capabilities; and a self-organizing network formed by nodes via wireless links and satellite relays.

[0056] 1. Heterogeneous resource awareness and allocation:

[0057] like Figure 2 As shown, the central coordination unit first collects the hardware capabilities and status information of each node. Satellite nodes, due to power consumption limitations, only have low-power embedded computing units, suitable for performing low-precision inference and a small number of simple updates; drone nodes have moderate GPU computing power and limited memory; ground nodes have the strongest computing and storage capabilities and a relatively stable power supply. The central coordination unit also calculates the impact of the parameters of each layer of the current training model on recognition accuracy, for example, by evaluating the sensitivity of each layer based on past gradient magnitudes. The specific calculation method is as follows:

[0058] S i,j =ω i *H j

[0059] Where H j The hardware capability score is calculated as follows:

[0060]

[0061] Where P j The quantization value representing the processor type (e.g., GPU = 1, TPU = 0.8, CPU = 0.5, embedded = 0.3), A j / A max M represents the ratio of accelerator computing power to the system's maximum computing power. j / M max B represents the ratio of the total memory capacity to the maximum system memory. j / B max This represents the ratio of the bandwidth to the system's maximum bandwidth, with a weighting coefficient α+β+γ+δ=1.

[0062] ω i For model sensitivity, optional, the model layer sensitivity is obtained by statistically analyzing the average norm of the gradients of each layer during historical training, and its calculation method is as follows:

[0063]

[0064] in This represents the gradient with respect to the parameters of the i-th layer under the k-th batch of data, where N is the number of statistical batches, and ω i To quantify the impact of the i-th layer on training, ω i The larger the value, the greater the impact of that layer on training performance.

[0065] Based on this information, the system generates a distributed mixed-precision quantization configuration table, as shown below:

[0066]

[0067] For example, key modules such as the convolutional feature extraction layer and the Transformer attention layer of the model are run on ground nodes with FP16 precision, while some computationally intensive secondary layers (such as embedding layers and normalization layers) are quantized to INT8 and processed by satellite and UAV nodes. At the same time, the gradient precision used by each node during local updates is limited. For example, the gradient of satellite nodes is only represented by 8 bits, while ground nodes can use 16 bits precision.

[0068] The configuration table also specifies the range of model layers each node is responsible for, in order to clarify the model partitioning scheme. This configuration process considers the communication bandwidth between nodes: since satellite links have the narrowest bandwidth, the configuration table tends to reduce the amount of data that satellites need to transmit. For example, it allows satellites to only provide image data and perform simple forward computations, while complex backward gradient calculations are delegated to downstream nodes. After the configuration table is generated, it is distributed to each node, and the collaborative parameter update strategy for this training is determined accordingly.

[0069] 2. Model Partitioning and Task Scheduling:

[0070] like Figure 2 As shown, at the beginning of the training phase, the model parameters are distributed to each node according to the scheme specified in the configuration table: the ground server loads and maintains a complete copy of the model parameters for summary updates; at the same time, some network layer weights are distributed to the UAV and satellite, enabling them to undertake some forward computation responsibilities.

[0071] In terms of training data, satellites continuously capture high-altitude images as new training samples; drones also use their onboard cameras to collect regional photos and participate in training.

[0072] The coordination unit schedules the execution order and pace of training tasks based on node performance and current load: satellite nodes preprocess each frame of image and perform preliminary feature extraction, then send intermediate features to ground or UAV nodes; ground nodes are responsible for the backpropagation calculation and parameter updates of the complete model, but to reduce the burden, some gradient calculation tasks are shared by UAV nodes (e.g., UAVs calculate gradients for the part of the model they are responsible for and send back the results). If a UAV experiences computational resource constraints due to performing other tasks, the scheduling module will reduce the number of task batches allocated to it and directly assign more data to ground nodes for processing; conversely, when ground nodes experience momentary overload (e.g., needing to process data from multiple UAVs simultaneously), some UAVs can temporarily complete several training steps independently before merging the results. Through this flexible scheduling, the computational load of each node remains relatively balanced, and overall hardware resources are utilized efficiently.

[0073] 3. Gradient residual transfer and local compensation:

[0074] In each training iteration, the ground node acts as the master node and collaborates with other cooperating nodes to complete the forward and backward computation of the model.

[0075] Specifically, the satellite node sends the preprocessed image features to the UAV and ground node for computation in the middle layer of the model; the UAV node continues to perform forward computation of part of the model layer on the received features and generates local gradients; finally, the ground node completes the forward inference of the remaining layers and summarizes the gradient information from the UAV for backpropagation of global error.

[0076] To reduce communication overhead, drones and satellites quantize and compress data according to a configuration table before transmitting gradients or intermediate activations. For example, floating-point gradient values ​​are reduced to 8 bits, and only changes in significance coefficients are transmitted. Optionally, quantization can be performed using the following formula:

[0077]

[0078] in This represents the gradient (ideal value) theoretically calculated with full precision (e.g., FP32), where Δ is the quantization step size, calculated as follows:

[0079]

[0080] Where max(g) t ) and min(g t ) represent the maximum and minimum values ​​in the gradient vector, respectively. The quantization step size determines the resolution during mapping. The smaller the step size, the narrower the interval between discrete levels, and the more precisely the quantized value can approach the original value, but at the same time, higher computational precision may be required.

[0081] Through the above quantization operations, the full-precision gradient g is obtained. t It maps to a discrete set of values, thereby using lower numerical precision (INT8) for updates and transmission, thus reducing computational and communication overhead.

[0082] On the ground node side, the gradient communication and error compensation module receives the compressed gradient and decodes and aggregates it. For each gradient tensor from the UAV, this module compares the approximate gradient with the ideal full-precision gradient (which can be obtained by recalculating locally at the ground node or by accumulating previous residual estimates), and the difference is accumulated in the residual buffer of the corresponding UAV node. The residual accumulation process is as follows:

[0083]

[0084] in For the ideal full-precision gradient, As an approximate gradient, G res This represents the accumulated error in the residual buffer.

[0085] Since satellite nodes only participate in the forward process, the feature maps they provide may be transmitted using low-precision compression. Ground nodes also record reconstruction errors. After several iterations, when G in the residual buffer of a certain UAV node... res When the error norm exceeds a threshold, the system will notify the drone to improve the gradient transmission accuracy in the next training cycle or directly transmit the accumulated residual for a compensation update. Optionally, the compensation update is performed by summing the errors in the residual buffer in a decaying form to obtain the compensation gradient G. comp :

[0086]

[0087] Where T is the number of training steps within the compensation period, γ∈(0,1] is the decay factor, which assigns higher weights to the most recent errors, while older errors are considered in a decaying manner to control the weights of historical errors in the compensation update; the compensation gradient G compThis represents the weighted sum of the accumulated errors from all low-precision updates within the current period. Subsequently, full precision is applied to G... comp The parameters are injected into the model to correct for biases caused by low-precision quantization. After compensation, the residual buffer G is cleared. res This restarts the process of accumulating errors for the next cycle.

[0088] Meanwhile, ground nodes periodically use high-precision calculations to correct for previously accumulated errors when updating their own parameters. For example, after every 10 low-precision synchronizations, a ground node initiates a full-precision All-reduce operation to synchronize the full-precision values ​​of key gradients across all nodes, ensuring that the model parameters of each node remain consistent with full-precision training. Through this mechanism of multiple local low-precision updates combined with occasional high-precision corrections, this embodiment achieves convergence results comparable to centralized full-precision training while significantly reducing communication overhead.

[0089] 4. Fault tolerance mechanism and adaptive adjustment

[0090] During the 50th training epoch, it is assumed that a drone temporarily loses contact due to flying out of communication range. At this time, the fault tolerance control module detects that the node has timed out and failed to upload gradients, and immediately initiates the fault tolerance process: First, the unreceived data is marked as missing, and the ground node is instructed to skip the gradient update of the part that the drone is responsible for, and approximate it with the model parameters from the most recent time that the node participated (i.e., those parameters are not updated for the time being); at the same time, the task scheduling module reduces the subsequent data allocation to the drone, and sends more newly acquired images to other drones and ground nodes that are still online for processing.

[0091] After several rounds of training, the lost drone regained communication and attempted to rejoin the training. The system then sent it a snapshot of the latest model parameters and reallocated an appropriate number of tasks based on its performance at the time of recovery. If the model underwent significant changes during the period the drone missed the training, the system can also schedule an additional synchronization to ensure its model state is consistent with the global model before resuming normal training.

[0092] For example, if during training it is found that satellite nodes consistently become the speed bottleneck due to hardware limitations—taking significantly longer to process each batch of data than other nodes—the system will perform adaptive adjustments: on the one hand, reducing the number of model layers that satellites need to participate in (potentially transferring some of the first few layers of the network, which would normally be performed by satellites, to UAVs or ground nodes); on the other hand, further reducing the precision of computations performed by satellites during configuration table updates (e.g., using simpler arithmetic operations or higher compression ratios) to speed up processing. Through these fault-tolerant and adjustment measures, the entire multi-agent system maintains the continuity and efficiency of collaborative work even with dynamic changes in nodes. Practice has proven that the method of this invention can significantly improve training robustness in this air-to-ground collaborative training scenario: even if a single UAV node goes offline for an extended period, the remaining nodes can still complete model training relatively smoothly; when they recover, only a short synchronization is needed to catch up with the progress, without affecting the final model accuracy.

[0093] In summary, this embodiment verifies the effectiveness of the present invention in distributed multi-agent collaborative training. In a heterogeneous air-space-ground network environment, using the method of this invention, each node shares the model training task according to its own capabilities, avoiding single-point bottlenecks and minimizing communication costs, thus achieving efficient incremental training of ultra-large-scale target recognition models. This scheme is also applicable to other similar multi-agent collaborative learning scenarios (such as joint training of autonomous driving models in vehicle-to-everything (V2X) networks and multi-robot collaborative learning in smart factories), and can promote better learning results in distributed intelligent systems.

[0094] The above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. More other equivalent embodiments may be included without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An online training method for a large-scale collaborative model integrating air, space, and ground systems, characterized in that: Includes the following steps: (1) Obtain the resource information and model parameter sensitivity index of each computing node in the distributed environment, and dynamically generate a distributed mixed precision quantization configuration table based on the heterogeneity of node computing power. The configuration table defines the computing precision allocation strategy of each part of the model on each node. (2) Based on the configuration table, the parameters of the model to be trained and the training tasks are flexibly divided and allocated among multiple nodes, and the task scheduling is dynamically adjusted according to the node status during the training process. (3) Each node independently executes the forward inference and gradient calculation of the assigned model part. When cross-node synchronization is required, only the quantized and compressed gradient residual information is transmitted, and the gradient error of the untransmitted part is stored locally for accumulation. (4) Periodically or when the accumulated residual exceeds the preset threshold, trigger a high-precision model update compensation step, including exchanging the accumulated gradient residuals of each node or improving the gradient synchronization accuracy, in order to correct the model error caused by low-precision calculation and transmission. (5) When a node is detected to be disconnected or has abnormal performance, the fault tolerance mechanism is automatically activated, the model parameters and task allocation among the remaining nodes are readjusted, and the faulty node is reconnected and synchronized with the latest model state before being rejoined for training, thereby ensuring that the training process continues to converge. The distributed mixed-precision quantization configuration table generated in step (1) comprehensively considers the processor type, hardware acceleration capability, memory capacity, network bandwidth of each node, and the sensitivity of each layer of the model to the final accuracy. The specific calculation method is as follows: in Used to quantify the degree of adaptation of node j to perform the task of layer i in the model. This indicates the hardware capability score. For model sensitivity; For model layers with high computational sensitivity, higher computational precision is assigned to most nodes, while for parts with low sensitivity or high computational overhead, low-precision quantization is used and the parts are assigned to resource-constrained nodes for execution, in order to balance model accuracy and training speed.

2. The method according to claim 1, characterized in that, The elastic partitioning and task scheduling in step (2) includes: continuously monitoring the computational load and communication latency of each node during training; when a node is found to be in a performance bottleneck, reducing the subsequent training data or model shards allocated to that node and transferring its tasks to other idle nodes; when a node is detected to have remaining computing power or a new node is added, increasing the model layers or data batches it participates in and updating the configuration table accordingly to keep the utilization of all nodes balanced.

3. The method according to claim 1, characterized in that, The gradient residual information transmitted by each node in step (3) includes: performing low-bit-width quantization or sparsification on the gradient vector calculated by the node, and extracting only the important gradient values ​​and their indices to send to the target node or parameter aggregation server.

4. The method according to claim 1, characterized in that, The high-precision model update compensation step in step (4) includes: each node or central coordination unit detects the norm of the accumulated error in the local residual buffer. When it exceeds a preset threshold or reaches a predetermined communication period, all nodes are triggered to exchange their respective key gradients or residual information with the first precision. Then, the model parameters are updated globally with the first precision, and the residual buffers of each node are cleared to zero, thereby eliminating the error accumulation caused by low-precision synchronization.

5. The method according to claim 1, characterized in that, The fault tolerance mechanism in step (5) specifically includes: when a node loses connection for more than a predetermined time threshold, the node is marked as failed and stops waiting for its gradient, and other nodes are used to estimate and replace the missing gradient or skip the update; at the same time, the scheduling module reassigns the model parameters that the node is responsible for to still connected nodes for temporary management to prevent training interruption due to single node failure; after the failed node reconnects, the system incrementally synchronizes the missed model updates, restores its model state to the current version, and then releases it from temporary management so that it can continue to participate in the remaining training.

6. A training system for implementing the method of any one of claims 1 to 5, characterized in that, The training system includes multiple heterogeneous computing nodes and a communication network connecting the nodes. It is configured with a central coordination unit or a distributed control module to perform the functions of the following units: The resource monitoring and precision configuration unit is used to collect computing resource information and model layer importance indicators of each node, generate a distributed hybrid precision quantization configuration table, and distribute it to each node. The task partitioning and scheduling unit is used to partition and allocate model parameters and training tasks among nodes according to the configuration table, and dynamically adjust the task load distribution of nodes during the training process. The gradient communication and compensation unit is used to transmit gradient information generated during training between nodes and aggregate and update model parameters, maintain gradient residual buffers of each node and trigger high-precision compensation updates to ensure training convergence accuracy. The fault-tolerant control unit monitors the online status of nodes, coordinates task reassignment and model data migration when nodes are offline or experience performance abnormalities, and synchronizes model parameters when nodes recover so that they can be re-added to training.

7. The system according to claim 6, characterized in that: The resource monitoring and precision configuration unit includes a heterogeneous resource perception module and a precision allocation decision module. The former is used to detect the hardware type, computing power performance and communication bandwidth of each node, while the latter calculates the precision configuration scheme of each node based on the detection results and the preset model layer sensitivity rules, and forms the hybrid precision quantization configuration table.

8. The system according to claim 6, characterized in that: The gradient communication and compensation unit includes a compression coding submodule and an error compensation submodule. The compression coding submodule uses gradient quantization, truncation or sparse coding algorithms to compress and aggregate the gradient information uploaded by each node. The error compensation submodule periodically triggers full-precision gradient synchronization operation or notifies relevant nodes to send accumulated residuals based on the gradient residual buffer content of each node, which is used to correct model parameters.

9. The system according to claim 6, characterized in that: The fault-tolerant control unit is equipped with a fault detection module and a scheduling interaction module. The fault detection module determines the online status of the node through a timeout mechanism. When a node is detected to be offline, it sends a reconfiguration request to the scheduling interaction module. The scheduling interaction module responds to the request, adjusts the mapping of the model and tasks on the remaining nodes, and coordinates the update of its model parameter replicas after the faulty node is restored to online status, so that its status is consistent with the current global model.

Images