A task vector-based distributed large model asynchronous training method and device

Abstract

The application relates to the technical field of deep learning engineering optimization, and discloses a distributed large model asynchronous training method and device based on a task vector, which is applied to a distributed network collaborative node, and the method comprises the following steps: deploying a unified initial global model and constructing an initial total weight vector by each node, independently iteratively training in a collaborative cycle based on a local heterogeneous data set, solving a local task vector in combination with the initial weight vector and the current cycle, storing in a background buffer and broadcasting; collecting external task vectors to form a task vector list, updating the weight and the local model by means of a dynamic alignment mechanism, and cyclically executing until a training termination condition is met, and merging optimization models of all nodes to obtain a final large model. The task vector is broadcasted in an asynchronous mode relying on the buffer, communication transmission is hidden in local calculation, the serial constraint of calculation and communication is released, the blocking problem of computing power waiting for a network is eliminated, the training throughput is improved, and the large model collaborative training in a low-bandwidth multi-center scene is adapted.

Description

Technical Field

[0001] This invention relates to the field of deep learning engineering optimization technology, specifically to a distributed large-scale asynchronous training method and apparatus based on task vectors. Background Technology

[0002] With the rapid expansion of large language model parameter scale, multi-center collaborative training or joint fine-tuning using computing power and private data distributed across heterogeneous nodes has become a core development trend in the industry. Standard data-parallel algorithms require gradient synchronization at each training step, which incurs excessively high communication costs due to the much larger bandwidth across nodes compared to intra-node bandwidth.

[0003] Current mainstream technologies are periodically synchronized federated learning (e.g., the classic FedAvg algorithm) or distributed learning paradigms (e.g., DiLiCo). These technologies change the step-by-step synchronization strategy in standard data parallelism. At the end of each collaborative cycle (containing multiple training steps), each computing node synchronizes its locally trained complete model parameters or full parameter differences (gradients / fine-tuning weights) through a cross-center network in a peer-to-peer or centralized manner. After receiving aggregated features from all external nodes, it updates its local model and starts the next cycle of computation. While this approach reduces the number of synchronizations, it still involves blocking transmission and has high communication costs. Some technologies incorporate a degree of asynchronous transmission, but still require blocking and waiting. Traditional parameter, weight, or gradient synchronization has an absolute strong temporal dependency. Since the next step of local optimization (e.g., forward propagation) strictly depends on the parameter results finally aggregated in the current step or cycle, the serial mode of computation-waiting for communication-re-computation leads to expensive computing resources being idle for a long time, severely reducing the overall throughput of distributed collaborative training. Summary of the Invention

[0004] This invention provides a method and apparatus for asynchronous training of distributed large models based on task vectors, in order to solve the problems of eliminating network communication latency and improving the utilization of computing hardware during cross-center training of distributed large models.

[0005] In a first aspect, the present invention provides a distributed large-scale asynchronous training method based on task vectors. The method is applied to any cooperating node in a distributed network and includes: Deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the current cycle total weight vector; The local task vector of the corresponding collaborative node is calculated based on the current period total weight vector and the initial global model total weight vector. The local task vector is stored in the background communication sliding buffer and broadcast to other collaborative nodes. At the same time, the process of independently iterating and updating the model weight vector using the local heterogeneous dataset is not interrupted. Receive external task vectors broadcast by other cooperating nodes, construct a task vector list based on local task vectors and external task vectors, update the total weight vector of the corresponding cooperating node using a dynamic alignment mechanism, update the global model of the corresponding cooperating node based on the updated total weight vector, and obtain the optimized global model. The process of repeatedly calculating the local task vectors of the collaborative nodes and updating the global model continues until the collaborative training termination condition is met. The optimized global models of each collaborative node are then merged to obtain the final global large model.

[0006] The distributed large-scale asynchronous training method based on task vectors provided by this invention hides network transmission in the local computing process by storing task vectors in a background communication sliding buffer and broadcasting them. This breaks the serial dependency between communication and computing in traditional distributed training, eliminates blocking events where computing units wait for cross-center network transmission, enables computing hardware to run at full capacity, maximizes the masking of communication overhead, and significantly improves the system throughput of collaborative training. It provides high engineering feasibility for multi-center large-scale model collaborative training in low-bandwidth scenarios.

[0007] In one alternative implementation, the same initial global model as other collaborating nodes is deployed, and the initial global model total weight vector is constructed, including: Obtain the initial model weights of the initial global model, and determine the feature tensor based on the initial model weights; According to the preset fixed tensor order, all feature tensors in the initial global model are flattened into one-dimensional tensors respectively; By sequentially concatenating all one-dimensional tensors, the initial global model total weight vector is constructed.

[0008] The distributed large-scale asynchronous training method based on task vectors provided by this invention unifies the initial models of each collaborative node, flattens and concatenates all feature tensors in a fixed order to obtain a global weight vector, standardizes the tensor arrangement format, realizes standardized and unified representation of model parameters, facilitates cross-node synchronization, parameter comparison and aggregation operations, reduces the difficulty of data alignment in distributed collaborative training, and improves the computational stability and processing efficiency of model interaction and parameter fusion.

[0009] In one optional implementation, the local task vector of the corresponding collaborative node is calculated based on the current periodic total weight vector and the initial global model total weight vector. The local task vector is then stored in the background communication sliding buffer and broadcast to other collaborative nodes, including: According to the preset fixed tensor order, the current model weights at the end of the current cycle are flattened into one-dimensional tensors and sequentially concatenated and spliced ​​to obtain the total weight vector of the current cycle. Subtract the current cycle total weight vector from the initial global model total weight vector element by element to calculate the local task vector of the corresponding collaborative node; The local task vector is stored in the background communication sliding buffer. The local task vector is packaged and broadcast to other cooperating nodes using the network protocol stack. At the same time, the front-end computing unit performs local model training for the next communication and cooperation cycle.

[0010] The distributed large-scale asynchronous training method based on task vectors provided by this invention generates weight vectors through a unified tensor flattening and splicing rule. The vectors are subtracted element by element to quickly obtain the locally updated task vector. Data is cached by a sliding buffer and broadcast synchronously. At the same time, the front-end and back-end execute communication transmission and local model training in parallel, thereby decoupling the computation and communication tasks, effectively shortening the collaboration cycle time, ensuring the synchronization and consistency of parameters of each node, and improving the overall running efficiency of distributed collaborative training.

[0011] In one optional implementation, a task vector list is constructed based on local task vectors and external task vectors, and the total weight vector of the corresponding collaborative nodes is updated using a dynamic alignment mechanism, including: Update the corresponding local task vector using the current task vector of the collaborating node, and construct a task vector list based on the updated local task vector and the external task vector; Extract multiple elements at the same feature position from the task vector list and construct a local scalar set; Retrieve the element with the largest absolute value in the local scalar set, and use the sign of the element with the largest absolute value as the dominant feature sign; The effective elements in the local scalar set are determined based on the dominant feature symbols, and the fusion elements at the corresponding feature positions are determined based on all effective elements. The global alignment task vector is determined based on the fusion elements at each feature location. The global alignment task vector is then added element by element to the initial global model total weight vector, and the updated total weight vector is obtained by reverse reconstruction.

[0012] The distributed large-scale asynchronous training method based on task vectors provided by this invention summarizes local and external task vectors to form a list, selects dominant symbols according to feature positions to divide effective elements and completes fusion, and uses a dynamic alignment mechanism to unify the update direction of multiple nodes and avoid conflicts in parameter updates of multiple nodes. By reconstructing the model inversely with the fused vector and initial weights, the training information of each node is taken into account, the accuracy of cross-node parameter fusion is improved, and the consistency of distributed collaborative model updates is guaranteed.

[0013] In one alternative implementation, the method further includes: Calculate the proportion of all valid elements to all elements at the corresponding feature position, and use it as a global manifold conflict metric. Adjust the number of steps in the next collaborative communication cycle based on the global manifold conflict metric.

[0014] The distributed large-scale asynchronous training method based on task vectors provided by this invention obtains a global manifold conflict metric by statistically analyzing the proportion of effective elements, quantifies the degree of conflict in multi-node parameter updates, and adaptively adjusts the number of steps in the next collaborative cycle based on this metric, thereby achieving a dynamic adaptive balance of training rhythm and taking into account both model fusion accuracy and overall collaborative training efficiency.

[0015] In one alternative implementation, adjusting the number of steps in the next cooperative communication cycle based on a global manifold conflict metric includes: If the global manifold conflict metric is greater than the preset threshold, the number of steps in the next collaborative communication cycle will be reduced by the first preset step size. If the global manifold conflict metric is not greater than the preset threshold, then the number of steps in the next collaborative communication cycle will be increased by the second preset step size.

[0016] The distributed large-scale asynchronous training method based on task vectors provided by this invention divides parameter conflict levels with a preset threshold. In low-conflict scenarios, the number of synchronization cycle steps is reduced to accelerate the convergence of multi-node parameter fusion; in high-conflict scenarios, the number of local training steps is increased to fully explore local data and reduce update divergence.

[0017] In one alternative implementation, the method further includes: The updated total weight vector is reshaped to obtain the updated feature tensors of the global model, and the updated feature tensors are used to cover the model parameters at the current time step. If the optimizer used for local model training has a momentum mechanism, the historical momentum information currently cached by the corresponding optimizer will be forcibly reset to a zero matrix.

[0018] The distributed large-scale asynchronous training method based on task vectors provided by this invention restores the feature tensors of each layer of the model by reshaping the vectors, thus facilitating local model parameter updates. For the optimizer with momentum, historical momentum is reset to clear the gradient accumulation interference left by the iteration of new and old weights, avoid the continuous impact of historical conflict update information on subsequent training, ensure that the gradient update of the new round of collaborative training is accurate and reliable, and improve the stability of model parameter iteration.

[0019] Secondly, the present invention provides a distributed large-scale asynchronous training device based on task vectors. The device is applied to any cooperating node in a distributed network, and the device includes: The local model independent training module is used to deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the total weight vector of the current cycle. The task vector calculation module is used to calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer and broadcast it to other collaborative nodes, and at the same time, the process of independently iterating and updating the model weight vector using the local heterogeneous dataset does not stop. The global model local optimization module is used to receive external task vectors broadcast by other collaborative nodes, construct a task vector list based on the local task vectors and external task vectors, update the total weight vector of the corresponding collaborative node using a dynamic alignment mechanism, and update the global model of the corresponding collaborative node based on the updated total weight vector to obtain the optimized global model. The model merging module is used to repeatedly calculate the local task vectors of the collaborative nodes to update the global model until the collaborative training termination condition is met. Then, the optimized global models of each collaborative node are merged to obtain the final global large model.

[0020] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0021] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the first process of a distributed large model asynchronous training method based on task vectors according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the second process of the distributed large model asynchronous training method based on task vectors according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the asynchronous training process using two cooperating nodes as an example in the distributed large model asynchronous training method based on task vectors according to an embodiment of the present invention. Figure 4 This is a complete flowchart of a specific embodiment of the distributed large model asynchronous training method based on task vectors according to an embodiment of the present invention; Figure 5 This is a structural block diagram of a distributed large-scale asynchronous training device based on task vectors according to an embodiment of the present invention. Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0026] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0027] This invention provides a distributed large-scale asynchronous training method based on task vectors. By hiding the network transmission of task vectors in the local computing process, it achieves the effect of eliminating the blocking of computing units waiting for cross-center network transmission.

[0028] According to an embodiment of the present invention, a method for asynchronous training of a distributed large model based on task vectors is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a distributed large-scale asynchronous training method based on task vectors, which can be used in any cooperating node in the aforementioned computer system or distributed network. Figure 2 This is a flowchart of a distributed large-scale asynchronous training method based on task vectors according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Deploy the same initial global model as other collaborative nodes, construct the total weight vector of the initial global model, and independently iterate and update the total weight vector of the initial global model using the local heterogeneous dataset within the current collaborative communication cycle to obtain the total weight vector of the current cycle.

[0030] Specifically, on n (n>1) cooperating nodes in a distributed network (either homogeneous or heterogeneous environments are acceptable), the same initial global model is deployed, and the total weight vector of the initial global model is constructed based on the parameters of the initial global model. At the same time, initialize the number of steps in the current cooperative communication cycle to the initial value. .

[0031] Each collaborative node utilizes its local heterogeneous dataset to independently execute forward propagation, backward propagation, and parameter updates, starting from the first iteration of the current cycle, until the completion of the [number of iterations]. The process involves iterative steps. During this period, no network communication occurs between nodes, and the optimization of the local training trajectory is entirely dominated by the foreground computing core.

[0032] When the local iteration reaches the current collaborative communication cycle... During each step, each collaborative node triggers a task vector extraction mechanism, using the global model weight vector at the end of the current collaborative communication cycle as the total weight vector for the current cycle. .

[0033] Step S202: Calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer and broadcast it to other collaborative nodes, and at the same time, the process of independently iterating and updating the model weight vector using the local heterogeneous dataset is not interrupted.

[0034] Specifically, based on the current period's total weight vector and the initial global model's total weight vector... The local task vector of the corresponding node is calculated to represent the phased macroscopic optimization trajectory of the corresponding node. The local task vector is written into the preset background communication sliding buffer in the video memory, and the control pointer of the foreground computing unit is released immediately to achieve data isolation. While storing the local task vector in the background communication buffer and broadcasting it to other cooperating nodes, the foreground computing unit maintains the iterative update process of the local model to improve efficiency.

[0035] After the local task vectors of each cooperating node are written into the background communication sliding buffer, the system's resident background communication thread is triggered to start, asynchronously reading the total local task vectors from the background communication sliding buffer in a non-blocking manner, and broadcasting them to the other n-1 external cooperating nodes.

[0036] Step S203: Receive external task vectors broadcast by other collaborative nodes, construct a task vector list based on local task vectors and external task vectors, update the total weight vector of the corresponding collaborative node using a dynamic alignment mechanism, and update the global model of the corresponding collaborative node based on the updated total weight vector to obtain the optimized global model.

[0037] Specifically, based on the background communication thread, external task vectors from all other external nodes are received. Once all vectors are received, they are securely written to the local receive buffer. During the background communication process, the foreground computing unit continuously trains the model. After all external task vectors are written to the local receive buffer, the foreground training is paused, and the dynamic alignment mechanism is invoked to inject external collaborative knowledge into the global model of the current collaborative node. The total weight vector of the corresponding collaborative node is updated, and the global model of the corresponding collaborative node is updated based on the updated total weight vector to obtain the optimized global model.

[0038] Step S204: Repeat the process of calculating the local task vector of the collaborative node to update the global model until the collaborative training termination condition is met. Then, merge the optimized global models of each collaborative node to obtain the final global large model.

[0039] Specifically, the frequency of overlapping collaborative training between the front-end and back-end adapts and elastically scales according to the manifold conflict state caused by heterogeneous data, until the preset maximum number of training steps is reached.

[0040] When the preset collaborative training endpoint is reached, each collaborative node stops its local training. At this point, because each node's model has undergone manifold space regularization constraints through task vectors throughout the training process, the parameters of each node have a high degree of structural compatibility. A preset end-model merging algorithm (including but not limited to Task Arithmetic, Ties-merging, or Drop And Rescale (DARE)) is invoked to merge the specialized local models of each node into the final global large model within a parameter space without performance conflicts, thus completing the training.

[0041] The task vector-based distributed large model asynchronous training method provided in this embodiment stores the task vector in the background communication sliding buffer and broadcasts it, thus hiding the network transmission in the local computing process. This breaks the serial dependency between communication and computing in traditional distributed training, eliminates the blocking events of computing units waiting for cross-center network transmission, enables the computing hardware to run at full capacity, maximizes the masking of communication overhead, and greatly improves the system throughput of collaborative training. It provides high engineering feasibility for multi-center large model collaborative training in low-bandwidth scenarios.

[0042] In some alternative implementations, the method further includes: Step S205: Reshape the updated total weight vector to obtain the updated feature tensors of each group of the global model, and use the updated feature tensors to cover the model parameters at the current time step.

[0043] Specifically, to facilitate unified computation and cross-node transmission, all layer parameter tensors of different shapes in the large model (such as the multi-head attention weights, FFN weights, bias terms, etc. of Transformer, with different matrix sizes for each layer) are decomposed, flattened into one-dimensional numbers, and then concatenated end to end into an ultra-long one-dimensional vector. The updated total weight vector It is a flat one-dimensional array, which cannot be directly used in large models (each network layer of a large model only accepts the corresponding two-dimensional / three-dimensional tensor parameters), so the structure must be reversed to reconstruct it.

[0044] Strictly follow the tensor splitting order and parameter length of each layer recorded during initialization to update the one-dimensional total weight vector. Segmentation: For example, the first layer weights of the model occupy the first 10,000 numbers, the second layer biases occupy the next 512 numbers, the multi-head attention matrix occupies the subsequent 65,536 numbers... According to the segmentation record when it was originally spliced, the long vector is cut back into independent one-dimensional small arrays.

[0045] To restore the original dimensional shape of each segmented one-dimensional array, the tensor reshaping operation is called to restore the original matrix dimension of the network layer. After all processing is completed, a complete set of updated feature tensors that perfectly match the original model structure is obtained (each layer corresponds to an updated weight / bias tensor).

[0046] The GPU memory still contains the old model parameters that were continuously trained up to the current moment during the communication wait period. The overwrite operation is performed to replace the original parameters of the corresponding network layer in the GPU memory with the newly reshaped set of updated feature tensors one-to-one. All weights of the local model are then globally aligned and updated.

[0047] Step S206: If the optimizer used for local model training has a momentum mechanism, then the historical momentum information currently cached by the corresponding optimizer will be forcibly reset to a zero matrix.

[0048] Specifically, the mainstream optimizers in deep learning are divided into two categories: momentum-based and momentum-free. Momentum-based optimizers continuously cache two types of historical information during model training: first-order momentum (gradient moving mean) and second-order momentum (gradient squared moving mean) to smooth gradients and stabilize convergence. These data are stored in the optimizer cache matrix. Momentum-free optimizers do not save any historical gradient cache.

[0049] Read the optimizer configuration bound to the current local training and determine whether it contains momentum cache logic: If it is a momentum-type optimizer, perform a subsequent reset, uniformly fill all first-order momentum matrices and second-order momentum matrices (corresponding to all network layers) stored in the optimizer with all zero values ​​to become zero matrices, set all m_t (first-order momentum) and v_t (second-order momentum) corresponding to the weights of each layer to 0, and clear all bias terms and their corresponding momentum caches synchronously; if there is no momentum optimizer, no operation is required.

[0050] After the reset logic is completed and the momentum is cleared, the foreground GPU training resumes. The next round of backpropagation will start accumulating gradient momentum from zero again, without adding historical gradient memories from the previous training phase before synchronization.

[0051] This synchronization involves overwriting the local parameters with newly unified, globally merged model weights. The gradient momentum generated during previous independent training steps was accumulated based on the old model parameters. The distribution of parameters and the optimization direction of the old and new models shift significantly at the global level. If the old momentum cache is retained, historical gradient memories will conflict with the new global parameters, causing training oscillations, loss rebound, decreased model accuracy, and even divergence. Clearing the momentum is equivalent to adapting the optimizer to the new global parameters, allowing for smoother iterations from the synchronized starting point, ensuring stable training convergence.

[0052] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment restores the feature tensors of each layer of the model by reshaping the vectors, thus facilitating local model parameter updates. For the optimizer with momentum, historical momentum is reset to clear the gradient accumulation interference left by the iteration of new and old weights, avoid the continuous impact of historical conflict update information on subsequent training, ensure that the gradient update of the new round of collaborative training is accurate and reliable, and improve the stability of model parameter iteration.

[0053] This embodiment provides a distributed large-scale asynchronous training method based on task vectors, which can be used in any cooperating node in the aforementioned computer system or distributed network. Figure 2 This is a flowchart of a distributed large-scale asynchronous training method based on task vectors according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S301: Deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the current cycle total weight vector.

[0054] Specifically, step S301 includes: Step S3011: Obtain the initial model weights of the initial global model, and determine the feature tensor based on the initial model weights.

[0055] Specifically, all data centers / computing nodes are loaded with the exact same pre-trained initial large model, with 100% identical model structure and initial values ​​of parameters for each layer, ensuring a unified benchmark across the entire network.

[0056] Iterate through all network layers of the model, reading the trainable parameters of each layer (including the weight matrix W and bias vector b, which are the feature tensors mentioned in the technical documentation). For example, the Transformer architecture includes multi-head attention weights, attention biases, feedforward network weights, layer normalization parameters, etc. Each set of independent parameters is a feature tensor, excluding non-trainable fixed constants and input / output placeholders. Only the parameter tensors used for training and updates are extracted. Simultaneously save the original dimensions, shape, element length, layer level, and sorting number of each layer's tensor. This record will be reused in reverse when reconstructing the model later to avoid errors during reconstruction.

[0057] Step S3012: According to the preset fixed tensor order, flatten all feature tensors in the initial global model into one-dimensional tensors respectively.

[0058] Specifically, for each feature tensor, a flattening operation is performed: if the current feature tensor is a two-dimensional weight matrix, such as a fully connected weight matrix of shape [768, 3072], the multi-row, multi-column matrix is ​​directly flattened into a continuous one-dimensional array; if the current feature tensor is a one-dimensional bias vector

[3072] , it is already one-dimensional and does not require additional transformation, so it is directly preserved; if there are three-dimensional or four-dimensional tensors (convolutional models), they are also flattened into a single one-dimensional sequence. After flattening, the total length of the flattened elements of the tensor is recorded simultaneously for subsequent slicing reconstruction.

[0059] Step S3013: Concatenate and splice all one-dimensional tensors in sequence to construct the initial global model total weight vector.

[0060] Specifically, a fixed tensor order is preset. The flattened one-dimensional tensors of each layer are concatenated end-to-end sequentially. First, the flattened one-dimensional array of the first layer's parameters is appended; then, the flattened one-dimensional array of the second layer is appended directly to its end. This process continues until all network layer flattened tensors are concatenated, without disrupting or skipping any layer's parameters. After concatenation, a one-dimensional vector of dimension 1×m is obtained, denoted as... .

[0061] After the splicing is completed, a segmented index lookup table is generated simultaneously, recording the start and end indices of each tensor in the one-dimensional vector.

[0062] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment unifies the initial models of each collaborative node, flattens and concatenates all feature tensors in a fixed order to obtain the global weight vector, standardizes the tensor arrangement format, realizes standardized and unified representation of model parameters, facilitates cross-node synchronization, parameter comparison and aggregation operations, reduces the difficulty of data alignment in distributed collaborative training, and improves the computational stability and processing efficiency of model interaction and parameter fusion.

[0063] Step S302: Calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer and broadcast it to other collaborative nodes, and at the same time, the process of independently iterating and updating the model weight vector using the local heterogeneous dataset is not interrupted.

[0064] Specifically, step S302 includes: Step S3021: According to the preset fixed tensor order, flatten the current model weights at the end of the current cycle into one-dimensional tensors and concatenate them sequentially to obtain the total weight vector of the current cycle.

[0065] Specifically, for any cooperating node Following a pre-defined fixed tensor order identical to the initial global model total weight vector, the current model weights, after the current iteration is completed, are flattened and sequentially concatenated to construct a current cycle total weight vector of dimension 1×m, denoted as . .

[0066] Step S3022: Subtract the total weight vector of the current cycle from the total weight vector of the initial global model element by element to calculate the local task vector of the corresponding collaborative node.

[0067] Specifically, the total weight vector of the current period Compared with the initial global model total weight vector By subtracting element by element, the local task vector representing the phased macroscopic optimization trajectory of the corresponding collaborative node is obtained. .

[0068] Step S3023: Store the local task vector into the background communication sliding buffer, use the network protocol stack to package the local task vector and broadcast it to other cooperating nodes, while the front-end computing unit performs local model training for the next communication and cooperation cycle.

[0069] Specifically, the calculated Write to the preset background communication sliding buffer in the video memory and immediately release the control pointer of the foreground computing unit to achieve data isolation.

[0070] A resident background communication thread is triggered to start, asynchronously reading the local task vector from the background communication sliding buffer in a non-blocking manner. It is packaged using the network protocol stack and broadcast to the other n-1 nodes through the cross-center network.

[0071] like Figure 3 The diagram illustrates an asynchronous training process using two collaborative nodes as an example. The red and blue arrows represent data transmission paths between different collaborative nodes. During the entire process of network packetization, cross-center transmission, and external reception of the task vector in the background communication thread, the foreground computing core (GPU) does not need to block or wait. Instead, based on the isolation mechanism of the background communication sliding buffer, it seamlessly continues to execute local model training for subsequent steps. This completely masks the background network communication latency on the timeline by utilizing the continuous computation time of the foreground.

[0072] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment generates weight vectors through a unified tensor flattening and splicing rule. The vectors are subtracted element by element to quickly obtain the locally updated task vectors. Data is cached and broadcast synchronously using a sliding buffer. At the same time, the front-end and back-end execute communication transmission and local model training in parallel, thereby decoupling the computation and communication tasks, effectively shortening the collaboration cycle time, ensuring the synchronization and consistency of parameters of each node, and improving the overall operating efficiency of distributed collaborative training.

[0073] Step S303: Receive external task vectors broadcast by other collaborative nodes, construct a task vector list based on local task vectors and external task vectors, update the total weight vector of the corresponding collaborative node using a dynamic alignment mechanism, and update the global model of the corresponding collaborative node based on the updated total weight vector to obtain the optimized global model.

[0074] Specifically, step S303 includes: Step S3031: Update the corresponding local task vector using the current task vector of the collaborating node, and construct a task vector list based on the updated local task vector and the external task vector.

[0075] Specifically, external task vectors obtained from communication are collected, and a task vector list is constructed by combining them with local task vectors. For the current node Since the front-end computing core has undergone multiple training steps during the communication, the model weights at the current moment are also calculated using the flattening and cascading method to obtain the latest task vector at the current moment, and then replaced in the list. This avoids problems such as timing misalignment and distortion of fusion results.

[0076] Step S3032: Extract multiple elements at the same feature position from the task vector list and construct a local scalar set.

[0077] Specifically, for each feature position in the 1×m dimension of the task vector list, traverse all n vectors at that position to generate a local scalar set.

[0078] All task vectors have the same dimension, with the same index representing the same feature location. Each feature location corresponds to the same network parameter in the model. The task vector length is m. Elements with indices 0, 1, 2, ..., m-1 are traversed sequentially and processed individually. All n vectors in the task vector list are traversed, and the scalar values ​​at the p-th position of each of the n vectors are extracted. These p-th position values ​​are then grouped together to form a local scalar set corresponding to the p-th position. Global vector fusion is decomposed into independent parameter-by-parameter computation. The update trend of all nodes is determined separately for each model parameter, achieving fine-grained parameter alignment.

[0079] Step S3033: Retrieve the element with the largest absolute value in the local scalar set, and use the sign of the element with the largest absolute value as the dominant feature sign.

[0080] Specifically, the sign of the element with the largest absolute value in the local scalar set is retrieved and defined as the dominant feature sign. The larger the absolute value, the stronger the update force of that node on this parameter and the higher its influence on the optimization trend. The sign of the strongest update direction is used as the globally unified benchmark direction to filter conflicting updates.

[0081] Step S3034: Determine the effective elements in the local scalar set based on the dominant feature symbols, and determine the fusion elements at the corresponding feature positions based on all effective elements.

[0082] Specifically, the local scalar set is traversed, and all elements inconsistent with the dominant feature sign are directly cleared to zero. Elements consistent with the dominant feature sign are considered valid elements (non-zero elements), and the number of retained non-zero elements, k, is counted. The algebraic sum of all elements in the local scalar set is calculated and divided by the number of non-zero elements, k, to obtain the mean of the valid elements. This mean is then used as the fusion element and filled into the fusion task vector. The corresponding feature positions are determined. Conflicting gradients that are opposite to the mainstream update direction are filtered out to avoid the offsetting of training directions at different nodes and to prevent model oscillation and divergence. Only updates with consistent directions are retained for averaging to generate a globally unified update vector that takes into account the majority of nodes and is convergent and stable.

[0083] In some alternative implementations, the method further includes: Step a1: Calculate the proportion of all valid elements at the corresponding feature position to all elements, and use it as a global manifold conflict metric.

[0084] Specifically, the proportion of sign consistency features to the total number of parameters is calculated and defined as a global manifold conflict metric. This is used to characterize the overall degree of divergence in the training directions of all collaborative nodes.

[0085] Iterate through all m feature parameter positions (each index of the vector) of the model. For the p-th feature position, assuming there are n node elements at this position, let the number of valid same-direction elements selected at this position be . (Elements with the same sign as the dominant element and not cleared to zero), then the percentage of elements with the same sign at the p-th characteristic position is:

[0086] The closer the symbol consistency ratio is to 1, the more unified the update direction of all nodes at that feature location is, and there is no conflict; the closer the symbol consistency ratio is to 0, the more opposite the update direction of most nodes at that feature location is, and there is a serious conflict.

[0087] By statistically analyzing the consistency percentage of all m parameter positions, we can identify the total number of feature positions with sign conflicts. The global manifold conflict metric is the ratio of the total number of feature positions with sign conflicts to the total number of all feature positions m. The value approaching 0 indicates that the update direction of most parameters is consistent across all nodes, with minimal divergence. A value close to 1 indicates that a large number of parameter nodes are updated in opposite directions, resulting in severe manifold conflicts and making the model prone to deviation and divergence.

[0088] Step a2: Adjust the number of steps in the next collaborative communication cycle based on the global manifold conflict metric.

[0089] Specifically, based on the global manifold conflict metric calculated in the previous step... Based on the relationship between the global manifold conflict metric and a preset threshold, the number of steps in the next collaborative communication cycle is dynamically adjusted. The adjusted cycle number, based on actual conditions, must be locked within the preset upper and lower limits. Inside.

[0090] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment obtains a global manifold conflict metric by statistically analyzing the proportion of effective elements, quantifies the degree of conflict in multi-node parameter updates, and adaptively adjusts the number of steps in the next collaborative cycle based on this metric, thereby achieving a dynamic adaptive balance of training rhythm and taking into account both model fusion accuracy and overall collaborative training efficiency.

[0091] In some alternative implementations, step a2 above includes: Step a21: If the global manifold conflict metric is greater than the preset threshold, then reduce the number of steps in the next collaborative communication cycle by the first preset step size.

[0092] Specifically, if the global manifold conflict metric If the value exceeds a preset threshold, indicating an increase in symbol conflicts between nodes (reaching the set threshold), then the number of steps in the next cooperative communication cycle will be reduced by a first preset step size. This increases communication frequency and enforces constraints to optimize the landscape. Smaller cycle steps mean that local training doesn't require many steps to trigger cross-node communication synchronization, allowing for frequent global updates to constrain each node, suppressing divergence in optimization direction, and preventing model training from spiraling out of control.

[0093] Step a22: If the global manifold conflict metric is not greater than the preset threshold, then the number of steps in the next collaborative communication cycle is increased by the second preset step size.

[0094] Specifically, if the global manifold conflict metric If the value exceeds a preset threshold, it indicates that the optimization direction of each node tends to be stable and consistent (with less conflict). In this case, the number of steps in the next cycle is increased by a preset step size. This reduces communication frequency, frees up network bandwidth, and allows for deeper specialization of local models. With a larger cycle number, local training can continue for more rounds before synchronization, significantly reducing cross-datacenter network transmissions, saving bandwidth resources, and allowing each node to perform deeper specialization fine-tuning based on local data.

[0095] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment divides parameter conflict levels with a preset threshold. In low-conflict scenarios, the number of synchronization cycle steps is reduced to accelerate the convergence of multi-node parameter fusion. In high-conflict scenarios, the number of local training steps is increased to fully utilize local data and reduce update divergence.

[0096] Step S3035: Determine the global alignment task vector based on the fusion elements at each feature location, and add the global alignment task vector to the initial global model total weight vector element by element to reconstruct the updated total weight vector.

[0097] Specifically, after the model resumes its training in the foreground, it is dynamically updated... The number of steps will be used as the new countdown interval to continue local optimization. When the new countdown is complete... At each step, the process of calculating the local task vector of the collaborative node and updating the global model is triggered and repeated. This continues, with the frequency of overlapping collaborative training between the front-end and back-end adaptively scaling up and down according to the manifold conflict caused by heterogeneous data, until the preset maximum number of training steps is reached.

[0098] The global alignment task vector and all node local task vectors have the same dimension, both being one-dimensional vectors of length m. For each feature index p (corresponding to the p-th parameter of the model), the calculated value of the fusion element specific to that index is assigned to the global alignment task vector. The p-th position. After all m parameter positions are filled, a complete, unified, and integrated one-dimensional vector representing the consistent update trend of all nodes is obtained. This vector represents the global model update increment jointly recognized by all nodes in this synchronization cycle. Using this global alignment task vector... Compared with the initial global model total weight vector Perform element-wise addition, and for each feature index p, perform point-to-point addition: This yields the latest parameters at position p after synchronization calibration. Reverse reconstruction involves deriving the computational logic of the task vector in reverse order to obtain the updated total node weight vector. .

[0099] The distributed large-scale asynchronous training method based on task vectors provided in this embodiment summarizes local and external task vectors to form a list, groups and filters dominant symbols according to feature positions to divide effective elements and complete the fusion, and uses a dynamic alignment mechanism to unify the update direction of multiple nodes and avoid conflicts in parameter updates of multiple nodes. By reconstructing the model inversely with the fused vector and the initial weights, the training information of each node is taken into account, the accuracy of cross-node parameter fusion is improved, and the consistency of distributed collaborative model updates is guaranteed.

[0100] Step S304: Repeat the process of calculating the local task vectors of the collaborative nodes and updating the global model until the collaborative training termination condition is met. Then, merge the optimized global models of each collaborative node to obtain the final global model. For details, please refer to [link to details]. Figure 1 Step S201 of the illustrated embodiment will not be described again here.

[0101] In one specific embodiment, such as Figure 4 The diagram shown is a complete flowchart of this embodiment, which specifically includes the following steps: (1) All collaborative nodes load the same initial global model and complete parameter preprocessing: flatten and splice the feature tensors of each layer to generate the total weight vector of the initial global model, and initialize the synchronization cycle step number of the system.

[0102] (2) Each node continuously and independently trains the model using the local dataset and performs forward propagation and backward propagation to update the local parameters.

[0103] (3) When the number of local training steps reaches the current synchronization cycle threshold, the node calculates the local task vector corresponding to this round of training and stores it in an independent buffer to achieve isolation between training data and data to be transmitted.

[0104] (4) Dual-thread parallel execution: The computing GPU does not stop and continues to carry out a new round of local training; the communication background thread reads the task vector in the buffer and transmits it asynchronously to other cooperating nodes. The two processes run simultaneously, and network transmission no longer blocks model computation.

[0105] (5) After all external node task vectors are received, local training is temporarily paused: the latest task vector of this node is updated, and the list of task vectors of all nodes is assembled; the effective updates in the same direction are filtered and fused to obtain the globally aligned task vectors, the updated model weights are reconstructed, and the local model parameters are overwritten; if the optimizer has a momentum mechanism, the momentum cache is reset; the global conflict metric is statistically analyzed, and the number of synchronization cycle steps in the next round is adaptively adjusted.

[0106] (6) Determine whether the overall training has reached the preset maximum number of training steps. If not, return to the local independent optimization iteration step and continue training in a loop using a new synchronization cycle. If it has reached the maximum number of training steps, perform end model merging, integrate the training results of each node, and the training process ends.

[0107] This embodiment also provides a distributed large-scale asynchronous training device based on task vectors, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0108] This embodiment provides a distributed large-scale asynchronous training device based on task vectors, applicable to any cooperating node in a distributed network, such as... Figure 5 As shown, it includes: The local model independent training module 501 is used to deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the current cycle total weight vector.

[0109] The task vector calculation module 502 is used to calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer and broadcast it to other collaborative nodes, and at the same time, the process of independently iteratively updating the model weight vector using the local heterogeneous dataset is not interrupted.

[0110] The global model local optimization module 503 is used to receive external task vectors broadcast by other collaborative nodes, construct a task vector list based on the local task vectors and external task vectors, update the total weight vector of the corresponding collaborative node using a dynamic alignment mechanism, and update the global model of the corresponding collaborative node based on the updated total weight vector to obtain the optimized global model.

[0111] The model merging module 504 is used to repeatedly calculate the local task vectors of the collaborative nodes to update the global model until the collaborative training termination condition is met. Then, the optimized global models of each collaborative node are merged to obtain the final global large model.

[0112] In some alternative implementations, the local model independent training module 501 includes: The initial model weight determination unit is used to obtain the initial model weights of the initial global model and determine the feature tensor based on the initial model weights.

[0113] Tensor flattening unit is used to flatten all feature tensors in the initial global model into one-dimensional tensors according to a preset fixed tensor order.

[0114] Tensor splicing unit is used to sequentially cascade and splice all one-dimensional tensors to construct the initial global model total weight vector.

[0115] In some alternative implementations, the task vector calculation module 502 includes: The total weight vector determination unit at the end of the cycle is used to flatten the current model weights at the end of the current cycle into one-dimensional tensors and concatenate them sequentially according to a preset fixed tensor order to obtain the total weight vector of the current cycle.

[0116] The task vector calculation unit is used to subtract the total weight vector of the current cycle from the total weight vector of the initial global model element by element to calculate the local task vector of the corresponding collaborative node.

[0117] The asynchronous processing unit stores the local task vector into the background communication sliding buffer, packages the local task vector using the network protocol stack, and broadcasts it to other cooperating nodes. At the same time, the front-end computing unit performs local model training for the next communication and cooperation cycle.

[0118] In some alternative implementations, the global model local optimization module 503 includes: The task vector list unit is used to update the corresponding local task vector using the current task vector of the cooperating node, and to construct a task vector list based on the updated local task vector and the external task vector.

[0119] The local scalar set building unit is used to extract multiple elements at the same feature position in the task vector list and construct a local scalar set.

[0120] The dominant feature symbol determination unit is used to retrieve the element with the largest absolute value in the local scalar set and use the symbol of the element with the largest absolute value as the dominant feature symbol.

[0121] The fusion element determination unit is used to determine the effective elements in the local scalar set based on the dominant feature symbol, and to determine the fusion element at the corresponding feature position based on all effective elements.

[0122] The total weight vector reconstruction unit is used to determine the global alignment task vector based on the fusion elements at each feature location, and then add the global alignment task vector to the initial global model total weight vector element by element to reconstruct the updated total weight vector.

[0123] The task vector-based distributed large-scale asynchronous training device provided in this embodiment of the invention can execute the task vector-based distributed large-scale asynchronous training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0124] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0125] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0126] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0127] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the task vector-based distributed large model asynchronous training method of the embodiments of the present invention.

[0128] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0129] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the task vector-based distributed large model asynchronous training method shown in the above embodiments is implemented.

[0130] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A distributed asynchronous training method for large models based on task vectors, characterized in that, The method is applied to any cooperating node in a distributed network, and the method includes: Deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the current cycle total weight vector; The local task vector of the corresponding collaborative node is calculated based on the current period total weight vector and the initial global model total weight vector. The local task vector is stored in the background communication sliding buffer and broadcast to other collaborative nodes. At the same time, the process of independently iterating and updating the model weight vector using the local heterogeneous dataset is not interrupted. Receive external task vectors broadcast by other cooperating nodes, construct a task vector list based on the local task vectors and external task vectors, update the total weight vector of the corresponding cooperating node using a dynamic alignment mechanism, update the global model of the corresponding cooperating node based on the updated total weight vector, and obtain the optimized global model. The process of repeatedly calculating the local task vectors of the collaborative nodes and updating the global model continues until the collaborative training termination condition is met. The optimized global models of each collaborative node are then merged to obtain the final global large model.

2. The method according to claim 1, characterized in that, Deploy the same initial global model as other collaborative nodes, and construct the initial global model's total weight vector, including: Obtain the initial model weights of the initial global model, and determine the feature tensor based on the initial model weights; According to a preset fixed tensor order, all feature tensors in the initial global model are flattened into one-dimensional tensors respectively; By sequentially concatenating all one-dimensional tensors, the initial global model total weight vector is constructed.

3. The method according to claim 1 or 2, characterized in that, Calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer, and broadcast it to other collaborative nodes, including: According to the preset fixed tensor order, the current model weights at the end of the current cycle are flattened into one-dimensional tensors and sequentially concatenated and spliced ​​to obtain the total weight vector of the current cycle. The local task vector of the corresponding collaborative node is calculated by subtracting the current period total weight vector from the initial global model total weight vector element by element. The local task vector is stored in the background communication sliding buffer, and the local task vector is packaged and broadcast to other cooperating nodes using the network protocol stack. At the same time, the front-end computing unit performs local model training for the next communication and cooperation cycle.

4. The method according to claim 1, characterized in that, A task vector list is constructed based on the local task vector and the external task vector, and the total weight vector of the corresponding collaborative node is updated using a dynamic alignment mechanism, including: Update the corresponding local task vector using the current task vector of the collaborating node, and construct a task vector list based on the updated local task vector and the external task vector; Extract multiple elements at the same feature position from the task vector list and construct a local scalar set; Retrieve the element with the largest absolute value in the local scalar set, and use the sign of the element with the largest absolute value as the dominant feature sign; The effective elements in the local scalar set are determined based on the dominant feature symbols, and the fusion elements at the corresponding feature positions are determined based on all effective elements. The global alignment task vector is determined based on the fusion elements at each feature location, and the global alignment task vector is added element by element to the initial global model total weight vector to reconstruct the updated total weight vector.

5. The method according to claim 4, characterized in that, The method further includes: Calculate the proportion of all valid elements to all elements at the corresponding feature position, and use it as a global manifold conflict metric. The number of steps in the next cooperative communication cycle is adjusted based on the global manifold conflict metric.

6. The method according to claim 5, characterized in that, Adjusting the number of steps in the next cooperative communication cycle based on the global manifold conflict metric includes: If the global manifold conflict metric is greater than a preset threshold, the number of steps in the next collaborative communication cycle is reduced by a first preset step size. If the global manifold conflict metric is not greater than a preset threshold, then the number of steps in the next collaborative communication cycle is increased by a second preset step size.

7. The method according to claim 4, characterized in that, The method further includes: The updated total weight vector is reshaped to obtain the updated feature tensors of the global model, and the updated feature tensors are used to cover the model parameters at the current time. If the optimizer used for local model training has a momentum mechanism, the historical momentum information currently cached by the corresponding optimizer will be forcibly reset to a zero matrix.

8. A distributed large-scale asynchronous training device based on task vectors, characterized in that, The device is applied to any cooperating node in a distributed network, and the device includes: The local model independent training module is used to deploy the same initial global model as other collaborative nodes, construct the initial global model total weight vector, and independently iterate and update the initial global model total weight vector using the local heterogeneous dataset within the current collaborative communication cycle to obtain the current cycle total weight vector. The task vector calculation module is used to calculate the local task vector of the corresponding collaborative node based on the current period total weight vector and the initial global model total weight vector, store the local task vector in the background communication sliding buffer and broadcast it to other collaborative nodes, and at the same time, the process of independently iteratively updating the model weight vector using the local heterogeneous dataset is not interrupted. The global model local optimization module is used to receive external task vectors broadcast by other collaborative nodes, construct a task vector list based on the local task vectors and external task vectors, update the total weight vector of the corresponding collaborative node using a dynamic alignment mechanism, and update the global model of the corresponding collaborative node based on the updated total weight vector to obtain the optimized global model. The model merging module is used to repeatedly calculate the local task vectors of the collaborative nodes to update the global model until the collaborative training termination condition is met. Then, the optimized global models of each collaborative node are merged to obtain the final global large model.

9. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images