A base model distributed training system and training method

Abstract

The application relates to the technical field of deep learning, in particular to a distributed training system and method of a basic model. First, an initial basic model is distributed to each intelligent terminal, the initial basic model is trained by using a local training set, and a personalized model after training is obtained; then, model parameters of multiple personalized models received are subjected to graph clustering updating in each edge server, preliminary aggregation processing is performed according to the clustering results, and multiple local models after aggregation are obtained; and the model parameters of the multiple local models are sent to a cloud server; the model parameters of the multiple local models are subjected to overall aggregation, and a new initial basic model is obtained. In this way, the initial basic model is distributed to multiple intelligent terminals for training, and the model parameters of the personalized models after training are aggregated, so that a final trained basic model is obtained, and the training efficiency of the basic model can be improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a distributed training system and training method for a basic model. Background Technology

[0002] With the rise of large-scale parametric models, the application of foundational models has attracted interest across various industries. These models can be trained on a wide range of data and are adaptable to most downstream tasks; we call these models pre-trained foundational models (also known as base models or large models). Compared to general AI (Artificial Intelligence) models, foundational models require fine-tuning to enhance the task performance of downstream models.

[0003] Existing basic models are trained on known or widely available corpora, which severely limits data collection. For example, data such as corporate operational data and personal privacy data cannot be included in the training of basic models. This data collection and model training method faces the risks and harms of feature convergence and data privacy leakage. Consequently, basic models obtained by centralized training methods using cloud platforms have certain limitations and poor generalization performance for task processing.

[0004] Meanwhile, the training of basic models in existing technologies still relies on centralized training on cloud platforms. This training method requires a lot of computing resources, resulting in low training efficiency of basic models. Summary of the Invention

[0005] This invention provides a distributed training system and method for a basic model, which solves the technical problem that the centralized training method using a cloud platform in the prior art results in low training efficiency of the basic model.

[0006] In a first aspect, the present invention provides a distributed training system for a basic model, comprising: multiple smart terminals, multiple edge servers and a cloud server, wherein one edge server is communicatively connected to multiple subordinate smart terminals, and the multiple edge servers are communicatively connected to the cloud server;

[0007] The cloud server is used to manage the multiple smart terminals and multiple edge servers to perform multiple rounds of iterative training on the initial basic model in order to obtain a trained basic model.

[0008] In one round of iterative training:

[0009] The cloud server is used to distribute the initial basic model to various smart terminals;

[0010] The smart terminal is used to train the initial basic model using a local training set to obtain a trained personalized model; and to send the model parameters of the personalized model to the corresponding edge server.

[0011] The edge server first performs graph clustering updates based on the received multiple personalized models to determine the model parameters that need to be aggregated; then it performs preliminary aggregation processing on these selected model parameters to obtain multiple aggregated local models; and then sends the model parameters of the multiple local models to the cloud server.

[0012] The cloud server is also used to aggregate the model parameters of the multiple local models to obtain a new initial base model; and to determine whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model; otherwise, the new initial base model is sent to the smart terminal for the next round of iterative training.

[0013] According to the present invention, a distributed training system for a basic model, including a smart terminal, is used to train the initial basic model using a local training set to obtain a trained personalized model, comprising:

[0014] Set a corresponding loss function for the initial base model;

[0015] The initial base model is trained using a local labeled training set until the loss function is minimized, resulting in a trained personalized model.

[0016] According to a distributed training system for a base model provided by the present invention, the cloud server is further configured to aggregate the model parameters of the multiple local models to obtain a new initial base model, including:

[0017] The cloud server assigns corresponding weight coefficients to each local model based on the size of the training set corresponding to each local model.

[0018] Based on the weight coefficients corresponding to each local model, the model parameters of the multiple local models are aggregated to obtain a new initial base model.

[0019] According to the distributed training system for a basic model provided by the present invention, the intelligent terminal is used to train the initial basic model using a local training set to obtain a trained personalized model, comprising:

[0020] The edge server receives model parameters of personalized models from multiple smart terminals and aggregates the received model parameters of multiple personalized models to obtain an aggregated local model.

[0021] According to a distributed training system for a base model provided by the present invention, the cloud server is further configured to determine whether the iterative training termination condition is met; if so, the new initial base model obtained in this round of training is determined as the trained base model, including:

[0022] When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or

[0023] Determine whether the number of iterations has reached the set number. If so, stop the iteration training and determine the new initial base model obtained in this iteration as the trained base model.

[0024] Secondly, the present invention provides a distributed training method for a basic model, which is applied to a distributed training system for a basic model to perform multiple rounds of iterative training on an initial basic model in order to obtain a well-trained basic model.

[0025] The distributed training system of the basic model includes multiple smart terminals, multiple edge servers and a cloud server, wherein one edge server is communicatively connected to multiple subordinate smart terminals, and the multiple edge servers are communicatively connected to the cloud server.

[0026] The distributed training method includes:

[0027] During one round of iterative training, the cloud server distributes the initial basic model to each smart terminal;

[0028] In each of the aforementioned smart terminals, the initial basic model is trained using a local training set to obtain a trained personalized model; and the model parameters of the personalized model are sent to the corresponding edge server.

[0029] In each edge server, graph clustering is performed based on the received multiple personalized models to determine the model parameters that need to be aggregated; then, these selected multiple personalized model parameters are aggregated to obtain multiple aggregated local models; and the model parameters of the multiple local models are sent to the cloud server.

[0030] In the cloud server, the model parameters of the multiple local models are aggregated to obtain a new initial base model; and it is determined whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model; otherwise, the new initial base model is sent to each of the smart terminals for the next round of iterative training.

[0031] According to a distributed training method for a base model provided by the present invention, the step of training the initial base model using a local training set to obtain a trained personalized model includes:

[0032] Set a corresponding loss function for the initial base model;

[0033] The initial base model is trained using a local labeled training set until the loss function is minimized, resulting in a trained personalized model.

[0034] According to a distributed training method for a base model provided by the present invention, the step of aggregating the model parameters of the multiple local models to obtain a new initial base model includes:

[0035] The cloud server assigns corresponding weight coefficients to each local model based on the size of the training set corresponding to each local model.

[0036] Based on the weight coefficients corresponding to each local model, the model parameters of the multiple local models are aggregated to obtain a new initial base model.

[0037] According to a distributed training method for a base model provided by the present invention, the step of training the initial base model using a local training set to obtain a trained personalized model includes:

[0038] The edge server receives model parameters of personalized models from multiple smart terminals and calculates the second-order similarity between the personalized model parameters of the current round and the new initial base model.

[0039] The graph topology of the computation nodes is constructed based on the second-order similarity, and the initial graph topology is updated by clustering.

[0040] The aggregation weights are determined based on the clustering results, and the model parameters of the received multiple personalized models are aggregated to obtain the aggregated local model.

[0041] According to a distributed training method for a base model provided by the present invention, the step of determining whether the iterative training termination condition is met, and if so, determining the new initial base model obtained in this round of training as the trained base model, includes:

[0042] When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or

[0043] Determine whether the number of iterations has reached the set number. If so, stop the iteration training and determine the new initial base model obtained in this iteration as the trained base model.

[0044] This invention provides a distributed training system for a basic model, using a cloud server to manage multiple smart terminals and multiple edge servers for iterative training of an initial basic model to obtain a trained basic model. In each iteration, the initial basic model is first distributed to each smart terminal. Each smart terminal trains the initial basic model using its local training set to obtain a trained personalized model. The model parameters of the personalized model are then sent to the corresponding edge server. On each edge server, the received model parameters of the multiple personalized models are initially aggregated to obtain multiple aggregated local models. The model parameters of these local models are then sent to the cloud server. Finally, on the cloud server, the model parameters of the multiple local models are aggregated to obtain a new initial basic model. The system then determines whether the iterative training termination condition is met. If so, the new initial basic model obtained in this round of training is designated as the trained basic model; otherwise, the new initial basic model is sent to the smart terminals for the next round of iterative training. This method of distributing the initial basic model to multiple smart terminals for training and then aggregating the parameters of the trained personalized models to obtain the final trained basic model improves the training efficiency of the basic model. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 A schematic diagram of the distributed training system structure of the basic model provided in the embodiments of the present invention;

[0047] Figure 2 A distributed training method for the basic model provided in the embodiments of the present invention;

[0048] Figure 3 One of the accuracy diagrams of the training method provided in the embodiments of the present invention;

[0049] Figure 4 This is the second schematic diagram illustrating the accuracy of the training method provided in this embodiment of the invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Because existing basic models are trained on known or widely available corpora, data collection is severely limited. For example, enterprise operational data and personal privacy data cannot be incorporated into the training of basic models. This data collection and model training method faces the risk and harm of feature convergence, resulting in basic models obtained through centralized training methods on cloud platforms having certain limitations and poor generalization performance for task processing. Furthermore, the training of basic models in current technologies still relies on centralized training on cloud platforms, which consumes significant computing resources, leading to low training efficiency.

[0052] To address the technical shortcomings of existing basic model training methods, this application proposes a distributed training method for the basic model. Furthermore, during distributed training, a federated learning approach is employed, ensuring both full utilization of training data and guaranteeing data privacy and security. Federated learning, as a privacy-preserving distributed machine learning system, has proven to offer numerous advantages in the Internet of Things (IoT), enabling the aggregation of model data from multiple data owners without sharing the original data. Therefore, this invention utilizes federated learning technology, viewing the federated learning architecture as a bridge between privacy-preserving data and the basic model. However, challenges remain, such as difficulties in fusing local models due to heterogeneous data and convergence of basic model features.

[0053] Existing research on foundational models focuses on designing pre-trained models for specific tasks. Well-established models include OpenAI's CLIP (Contrastive Language-Image Pretraining) and Google's ALIGN (A Large-scale Imaging and Noisy-text Embedding). However, current technologies still face bottlenecks in centralized training, such as difficulties in collecting privacy-preserving data across industries and institutions, high data sharing costs, and challenges in cross-domain model fusion. Building foundational models using federated learning requires ensuring strong knowledge transfer capabilities. Currently, methods and mechanisms for building foundational models using federated learning are lacking, and high-performance collaborative training architectures for foundational models with privacy-preserving properties are still in their infancy.

[0054] To fill the gap in distributed collaborative training of pre-trained base models, this invention proposes a collaborative training method for base models based on "pre-training + personalization" under a federated setting, which uses federated learning as a bridge between the upstream pre-trained base model and the downstream personalized model trained on privacy data.

[0055] The core of the distributed training system and method for the basic model provided by this invention lies in distributed collaborative training, which can be divided into horizontal collaboration and vertical collaboration. In terms of vertical collaboration, a federated basic model for structured information perception is established, and distributed intelligent terminals are set up for training the basic model. The initial basic model is trained on each intelligent terminal, and the personalized features of the training set are successfully learned without the training data leaving the local machine. This forms a bidirectional collaborative training method where the cloud server trains the basic model and the intelligent terminals update the initial basic model. In terms of horizontal collaboration, this invention employs a federated learning topological clustering method to aggregate the personalized models trained on each intelligent terminal, thereby integrating and optimizing the learned model parameters and improving the convergence speed and generalization ability of the basic model.

[0056] The following is combined with Figures 1-4 This invention describes a distributed training system and training method for the basic model.

[0057] Figure 1 This is a schematic diagram of the distributed training system structure of the basic model provided in the embodiments of the present invention, as shown below. Figure 1 As shown, the distributed training system of the basic model includes: multiple smart terminals 101, multiple edge servers 102 and cloud server 103, wherein one edge server 102 is communicatively connected to multiple subordinate smart terminals 101, and multiple edge servers 102 are communicatively connected to cloud server 103.

[0058] The cloud server 102 is used to manage multiple smart terminals 101 and multiple edge servers 102 to perform multiple rounds of iterative training on the initial base model in order to obtain a trained base model.

[0059] In one round of iterative training:

[0060] Among them, cloud server 103 is used to distribute the initial basic model to various smart terminals;

[0061] The smart terminal 101 is used to train the initial basic model using a local training set to obtain the trained personalized model; and to send the model parameters of the personalized model to the corresponding edge server 102.

[0062] Edge server 102 is used to perform preliminary aggregation processing on the model parameters of multiple received personalized models to obtain multiple aggregated local models; and send the model parameters of multiple local models to cloud server 103.

[0063] The cloud server 103 is also used to aggregate the model parameters of multiple local models to obtain a new initial base model; and to determine whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model; otherwise, the new initial base model is sent to the smart terminal for the next round of iterative training.

[0064] In one embodiment, the smart terminal 101 is specifically used for:

[0065] Set a corresponding loss function for the initial base model; for example, each smart terminal 101 sets a different loss function for the application scenario, function and accuracy requirements of its trained personalized model, or personalized models used to process similar businesses can set the same loss function.

[0066] The initial base model is trained using a local labeled training set until the loss function is minimized, resulting in a trained personalized model.

[0067] In one embodiment, the cloud server 103 is specifically used for:

[0068] Assign corresponding weight coefficients to each local model based on the size of the training set corresponding to each local model; for example, determine the total training set data of multiple personalized models corresponding to each local model, and assign a larger weight coefficient to the local model with a larger total data.

[0069] Based on the weight coefficients corresponding to each local model, the model parameters of multiple local models are aggregated to obtain a new initial base model.

[0070] In one embodiment, the edge server 102 is specifically used for:

[0071] The system receives model parameters of personalized models from multiple smart terminals 101, and aggregates the received model parameters of multiple personalized models to obtain an aggregated local model.

[0072] For example, each edge server 102 assigns a corresponding weight coefficient to each personalized model based on the size of the training set data corresponding to each personalized model, and then aggregates the model parameters of multiple personalized models based on the weight coefficients corresponding to each personalized model to obtain an aggregated local model.

[0073] In one embodiment, the cloud server 103 is further used for:

[0074] When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or

[0075] Determine whether the number of iterations has reached the set number. If so, stop the iteration training and determine the new initial base model obtained in this iteration as the trained base model.

[0076] In one embodiment, the cloud server 103 is further used for:

[0077] After obtaining a new initial base model in each iteration of training, if it is determined that the iteration termination condition has not been reached, the new initial base model is sent to each smart terminal 101 for the next round of iteration training.

[0078] According to the training system provided in this embodiment, the initial basic model can be distributed to multiple smart terminals for training, and then the parameters of the trained personalized models can be aggregated to obtain the final trained basic model, which can improve the training efficiency of the basic model.

[0079] The distributed training method for the basic model provided by this invention is described below. The distributed training method for the basic model described below is based on the training system provided above and can be referred to in correspondence with the distributed training system for the basic model described above.

[0080] Figure 2 This invention provides a distributed training method for a base model. This method is applied to a distributed training system for the base model to perform multiple rounds of iterative training on an initial base model to obtain a trained base model. Please refer to... Figure 2 As shown, the distributed training method for this basic model includes:

[0081] S201. In one round of iterative training, the cloud server distributes the initial basic model to each smart terminal.

[0082] For example, communication connections are pre-established between multiple smart terminals in the training system and the cloud server, such as 20 smart terminals connected to the cloud server. During the first iteration of training, the initial basic model can be distributed to the associated smart terminals through the cloud server.

[0083] The initial basic model first distributed to each smart terminal by the cloud server is an untrained model, and its model parameters are all preset industry-standard settings, which will not be elaborated here.

[0084] S202. In each smart terminal, the initial basic model is trained using the local training set to obtain the trained personalized model; and the model parameters of the personalized model are sent to the corresponding edge server.

[0085] For example, to better understand the training method in this embodiment, all smart terminals can be defined as a smart terminal set. Define multiple edge server sets

[0086] For example, in one round of training, each smart terminal uses its local training set to train the initial base model distributed to it, resulting in a trained personalized model. For instance, in one round r, the smart terminal... The initial base model, distributed by the cloud server, is trained using a local training set. For example, the initial base model can be instantiated as follows: Where d represents the number of model parameters. f can be a linear regression model, a classification model, or a deep neural network model. Unlike traditional distributed machine learning models, the training method proposed in this embodiment can protect user privacy to the greatest extent, ensuring that the training dataset does not leak out of the smart terminal during the entire initial basic model training process. For each smart terminal, the goal is to use the initial basic model for local fine-tuning, resulting in a better personalized model without consuming too many resources—that is, a trained personalized model—to serve its own reasoning tasks, such as news classification and industrial anomaly detection. This model lowers the barrier for small and micro enterprises to conduct large-scale model training.

[0087] For example, each smart terminal sets different or the same loss function for the initial basic model according to its own business needs; then it uses a local labeled training set to train the initial basic model until the loss function is minimized, thus obtaining the trained personalized model.

[0088] In this embodiment, federated learning is used to train a base model G that performs a vision task. There are C smart terminals. Furthermore, each smart terminal can access a labeled training set. Right now

[0089]

[0090] Assume its feature space is Where d is the feature dimension of the input (the first layer of the NN model). The label space is... Where K represents the category. For example, in an obstacle recognition task, the image information input by the visual sensor is the input data, and the label is the judgment result, which is divided into two categories: "is an obstacle" and "is not an obstacle". Different obstacle recognition needs will have different types of labels. In addition to the "whether" judgment label, there are also "what" label types, such as "person", "dog", "fence", and other labels.

[0091] S203. In each edge server, the model parameters of the received multiple personalized models are initially aggregated to obtain multiple aggregated local models; and the model parameters of the multiple local models are sent to the cloud server.

[0092] For example, edge servers can be small base stations, cache servers, load balancers, etc.

[0093] Generally, a cloud server can be connected to multiple edge servers for unified management and data processing. Similarly, an edge server can be connected to multiple smart terminals for unified management and data processing.

[0094] For example, an edge server is used to perform preliminary aggregation processing on the model parameters of personalized models received from multiple smart terminals according to a federated model aggregation strategy, so as to coordinate the collaborative training between the initial base model and the personalized model, thereby obtaining an aggregated local model.

[0095] For example, a graph clustering method based on data features can be used to perform preliminary aggregation processing on the model parameters of the personalized models received from multiple smart terminals, ensuring the convergence of the personalized model fusion.

[0096] In one embodiment, the edge server is also used to manage multiple smart terminals associated with it, such as performing statistics on training set data, computing power, and remaining computing resources. Furthermore, based on the size of the training set data corresponding to each smart terminal, different weight coefficients are set for each smart terminal; generally, a smart terminal with more training set data is assigned a larger weight coefficient.

[0097] Define input and output The random variable follows a potential joint distribution This joint distribution can be determined by the class prior probability, which is expressed as: Class conditional probability is Use p c (y = k) represents all classes Data distribution on smart terminal c. These smart terminals connect to edge servers to perform distributed collaborative training in order to collaboratively find a base model instantiated by parameter ω, in order to minimize model loss. The base model training process can be represented as follows:

[0098]

[0099] The data sample size of the smart terminal c is n. c ω represents the currently converged model parameters, and α c It is the aggregate weight of smart terminal c, and The model parameters that the edge servers need to aggregate are determined by the FL network topology update strategy. c (ω i Let be the loss function of the smart terminal c. FL optimizes the initial base model parameters by minimizing the loss function; the optimization process can be expressed as follows:

[0100] ω=arg min G(ω) (3)

[0101] Unlike traditional centralized training of basic models, training large models in a federated learning system requires taking into account the imbalanced distribution of data. The smart terminal c in the data... The above may measure local empirical risks (e.g., crossover loss entropy) with different data distributions, and the local model loss is expressed as:

[0102]

[0103] in It is a classification model (composed of multiple classification functions), which is a downstream target model based on the initial base model ω. k This indicates that data samples (x) can be obtained through the initial basic model ω. c ,y c The probability of belonging to class k. Using stochastic gradient descent as the parameter update method for the personalized model can significantly reduce computation, but requires sufficient training data and training iterations to ensure the performance of the personalized model. The update rule for the local personalized model parameters is as follows:

[0104]

[0105] Where η is the learning rate. This represents the reference of the previous iteration of user terminal c.

[0106] In this embodiment, a graph topology clustering method can be used to initially aggregate the model parameters of multiple received personalized models, resulting in multiple aggregated local models. Graph topology clustering aims to construct a graph representation model for collaborative training between edge servers and user terminals, finding a federated learning network topology that is conducive to training the initial basic model. Simultaneously, utilizing a distributed crowdsourcing computing strategy can reduce the computing resources required for edge servers and cloud servers to build the basic model.

[0107] Before graph clustering, a graph topology must be constructed. For each smart terminal node, its most similar neighboring smart terminals are first found, and edges are set to connect the smart terminal and its neighboring smart terminals, preparing for FL (Federated Learning) aggregation. In this embodiment, the KL (Kullback-Leibler divergence) divergence between the personalized models output by each smart terminal is used to construct a similarity matrix. In the clustering stage, the number of clusters is determined by algorithms such as K-means, and then GCN (Graph Convolutional Network) is used to learn and find the distribution pattern of data features. Finally, FL (Federated Learning) aggregation is performed by reconstructing the graph topology.

[0108] Unlike traditional FL architectures, the distributed training method proposed in this embodiment avoids inheriting all biases from the base model and expands the sources of the corpus. It also ensures the convergence of the base model under imbalanced data. This is closely related to the aggregation strategy of the base model. After each round of local training, the edge server aggregates personalized models from different smart terminals according to a GCN (Graph Convolutional Network)-based aggregation strategy. This paper assumes that while the personalized model is iterating, the instantiated personalized model parameters are uploaded to the edge server for global aggregation, thereby enhancing the performance of the final base model (i.e., the target base model). This embodiment uses a modeling approach based on "pre-training + personalization" to train the base model, which is a novel bidirectional optimization mechanism for sharing and feedback between the base model and the personalized model. This is also the first time a method for collaborative training of the base model and downstream personalized models has been proposed.

[0109] S204. In the cloud server, the model parameters of multiple local models are aggregated to obtain a new initial base model; and it is determined whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model; otherwise, the new initial base model is sent to each smart terminal for the next round of iterative training.

[0110] The model parameter aggregation method in step S203 of this embodiment is the same as the aggregation method in step S204, and will not be described again here.

[0111] For example, determine whether the iterative training termination condition is met. If so, determine the new initial base model obtained in this round of training as the trained base model, including:

[0112] When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or

[0113] Determine if the set number of iterations has been reached. If so, stop the iteration and determine the new initial base model obtained in this iteration as the trained base model.

[0114] For example, cloud servers typically possess powerful computing and storage capabilities, used for the final aggregation and updating of various local models. In other words, cloud servers are used to coordinate the personalized models of various smart terminals. To train a more robust base model G t For the smart terminals in the system, they download the initial basic model from the cloud server to complete the training of their local personalized models.

[0115] In one embodiment, the model parameters of multiple local models are aggregated to obtain a new initial base model, including:

[0116] Each local model is assigned a corresponding weight coefficient. For example, the size of the training datasets for all personalized models corresponding to each local model is first calculated. The larger the training datasets for the personalized models, the larger the weight coefficient assigned to that local model. Finally, based on the weight coefficients of each local model, the model parameters of the multiple local models are aggregated to obtain a new initial base model.

[0117] It should be noted that the training process described in S201-S204 above is only one round of training in the distributed training method of this embodiment. In practical applications, multiple rounds of training are required to complete the training of the final base model. After obtaining a new initial base model in each round of training, the new initial base model is distributed to each smart terminal again for the next round of training, until the entire training process is completed.

[0118] In one embodiment, after obtaining a new initial base model, the loss function of the initial base model is determined. When the value of the loss function meets a preset value, the iterative training is stopped, and the initial base model obtained in this iteration is determined as the target base model.

[0119] In another embodiment, after obtaining a new initial base model, it is determined whether the number of iterations has reached a set number. If so, the iteration training is stopped, and the initial base model obtained in this iteration is determined as the target base model. For example, the number of iterations can be set to 10. When the number of iterations reaches 10, the training process is stopped, and the initial base model obtained in the last training iteration is determined as the target base model.

[0120] The distributed training method for the base model proposed in this embodiment is an efficient distributed model generation method that takes privacy protection into account, aiming to collaboratively train the base model and the personalized model. The training objective is to minimize the loss of the personalized model, that is, to minimize the loss function of the personalized model, and then to continuously update the base model through an aggregation strategy. Combining with formula (4), the above-mentioned minimization of the loss of the personalized model can be expressed as:

[0121]

[0122] In fact, uneven data distribution is obvious in practical applications. Therefore, it is assumed that different smart terminals c have different data distributions. Soft parameter sharing can be used to bring the similar hidden layer parameters of different smart terminals closer together, thereby reducing generalization error. Soft parameter sharing has a similar effect to regularization. The problem in the above formula (6) can be optimized so that the basic model can effectively adapt to the statistical heterogeneity of the data and achieve personalized training. Assuming at a certain time t, then:

[0123]

[0124]

[0125] Where R(·) represents the regularization function term, and λ is the regularization coefficient. ω * For the optimal basic model parameters, These are the optimal personalized model parameters or local model parameters. Specifically, ω in formula (7) * It is subject to the conditions expressed by formula (7.1).

[0126] To mitigate the challenges of data heterogeneity and improve the generalization ability of the base model, this embodiment proposes a parameter matching method. The personalized model in a smart terminal contains hidden layers with key feature information. This information needs to be further clustered to train a more robust base model.

[0127] First, construct the graph. This implementation achieves graph clustering based on FL topology, where the attributes of vertex v contain data features from both the user terminal and the edge server. The attributes of each vertex consist of local model initialization and iterative updates on the user terminal, i.e., v i The non-Euclidean distance between the model parameters of each user terminal is a property of the edge. In the proposed collaborative training architecture, the edge property is the weight dispersion of the user terminal gradient, denoted by u. i This is represented by a graph. Clearly, these attributes change with each aggregation round, therefore a graph is used to represent them. The algorithm updates the parameters to find the optimal co-training topology. In fact, the goal of the algorithm is to find an optimal base model parameter ω. * and optimal local personalized model parameters To minimize the weighted average loss of all devices while also minimizing the weight dispersion among subgraphs, the problem model transforms into:

[0128]

[0129]

[0130]

[0131] Where A i,j ∈{0,1} is the topology of the FouFed network. The adjacency matrix of S(u). i ,u j The sum of the Euclidean distances between vertex i and all its neighbors j in the graph is represented by . The rate at which the local loss decreases after vertex i receives shared parameters from all its neighboring nodes j is denoted by u. i The value of λ1 and λ2 is the regularization coefficient. The larger the value, the stronger the restriction on the corresponding random variable. Among them, ω in formula (8) * and u i It is subject to the constraints expressed by formulas (9) and (10).

[0132] In this embodiment, the local loss F is first solved using two regularization terms. i (f(v i Then, perform FL aggregation of the model on the edge side and update ω and u. i The value of is used. Through the graph update operator, node i aggregates the embeddings of its neighboring nodes to update its own node attributes. This rule needs to satisfy the second constraint. Simultaneously, an adjacency matrix A with classification feature information needs to be found. i,j This yields its regularized expression. Furthermore, the base model aggregates all individual models using the FL topology update rule, which requires satisfying the first constraint.

[0133] Therefore, this implementation independently proposes the evolution process of the basic model and downstream personalized model based on FL topology, as shown in Algorithm 1:

[0134]

[0135]

[0136] The training method provided in this embodiment is applicable to training pre-trained basic models based on deep neural networks and residual networks. This embodiment uses VGG16 and ResNet9 models as examples to verify the effectiveness of the training method provided in this embodiment.

[0137] This example uses the publicly available image classification dataset Cifar10 for experimentation. Four data distributions were first set up, and the degree of data heterogeneity was represented by the symbol τ. A smaller τ indicates greater differences in the data distribution.

[0138] Figure 3 This is one of the accuracy diagrams of the training method provided in the embodiments of the present invention, such as... Figure 3 As shown, ResNet9 performs poorly when trained on a single user terminal, especially when there is a significant difference in data distribution (τ=2), because the test set may contain images with different labels. In contrast, the method proposed in this embodiment performs outstandingly, with both models consistently achieving an accuracy of over 90% on the test set. This represents a substantial improvement over the accuracy of local training (i.e., training on a smart terminal), with accuracy ranging from 11% to 70% for different levels of data distribution differences.

[0139] Figure 4 This is a second schematic diagram illustrating the accuracy of the training method provided in this embodiment of the invention, as shown below. Figure 4 As shown, the accuracy of the VGG16 model tested by a single smart terminal is also poor. When the data distribution is extreme, the convergence accuracy is less than 35%. When τ = 10 and 20, the training effect of a single smart terminal is enhanced, but the model training method provided in this embodiment shows the best performance across a wide range of data distributions, with accuracy improvements of 69%, 47%, 35%, and 6%, respectively. This indicates that the training method proposed in this embodiment can effectively alleviate the feature bias problem, improve the generalization ability of the base model, and enhance the personalization ability of the trained model.

[0140] Figure 3 and Figure 4The horizontal axis represents the number of communication rounds between the edge server and the terminal device, i.e., the number of aggregations. The vertical axis represents the model's test performance, i.e., the accuracy of image classification. As the number of aggregations increases, the model converges to an optimal constant. In summary, the distributed training method for the basic model provided in this embodiment has the following advantages compared with existing technologies:

[0141] 1. The distributed training method for the basic model provided in this embodiment deploys personalized model training and preliminary aggregation processing on smart terminals and edge servers, respectively. Collaborative training of the basic model is achieved through structured clustering of the parameter features of the personalized model. This is a novel bidirectional optimization mechanism for sharing and feedback between the basic and personalized models. It enables distributed collaborative training while protecting the data privacy of individuals and enterprises, enhancing the scalability and generalization ability of the basic model, and improving the training efficiency of the model.

[0142] 2. The distributed training method for the basic model provided in this embodiment can effectively learn the topological representation information of the basic model in the federated learning network, which helps to alleviate the problem of poor model training effect caused by the statistical heterogeneity of scattered data during the basic model training process, and improves the robustness of the basic model under the distributed collaborative training framework.

[0143] Finally, it should be noted that the distributed training method of the basic model described above can be applied to a distributed training system. In other words, the distributed training system described above can implement the training process of the basic model based on this distributed training method. To avoid redundancy, the specific implementation process of the training system is not described in detail. You can refer to the training method process provided in this embodiment.

[0144] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed training system for a basic model, characterized in that, include: Multiple smart terminals, multiple edge servers, and a cloud server, wherein one edge server is communicatively connected to multiple subordinate smart terminals, and the multiple edge servers are communicatively connected to the cloud server; The cloud server is used to manage the multiple smart terminals and multiple edge servers to perform multiple rounds of iterative training on the initial basic model in order to obtain a trained basic model. In one round of iterative training: The cloud server is used to distribute the initial basic model to various smart terminals; The smart terminal is used to train the initial basic model using a local training set to obtain a trained personalized model; and to send the model parameters of the personalized model to the corresponding edge server. The edge server is used to calculate the second-order similarity between the personalized model parameters of the current round and the new initial base model. Based on the second-order similarity, it constructs a graph topology for the computing nodes. It uses a graph convolutional network (GCN) to learn the distribution patterns of data features and updates the model parameters of the received personalized models through graph topology reconstruction. Based on the clustering results, it generates a graph topology-based aggregation operator. The received base model parameters are divided into tensors according to Transformer blocks and input into the aggregation operator. The aggregation operator performs preliminary personalized aggregation processing on the model parameters to be aggregated, obtaining multiple aggregated local models. Finally, it sends the model parameters of these multiple local models to the cloud server. The cloud server is further configured to assign corresponding weight coefficients to each local model according to the size of the training set corresponding to each local model, and to aggregate the model parameters of the multiple local models according to the weight coefficients corresponding to each local model to obtain a new initial base model; and to determine whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model; otherwise, the new initial base model is sent to the smart terminal for the next round of iterative training.

2. The distributed training system for the basic model according to claim 1, characterized in that, The smart terminal is used to train the initial base model using a local training set to obtain a trained personalized model, including: Set a corresponding loss function for the initial base model; The initial base model is trained using a local labeled training set until the loss function is minimized, resulting in a trained personalized model.

3. The distributed training system for the basic model according to claim 1, characterized in that, The smart terminal is used to train the initial base model using a local training set to obtain a trained personalized model, including: The edge server receives model parameters of personalized models from multiple smart terminals and aggregates the received model parameters of multiple personalized models to obtain an aggregated local model.

4. The distributed training system for the basic model according to claim 1, characterized in that, The cloud server is also used to determine whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model, including: When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or Determine whether the number of iterations has reached the set number. If so, stop the iteration training and determine the new initial base model obtained in this iteration as the trained base model.

5. A distributed training method for a basic model, characterized in that, A distributed training system applied to the base model to perform multiple rounds of iterative training on the initial base model in order to obtain a well-trained base model; The distributed training system of the basic model includes multiple smart terminals, multiple edge servers and a cloud server, wherein one edge server is communicatively connected to multiple subordinate smart terminals, and the multiple edge servers are communicatively connected to the cloud server. The distributed training method includes: During one round of iterative training, the cloud server distributes the initial basic model to each smart terminal; In each of the aforementioned smart terminals, the initial basic model is trained using a local training set to obtain a trained personalized model; and the model parameters of the personalized model are sent to the corresponding edge server. In each edge server, the second-order similarity between the personalized model parameters of the current round and the new initial base model is calculated. Based on the second-order similarity, a graph topology of the computing nodes is constructed. The distribution pattern of data features is learned using a graph convolutional network (GCN). The model parameters of the received personalized models are updated by graph clustering in a graph topology reconstruction manner. An aggregation operator based on graph topology is generated based on the clustering results. The received base model parameters are tensor-segmented according to Transformer blocks and input into the aggregation operator. The aggregation operator performs preliminary personalized aggregation processing on the model parameters that need to be aggregated, resulting in multiple aggregated local models. The model parameters of the multiple local models are then sent to the cloud server. In the cloud server, each local model is assigned a corresponding weight coefficient based on the size of the training set corresponding to each local model. Based on the weight coefficients corresponding to each local model, the model parameters of the multiple local models are aggregated to obtain a new initial base model. It is then determined whether the iterative training termination condition is met. If so, the new initial base model obtained in this round of training is determined as the trained base model. Otherwise, the new initial base model is sent to each of the smart terminals for the next round of iterative training.

6. The distributed training method for the basic model according to claim 5, characterized in that, The step of training the initial base model using a local training set to obtain the trained personalized model includes: Set a corresponding loss function for the initial base model; The initial base model is trained using a local labeled training set until the loss function is minimized, resulting in a trained personalized model.

7. The distributed training method for the basic model according to claim 5, characterized in that, The process of aggregating the model parameters of the multiple local models to obtain a new initial base model includes: The cloud server assigns corresponding weight coefficients to each local model based on the size of the training set corresponding to each local model. Based on the weight coefficients corresponding to each local model, the model parameters of the multiple local models are aggregated to obtain a new initial base model.

8. The distributed training method for the basic model according to claim 5, characterized in that, The step of training the initial base model using a local training set to obtain the trained personalized model includes: The edge server receives model parameters of personalized models from multiple smart terminals and aggregates the received model parameters of multiple personalized models to obtain an aggregated local model.

9. The distributed training method for the basic model according to claim 5, characterized in that, The step of determining whether the iterative training termination condition is met, and if so, determining the new initial base model obtained in this round of training as the trained base model, includes: When it is determined whether the loss function of the new initial base model meets the preset value, the iterative training is stopped, and the initial base model obtained from this iteration is determined as the trained base model; or Determine whether the number of iterations has reached the set number. If so, stop the iteration training and determine the new initial base model obtained in this iteration as the trained base model.

Images