Distributed training method and system for models

Abstract

The application relates to the field of model training, and provides a distributed training method and system of a model. The method comprises the following steps: a cloud end acquires sample data, and performs first training on a previously constructed backbone model and a probe model according to the sample data; the cloud end issues the backbone model and the probe model after the first training to each client associated with the backbone model; each client performs secondary training on the backbone model after the first training according to pre-stored client sample data, obtains a backbone model after the secondary training, and obtains feature value scores of feature extraction layers corresponding to the backbone model after the secondary training based on the probe model after the first training; the cloud end receives the backbone model after the secondary training uploaded by each client, and performs aggregation according to the scores of the feature extraction layers and the received backbone models after the secondary training based on a preset aggregation principle, to obtain a distributed model. The method solves the problem that the performance of a model is poor due to the fact that real data of a client cannot be acquired during model training.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of model training, in particular to a distributed training method and system of a model. BACKGROUND

[0002] Nowadays in the electronic and electrical architecture of the vehicle end, the computing power for neural network calculation is no longer the bottleneck of the system. Combined with other application scenarios, neural networks have been widely used to build controlled object models or provide more advanced control and diagnosis strategies based on complex sensor signals. These network models are trained offline and mostly used for inference when used in the vehicle end. In addition, since the controlled object is usually not fixed and often ages over time, such factors will cause the degradation of the performance of the neural network model. Further, in view of the need for national information security protection, relevant policies will limit the large amount of data from the vehicle end to be transmitted back to the cloud through the vehicle gateway, so that the network model in the cloud cannot be adjusted based on the data from the vehicle end, thereby limiting the performance of the network model. Therefore, it is necessary to provide a distributed training method and system of a model. SUMMARY

[0003] The purpose of the present application is to provide a distributed training method and system of a model. To solve the problem of poor model performance in the prior art due to the inability to obtain real data from the vehicle end during model training.

[0004] In a first aspect, a distributed training method of a model is provided, comprising:

[0005] The cloud obtains sample data and performs first training on a pre-constructed backbone model and a probe model according to the sample data; wherein at least one feature extraction layer of the backbone model corresponds to a probe model;

[0006] The cloud distributes the first trained backbone model and the probe model to each client associated with the backbone model;

[0007] Each client performs secondary training on the first trained backbone model according to pre-stored client sample data, obtains a second trained backbone model, and obtains the feature value score of the corresponding feature extraction layer in the second trained backbone model based on the first trained probe model;

[0008] The cloud receives the second trained backbone model uploaded by each client, and performs aggregation based on the feature value score of the corresponding feature extraction layer and the received second trained backbone model according to a preset aggregation principle, to obtain a distributed model.

[0009] In a second aspect, a distributed training system of a model is provided, comprising a cloud and a plurality of clients, wherein the cloud and the clients are in communication connection.

[0010] The cloud end comprises:

[0011] a first training module, configured to acquire sample data, and perform first training on a pre-constructed backbone model and a probe model according to the sample data; wherein at least one feature extraction layer of the backbone model corresponds to a probe model;

[0012] a first communication module, configured to distribute the first-trained backbone model and the probe model to each client associated with the backbone model, and receive the second-trained backbone model uploaded by each client;

[0013] a distributed model acquisition module, configured to aggregate the feature value scores of the corresponding feature extraction layers and the received second-trained backbone models based on a preset aggregation principle, to obtain a distributed model;

[0014] each client comprises:

[0015] a second training module, configured to perform second training on the first-trained backbone model according to pre-stored client sample data, to obtain a second-trained backbone model, and obtain the feature value scores of the corresponding feature extraction layers in the second-trained backbone model based on the first-trained probe model;

[0016] a second communication module, configured to receive the first-trained backbone model and the probe model distributed by the cloud end, and upload the second-trained backbone model to the cloud end.

[0017] The distributed training method and system of the model of the application, the cloud uses sample data to first train the backbone model and the probe model, after the training is completed, the first trained backbone model and the probe model are issued to each client connected with the cloud. Each client can perform secondary training on the first trained backbone model issued by the cloud based on the client sample data stored by itself, fine-tune the first trained backbone model based on the self data, and obtain the secondary trained backbone model, thereby improving the individualization performance of the model. The client uses the first trained probe model to score the corresponding feature extraction layer of the secondary trained backbone model, and obtains the feature value score of the corresponding feature extraction layer. The cloud receives the secondary trained backbone model uploaded by the client, and according to the aggregation principle, the same feature extraction layer of each secondary trained backbone model is weighted and aggregated based on the scores of the same feature extraction layer in each secondary trained backbone model, and the distributed model is obtained. The first training of the backbone model is performed in the cloud, and the secondary training of the first trained backbone model is performed in the client, so that the secondary trained backbone model can fully represent the characteristics of the client, and the problem that the training result is inaccurate due to the inability to obtain the client data when the model is trained in the cloud is solved. In addition, through the probe model, the feature value score of the feature extraction layer is obtained, and the feature value score is processed in a weighted manner, which greatly stabilizes the distributed training process. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor under the premise of the drawings:

[0019] Figure 1 An application environment schematic diagram of the distributed training method of the model in an embodiment of the application is shown;

[0020] Figure 2 A flowchart schematic diagram of the distributed training method of the model in an embodiment of the application is shown;

[0021] Figure 3 A flowchart schematic diagram of screening the client in an embodiment of the application is shown;

[0022] Figure 4 Another flowchart schematic diagram of screening the client in an embodiment of the application is shown;

[0023] Figure 5 A flowchart schematic diagram of obtaining the first feature value score in an embodiment of the application is shown;

[0024] Figure 6A flowchart for acquiring the second score and the third score of the feature value in an embodiment of the present application is shown.

[0025] Figure 7 A flowchart for acquiring the parameter of the feature extraction layer in an embodiment of the present application is shown.

[0026] Figure 8 A flowchart for acquiring the mask in an embodiment of the present application is shown.

[0027] Figure 9 A structure block diagram of the distributed training system of the model in an embodiment of the present application is shown.

[0028] Figure 10 A comparison diagram of the distributed model and the FedAvg loss function value in 10 fusion cycles of a client in a group in an embodiment of the present application is shown.

[0029] Figure 11 A comparison diagram of the average of the loss function values of the models in the 5th fusion cycle in multiple tests when there is no pruning in an embodiment of the present application is shown.

[0030] Figure 12 A comparison diagram of the average of the loss function values of the models in the 5th fusion cycle in multiple tests when there is 20% pruning in an embodiment of the present application is shown.

[0031] Figure 13 A comparison diagram of the loss function convergence of the algorithms in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0033] The distributed training method of the model provided in the embodiments of the present application can be applied to, for example, Figure 1application environment, wherein the client communicates with the cloud through the network. The cloud uses sample data to first train the backbone model and the probe model, and after the training is completed, the first trained backbone model and the probe model are issued to each client connected to the cloud. Each client can perform secondary training on the first trained backbone model issued by the cloud based on the client's own stored client sample data to obtain a second trained backbone model. During the secondary training, the client fine-tunes the parameters of the first trained backbone model based on its own data, greatly improving the individualization performance of the model. Then the client uses the first trained probe model to score the corresponding feature extraction layer of the second trained backbone model to obtain the feature value score of the feature extraction layer. The cloud receives the second trained backbone model uploaded by the client, and according to the aggregation principle, based on the scores of the corresponding feature extraction layers in the second trained backbone models of each client, the same feature extraction layer of each second trained backbone model is weighted and aggregated to obtain a distributed model. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, vehicle terminals and portable wearable devices. The cloud can be implemented by an independent server or a server cluster composed of multiple servers. The application will be described in detail below through specific embodiments.

[0034] Referring to Figure 2 , as shown in the figure, Figure 2 A flowchart of a distributed training method of a model provided by an embodiment of the application is shown, comprising the following steps:

[0035] S1, the cloud obtains sample data, and performs first training on a pre-constructed backbone model and a probe model according to the sample data; wherein at least one feature extraction layer of the backbone model corresponds to a probe model.

[0036] In the application, the client is taken as an example of a vehicle terminal, but it can be understood that the client can also be other types, such as a mobile phone terminal, etc., which are not limited here. For different training tasks, the structure of the backbone model is different, and the skilled person in the art can adaptively construct the required backbone model according to actual needs. The cloud inputs a large amount of sample data obtained to the constructed backbone model to perform first training (i.e. pre-training) on the backbone model, to obtain a first trained backbone model, wherein the backbone model comprises a deep feature extraction part and a task part connected in turn. The deep feature extraction part comprises N L feature extraction layers from shallow to deep, and each feature extraction layer extracts feature information of input data through a corresponding parameter feature extraction filter. For example, the i-th layer feature extraction layer outputs Each feature map (i.e., feature channel) contains two 2D feature maps, where the feature extraction filters include, but are not limited to, convolutional filters, pooling filters, or MHA (Multi-head Attention) structures. The task part can consist of one or more fully connected layers, which classify the input data by mapping the extracted features to the sample label space. After the initial training, the probe model is connected to at least one feature extraction layer of the backbone model for initial training, resulting in the probe model after initial training. Feature extraction layers in the backbone model with fewer than a preset layer number threshold can be designated as shallow feature extraction layers, while those with more than or equal to the threshold can be designated as deep feature extraction layers. Different strategies are used to add probe models to the shallow and deep feature extraction layers. For example, probe models can be selectively set for each layer in the shallow feature extraction layers, while probe models can be set for every layer in the deep feature extraction layers, thus improving training efficiency while ensuring training effectiveness. The feature extraction layer of this invention includes a feature extraction filter and a nonlinear activation layer connected in sequence. The nonlinear activation layer nonlinearly processes the features output by the feature extraction filter to enhance the expressive power of the backbone model.

[0037] Furthermore, since the Batchnorm layer is generally used to normalize the data distribution between layers, ensuring that the output of each layer conforms to an N(0,1) normal distribution as much as possible, thereby aiding network convergence, in another embodiment of this invention, the feature extraction layer further includes a Batchnorm layer to improve operating efficiency. Since the data features learned by each layer's feature extraction filter can be represented by a feature map, for the i-th layer's feature extraction layer, the output of the nonlinear activation layer can be... Each 2D feature map is input into the probe model of the i-th layer, thereby enabling the analysis of the relevant characteristics of the feature channels of that layer. The probe model consists of a global average pooling layer and a linear classifier. First, global average pooling (GAP) is used to convert each 2D feature map into a floating-point value. Each floating-point value is processed by a linear classifier to obtain a floating-point value between 0 and 1, which is then used as the final output feature score of the i-th layer probe model. The type of linear classifier includes, but is not limited to, logistic regression classifiers, least squares linear classifiers, or perceptrons. However, considering the universality and computational speed requirements of the algorithm, in one embodiment of this invention, a logistic regression classifier is used. Furthermore, after the initial training, the parameters of the backbone model after the initial training, the probe model after the initial training, and the i-th layer feature extraction layer of the backbone model after the initial training are processed by GAP. The probe model outputs a feature value score corresponding to each feature extraction layer, including a feature value score corresponding to the training data and a feature value score corresponding to the test data.

[0038] In an embodiment of the present application, the cloud obtains sample data, including: the cloud inputs pre-stored cloud data into a pre-trained clustering model, clusters the cloud data, and obtains sample data corresponding to the backbone model. When the backbone model is trained for the first time, because different sample data correspond to different backbone models, the cloud data is first clustered by the clustering model to obtain a plurality of different types of cloud sub-data, wherein the cloud data is mixed with a plurality of different types of cloud sub-data. The cloud searches for cloud sub-data corresponding to the current backbone model from the clustering result based on the current backbone model, as sample data. The sample data is input into the backbone model for first-time training. In this way, the current backbone model can be analyzed based on the sample data, so that the backbone model after first-time training is more personalized, thereby improving the performance of the model.

[0039] Further, in order to comprehensively consider the related characteristics of each feature channel in the backbone model, so that the finally obtained distributed model has good robustness and fully reflects the characteristics of each client, in an embodiment of the present application, each feature extraction layer of the backbone model corresponds to a probe model. By connecting each feature extraction layer of the backbone model with a probe model, the result of pruning the backbone model after secondary training is more accurate. For example, if the task of the backbone model is an N cls classification task, and the backbone model has N L feature extraction layers, there are N cls ×N L probe models, each probe model corresponds to one of the feature map categories output by each feature extraction layer, and a 0-1 classification label is used to train the probe model. That is, if the probe model is for the i-th type of data, i∈(1,…,N cls ​​​​For the feature map belonging to the i-th class, the output of the probe model opposite to the feature map is 1, and the output of the probe model is 0 for the feature map of other classes. The reason for innovatively using the 0-1 classification design instead of the general multi-class output design is that the shallow feature extraction layer generally focuses on local features, and such features have little semantic attribute, i.e., little description ability for the final multi-class classification task. Therefore, even if the final multi-class output of the backbone model has converged well, the feature value score output by the multi-class output design on the shallow feature extraction layer is generally not good. The 0-1 classification design can avoid such problems, because if the backbone network is sparse enough, each feature extraction layer output has a specific feature that can be related to the final class, so the 0-1 classification binary classification design generally has a good feature value score. More conducive to subsequent pruning, based on the feature value score, setting the related parameters of the feature extraction layer, so as to make the pruning result more accurate.

[0040] When the backbone model is a regression model, for example, predicting the motion trajectory of a vehicle, the output of the backbone model is not a class in the multi-class, but the coordinate value of the vehicle at a certain time. In order to refine the application scenario and improve the accuracy of the model, in an embodiment of the present application, the clustering model is trained by the cloud based on the unsupervised learning method. The cloud data is unsupervisedly clustered into several cloud sub-data of different categories by the clustering model, so that each category of cloud sub-data corresponds to a backbone model, thereby making the output result of the backbone model more accurate. Illustratively, when predicting the trajectory, the cloud data is clustered into three different categories of cloud sub-data, i.e., uniform straight running condition, rapid acceleration and rapid deceleration straight running condition, and lane changing condition. Further, in order to realize the unification of the regression task and the classification task, when the backbone model is a regression model, the final output result of the regression model can be divided into several different ranges, thereby converting the regression task into a classification task. It can be understood that, in order to improve the accuracy of the backbone model, the classification task can also use the clustering model to pre-classify the sample data.

[0041] S2, the cloud end sends the first trained backbone model and the probe model to each client associated with the backbone model.

[0042] Since the cloud uses a large amount of sample data for training, the demand for powerful computing power and storage capacity in the training process of the artificial intelligence method can be met. However, these sample data are different from the sample data of the vehicle end, and cannot fully reflect the current characteristics of the vehicle end. To solve this problem and make the distributed model more suitable for the characteristics of the vehicle end, the cloud end downloads the first trained backbone model, the parameters of the backbone model and the data of the probe model to each vehicle end associated with the cloud end. So that different vehicle ends can perform secondary training on the first trained backbone model based on their own vehicle end sample data, wherein the cloud end is in communication connection with multiple vehicle ends.

[0043] In an embodiment of the present application, before the cloud end downloads the first trained backbone model and the probe model to each client associated with the backbone model, the cloud end further comprises: the cloud end filters each client based on the clustering model, and associates each filtered client with the backbone model. Considering that different classification results correspond to different backbone models, if the backbone model has a clustering model, when one of the sessions of the distributed training is started, the cloud end selects the vehicle end suitable for participating in the training of the current backbone model based on the clustering model for secondary training. Specifically, in an embodiment of the present application, the cloud end filters each client based on the clustering model, and associates each filtered client with the backbone model, comprising:

[0044] S021, the cloud end downloads the clustering model to each client;

[0045] S022, each client inputs pre-stored client data into the clustering model, clusters the client data, and initiates an association request to the cloud end if the clustered data has client sample data corresponding to the backbone model;

[0046] S023, the cloud end filters the corresponding client associated with the backbone model from the association request uploaded by each client.

[0047] At the start of the distributed training session, the server in the cloud end sends the current training task and the corresponding clustering model to the associated vehicle end. In the vehicle end, the vehicle end data stored is input into the clustering model sent by the cloud end. All data in the vehicle end cache is quickly inferred through the clustering model, so as to obtain different vehicle end sub-data. For example, the clustering model divides the vehicle end data into uniform speed working conditions and variable speed working conditions, and two different types of vehicle end sub-data. Based on the divided vehicle end sub-data, the vehicle end determines whether the vehicle end sub-data matches the training task sent by the cloud end. If it matches, the matching vehicle end sub-data is taken as vehicle end sample data, which is considered to correspond to the current backbone network, and it is determined that it can participate in secondary training. Otherwise, it cannot participate in secondary training. For the vehicle end that can participate in secondary training, an association request is sent to the cloud end. After receiving the association request of the vehicle end, the cloud end takes the vehicle end sending the association request as the screened vehicle end. After screening, the cloud end sends the training task to the screened vehicle end again, so that each screened vehicle end participates in the secondary training of the backbone model. The training task includes the first training backbone model parameters, the first training probe model, the GAP of the first training backbone model i-th layer feature extraction layer, C Li The application selects the vehicle end that adapts to the current backbone model for secondary training through the use of the clustering model, greatly improving the accuracy of the model. In addition, through the clustering model, the heterogeneity of data can be effectively solved, so as to cope with the characteristics of the vehicle end data Non-IID (NonIndependent and Identically Distributed).

[0048] In another embodiment of the application, the cloud end screens each client based on the clustering model and associates each screened client with the backbone model, comprising:

[0049] S121, the cloud end sends the clustering model to each client;

[0050] S122, each client inputs the pre-stored client data into the clustering model, clusters the client data, obtains the clustering result, and uploads the clustering result to the cloud end;

[0051] S123, the cloud end selects the client whose clustering result contains client sample data corresponding to the backbone model from the clustering result uploaded by each client as the client associated with the backbone model.

[0052] After the cloud server distributes the clustering model to each vehicle, the vehicle inputs its locally stored data into the clustering model for clustering, thus obtaining the clustering results. For example, the vehicle data, after being processed by the clustering model, yields the following clustering results: 90% for constant-speed data and 10% for variable-speed data. The vehicle uploads the clustering results to the cloud. The cloud then selects the clustering result corresponding to the currently trained backbone model from the received clustering results, and uses the vehicle corresponding to this clustering result as the selected vehicle to participate in the secondary training of the backbone model.

[0053] Furthermore, for the backbone model without a clustering model, the cloud server distributes the training task of the current backbone model to each associated vehicle terminal. Each vehicle terminal, based on this training task, determines whether its own vehicle terminal data meets the requirements of the training task, and if it does, it is selected as a filtered vehicle terminal. Further, the cloud stores the filtered vehicle terminal nodes in the cloud database as a cache list, so that when a subsequent session starts, it can directly search for the stored vehicle terminal nodes in the cache list and use the found vehicle terminal nodes as potential participants in the current training, thereby reducing network bandwidth usage and improving model training speed. However, considering the data drift problem, the vehicle terminal nodes found by the cloud from the cache list may not be the vehicle terminals required for the current training, resulting in the corresponding vehicle terminal data not matching the current backbone model, severely reducing the model's accuracy. To solve the above problem and improve the data hit rate, this invention increases the storage space of the cache list and, based on the historical data of each previously associated vehicle terminal, evaluates whether the current vehicle terminal meets the training requirements of the backbone model to be trained according to a preset evaluation principle. Furthermore, to achieve iterative data updates, this invention uses a clustering model to detect whether the relevant vehicle-side data meets the training requirements during the interval between two sessions. If the current vehicle-side data does not meet the training requirements, other relevant vehicle-side data is promptly added. In addition, before the cloud distributes the clustering model to each client, it checks whether the current clustering model has been updated based on the previously distributed model. If no update has occurred, the clustering model does not need to be distributed to previously sent clients; if an update has occurred, the updated clustering model is sent to all associated clients. This further saves bandwidth usage and effectively improves data transmission rate.

[0054] S3. Each client performs secondary training on the backbone model after the first training based on the pre-stored client sample data, and obtains the backbone model after secondary training. Based on the probe model after the first training, the feature value scores of the corresponding feature extraction layer in the backbone model after secondary training are obtained.

[0055] The sample data of the vehicle end is pre-stored in the cache of the vehicle end locally, and the capacity of the cache is smaller than that of the cloud due to the limitation of the hardware architecture. Therefore, the number of the sample data of the vehicle end is much smaller than that of the cloud, and the secondary training of the backbone model after the first training is implemented by using the sample data of the vehicle end to fine-tune the backbone model after the first training, so as to obtain the backbone model after the secondary training. Then, the vehicle end uses the probe model after the first training to calculate the feature value scores of the feature extraction layer corresponding to the probe model in the backbone model after the secondary training, so as to be aggregated for subsequent processing. The backbone model after the secondary training is obtained by using the sample data of the vehicle end, which can not only improve the individualization of the model, but also reduce the memory occupation of the client, and effectively solve the problem of insufficient computing power of the client in distributed training.

[0056] In an embodiment of the present application, the feature value scores of the corresponding feature extraction layer in the backbone model after the secondary training are obtained based on the probe model after the first training, and the method comprises the following steps:

[0057] S31, the client inputs the client sample data into the backbone model after the secondary training, and inputs the multiple feature maps output by the feature extraction layer into the probe model corresponding to the feature extraction layer, respectively performs global average pooling on each feature map to obtain multiple first global averages; wherein each first global average corresponds to one feature map;

[0058] S32, the client inputs each first global average into the linear classifier of the probe model, and obtains the feature value first score of the current feature extraction layer based on the preset linear classification rule.

[0059] The vehicle end inputs the vehicle end sample data stored by the vehicle end into the backbone model after the first training for secondary training, and obtains the backbone model after the secondary training after fine-tuning of the training task for a preset number of times. The process of the secondary training is the same as that of the first training of the cloud, which will not be repeated here. It should be noted that the secondary training is only for the backbone model after the first training, and the probe model will not be trained, so that the stability of the current training can be measured based on the probe model. After the secondary training is completed, the vehicle end sample data is input into the backbone model after the secondary training, and the feature extraction layer in the i-th layer outputs the feature map corresponding to the vehicle end sample data. Then, the feature map output by the current feature extraction layer is input into the corresponding probe model, and each feature map is converted into a first global average by global average pooling, wherein the first global average is represented by a floating point value. The first global average of the feature extraction layer is input into the linear classifier, and the feature value first score of the i-th layer feature extraction layer in the vehicle end using the probe model prob is obtained according to the preset linear classification rule. wherein,​​​ It should be noted that different types of linear classifiers correspond to different linear classification rules.

[0060] In an embodiment of the present application, after the first training of the pre-constructed backbone model and the probe model according to the sample data, the method further comprises:

[0061] S101, the cloud inputs the training data in the sample data into the backbone model after the first training, and inputs each feature map output by the feature extraction layer into the corresponding probe model, respectively performs global average pooling on each feature map, and obtains a plurality of second global means;

[0062] S102, the cloud inputs each global mean into the linear classifier to obtain the second score of the feature value of the current feature extraction layer;

[0063] S103, the cloud inputs the test data in the sample data into the backbone model after the first training, and inputs each feature map output by the feature extraction layer into the corresponding probe model, respectively performs global average pooling on each feature map, and obtains a plurality of third global means;

[0064] S104, the cloud inputs each third global mean into the linear classifier to obtain the third score of the feature value of the current feature extraction layer.

[0065] When the backbone model is first trained, the sample data can be divided into training data and test data according to a preset proportion. The cloud inputs the training data into the backbone model after the first training, and outputs corresponding feature maps of the training data in the i-th layer feature extraction layer. Then, the cloud inputs the feature maps output by the current feature extraction layer into the backbone model after the first training and the probe model corresponding to the i-th layer feature extraction layer, and converts each feature map into a second global mean by global average pooling, wherein the second global mean is represented by a floating point value. The cloud inputs the second global means of the i-th layer feature extraction layer into the linear classifier, and obtains the second score of the feature value of the training data tr using the probe model prob in the i-th layer feature extraction layer glb of the cloud according to the preset linear classification rule. Correspondingly, the cloud inputs the test data into the backbone model after the first training, and outputs corresponding feature maps of the training data in the i-th layer feature extraction layer. Then, the cloud inputs the feature maps output by the current feature extraction layer into the backbone model after the first training and the probe model corresponding to the i-th layer feature extraction layer, and converts each feature map into a third global mean by global average pooling, wherein the third global mean is represented by a floating point value. The cloud inputs the third global means of the i-th layer feature extraction layer into the linear classifier, and obtains the third score of the feature value of the test data te using the probe model prob in the i-th layer feature extraction layer glb of the cloud according to the preset linear classification rule. ​​​​​The feature map is input into the backbone model after the first training, and is input into the probe model corresponding to the i-th feature extraction layer. The feature map is converted into a third global mean value by global average pooling, wherein the third global mean value is represented by a floating point value. The third global mean value of the feature extraction layer is input into the linear classifier to obtain the feature value third score of the test data te of the cloud glb i-th feature extraction layer using the prob probe model.

[0066] S4, the cloud receives the secondary trained backbone model uploaded by each client, and performs aggregation based on a preset aggregation principle according to the feature value score of the corresponding feature extraction layer and the received secondary trained backbone model, to obtain a distributed model.

[0067] Each vehicle end uploads the secondary trained backbone model to the cloud, and the same feature extraction layer of each secondary trained backbone model is aggregated based on the feature value score of each same feature extraction layer according to the preset aggregation principle, to obtain the parameters of the same feature extraction layer in the distributed model. The above process is repeated until all feature extraction layers are aggregated, to obtain the distributed model. Specifically, in an embodiment of the present application, for the same feature extraction layer of the secondary trained backbone model: the aggregation based on the preset aggregation principle according to the feature value score of the corresponding feature extraction layer and the received secondary trained backbone model comprises:

[0068] S41, the cloud obtains the weight corresponding to each feature extraction layer based on the feature value score of the same feature extraction layer in each secondary trained backbone model.

[0069] S42, the cloud respectively weights and aggregates the weight corresponding to each feature extraction layer and the parameters of the corresponding feature extraction layer, to obtain the parameters of the same feature extraction layer in the distributed model.

[0070] For the i-th feature extraction layer, the cloud receives the feature value first score of the current feature extraction layer uploaded by each vehicle end and compares with the feature value second score of the same feature extraction layer in the backbone model after the first training Based on the comparison result of the feature value first score and the feature value second score, different weights are selected correspondingly. The secondary trained backbone model of each vehicle end is weighted and summed with the weight of the current feature extraction layer and the parameters of the corresponding feature filter of the layer, to obtain the parameters of the current feature extraction layer in the distributed model.

[0071] Specifically, in an embodiment of the present application, the cloud obtains the weight corresponding to each feature extraction layer based on the feature value score of the same feature extraction layer in each secondary trained backbone model, comprising:​​

[0072] S411. Select one of the feature extraction layers in the backbone model after secondary training;

[0073] S412. Determine whether the first feature value score corresponding to the feature extraction layer is greater than or equal to the second feature value score;

[0074] S413. If the first score of the feature value is greater than or equal to the second score of the feature value, then the weight of the feature extraction layer is set to the first value.

[0075] S414. If the first feature value score is less than the second feature value score, then the weight of the feature extraction layer is set to a second value; wherein the second value is less than the first value.

[0076] S415. Select the same feature extraction layer in another secondary training backbone model, and repeat steps S412 to S414 until the same feature extraction layer in each secondary training backbone model has been selected, and obtain the weights corresponding to the same feature extraction layer in each secondary training backbone model.

[0077] For each vehicle terminal after screening, after secondary training of the backbone model following the initial training using vehicle terminal sample data, each vehicle terminal has a secondary trained backbone model. The i-th feature extraction layer in one of the secondary trained backbone models is selected, and the first score of the feature values ​​of that feature extraction layer is obtained. Then, obtain the second score of the feature values ​​of this feature extraction layer from the corresponding backbone model after the first training. First score of the comparison feature value With the second fraction of the eigenvalue If the first fraction of the eigenvalue Greater than or equal to the second fraction of the eigenvalue This indicates that the backbone model can learn relevant features better after this second training, therefore the weights of this feature extraction layer are adjusted. Set it to the first value. Conversely, if the eigenvalue is the first score... Second fraction less than eigenvalue This indicates that the backbone model can learn the relevant features better after the first training, therefore the weights of the feature extraction layer are adjusted accordingly. The first and second values ​​are set to a second value, where the first and second values ​​can be adaptively set to relevant values, as long as the first value is greater than the second value. Preferably, the first value is 1, and the second value is as shown in formula (1):

[0078]

[0079] By setting the first value as 1 and the second value as dynamically changing data, the influence of the characteristic value first score, the characteristic value second score and the characteristic value third score on the weight is fully exhibited to realize the purpose of stabilizing the entire aggregation process through the output of the probe model. The above process is repeated until the same layer feature extraction layer in the backbone model after secondary training is selected, and the aggregation result of the layer feature extraction filter is obtained after aggregation processing and as the parameters of the same feature extraction layer in the distributed model, wherein f is the parameter of the feature extraction filter.

[0080] In order to reduce the network bandwidth overhead and reduce the parameter amount of the backbone model after secondary training, in an embodiment of the present application, after the client performs secondary training on the backbone model after primary training to obtain the backbone model after secondary training according to the pre-stored client sample data, the client further performs compression pruning on the backbone model after secondary training based on a pre-set quantization pruning strategy. Through compression pruning, unnecessary network parameters are pruned, so that the backbone model after secondary training uploaded by the vehicle end to the cloud end is smaller in size and the transmission speed is faster.

[0081] Further, in an embodiment of the present application, for one of the feature extraction layers of the backbone model after secondary training of one of the clients: the compression pruning of the backbone model after secondary training based on the pre-set quantization pruning strategy includes:

[0082] S4151, the client sorts the first global mean values of the feature extraction layer in the backbone model after secondary training, and screens the sorted first global mean values based on a pre-set screening rule to obtain a screened first global mean value set;

[0083] S4152, the client sorts the second global mean values of the same feature extraction layer in the backbone model after primary training, and screens the sorted second global mean values based on the screening rule to obtain a screened second global mean value set;

[0084] S4153, the client sorts the third global mean values of the same feature extraction layer in the backbone model after primary training, and screens the sorted third global mean values based on the screening rule to obtain a screened third global mean value set;

[0085] S4154, the client takes the intersection of the screened first global mean value set, the screened second global mean value set and the screened third global mean value set as a mask set;

[0086] S4155, the client performs pruning processing on the current feature extraction layer of the backbone model after secondary training based on the mask set.

[0087] For one of the backbone models after secondary training, the vehicle-end loc will use the i-th feature extraction layer in the backbone model after secondary training. First global mean Sort the data in descending order and discard the last k values. The value of k can be adaptively set by those skilled in the art based on the model's accuracy requirements, and is not limited here. After discarding the last k first global means, a filtered set of first global means is obtained. This filtered set of first global means contains... The first global mean. Accordingly, the vehicle-side will use the training data tr in the backbone model after the first training of the corresponding cloud-based GLB to perform feature extraction on the i-th layer. The second global mean Sort the values ​​in descending order and discard the last k values ​​to obtain the second global mean set after filtering. Similarly, on the vehicle side, after the corresponding cloud-based GLB is trained for the first time, the test data te is used to refine the i-th feature extraction layer. The third global mean Sort the values ​​in descending order and discard the last k values ​​to obtain the third global mean set after filtering. The system calculates the intersection of the first, second, and third global mean sets after filtering, and uses this intersection as a mask set. The data in the mask set represents the channels that all three datasets deem necessary to prune for the current feature extraction layer. The parameter values ​​of the corresponding feature filters in the mask set are set to 0, thus pruning a set of feature filters in the current feature extraction layer. During backpropagation, the parameters of the pruned filters can be uniformly encoded; for example, only the index of the feature extraction filter is transmitted, and only the parameters of the uncropped filters need to be transmitted, rather than all the parameters of the feature extraction filters. This greatly reduces the amount of data transmitted and improves training efficiency. This process is repeated until all feature extraction layers have been pruned, achieving pruning of the backbone network after each secondary training iteration.

[0088] Furthermore, in one embodiment of the present invention, to increase the sparsity of the network, L1 regularization is used for each feature extraction layer during the initial cloud-based training of the backbone model and the secondary training of the backbone model on the vehicle-side, thereby increasing the sparsity of different feature maps. In addition, when the later layers of the backbone model contain linear layers, the LoRA algorithm is used to fine-tune the backbone model after the initial training during the secondary training on the vehicle-side, considering only the low-rank portion to further reduce bandwidth.

[0089] Please see Figures 10 to 12In the present application, Cifar10 data and a simplified network based on the VGG network are used to train the distributed 10 classification task. The simplified network of the VGG network is obtained by reducing part of the convolution operation on the basis of the VGG network. The VGG is pre-trained in the cloud, and then distributed training is performed on 50 clients. The 50 clients are divided into 5 groups, each group having 10 clients. For each group of clients: first, a to-be-trained class is randomly selected, and the data of the current to-be-trained class is included in all clients in the group. In addition to the data of the current to-be-trained class, each client in the current group also includes training data of another other class as supplementary data. It should be noted that in the current group of clients, there are two types of supplementary data. Among the 10 clients in the current group, one of the two types of supplementary data is included according to a 7:3 ratio. For example, in the first group, the to-be-trained data class is 1, and the supplementary data class is randomly selected 2, 3 data. Then, among the 10 clients in the current first group, 7 clients contain 1, 2 data, and 3 clients contain 1, 3 data. By comparing the traditional FedAvg algorithm and the pruning aggregation method based on the Probe probe model of the present application, it can be seen that the loss function value of the model distributed training method of the present application is significantly better than that of the traditional FedAvg algorithm. In the present application, the loss function value is represented as sp_merge_mean, the loss function value in the FedAvg algorithm is represented as ave_merge_mean, and the loss function value of the original model is represented as ori_model. Therefore, by using the probe model to obtain the score of the feature extraction layer and based on the weighted processing of the feature extraction layer score, the process of distributed training can be effectively stabilized.

[0090] Further, please refer to Figure 13 In the present application, the NGSim data set and the socalLSTM model are used to do the traffic individual trajectory prediction task. SocalLSTM is different from VGG network, which is a regression network. Compared with the traditional FedAvg algorithm, the federated learning method based on clustering and agent selection proposed in the present application can achieve faster convergence of the loss function curve of distributed training. In the present application, driving cycle cluster is used to show that by clustering the data through the clustering model, the corresponding vehicle end of the backbone model after the first training can be selected to significantly improve the individualization of the model, effectively cope with the characteristics of non-independent and identically distributed vehicle end data, and improve the performance of the model. Thus, it helps to reduce the loss of distributed training faster (fewer communication rounds), and significantly reduces the training bandwidth.

[0091] In the present application, a large amount of sample data is used in the cloud, and first training of the backbone model and the probe model in an offline situation is performed to obtain a first training backbone model and a first training probe model. At an appropriate time, the cloud issues the first training backbone model and the first training probe model to selected clients, and the large SOC of the clients uses local data to perform secondary training on the first training backbone model to realize controllable model fine-tuning. The secondary training backbone model is uploaded to the cloud server for parameter aggregation to obtain a distributed model. Before uploading, the client also performs pruning on the secondary training backbone model based on a preset controllable strategy, thereby effectively reducing network bandwidth consumption and speeding up model uploading. In view of the characteristics of each layer of feature channels in the backbone model, the corresponding filter is adaptively cropped, and the full connection part of the backbone model is cropped. In the process of investigating the characteristics of each layer of feature channels, the method of the probe model is creatively proposed, and based on the probe model, the cropping of the filter is realized, and finally the distributed model suitable for the characteristics of the vehicle end is obtained.

[0092] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of the sequence, and the order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0093] In an embodiment, a distributed training system of a model is provided. The distributed training system of the model corresponds to the distributed training method of the model in the above embodiment one by one. As Figure 9As shown, the distributed training system of the model includes a cloud and each client, and the cloud and each client are in communication connection, wherein the cloud includes: a first training module 11, a first communication module 12, and a distributed model acquisition module 13. Each client includes: a secondary training module 14 and a second communication module 15. The functions of each module are described in detail as follows: the first training module 11 is configured to obtain sample data, and perform first training on a pre-constructed backbone model and a probe model according to the sample data; wherein at least one feature extraction layer of the backbone model corresponds to a probe model. The first communication module 12 is configured to distribute the backbone model and the probe model after first training to each client associated with the backbone model, and receive the backbone model after secondary training uploaded by each client. The distributed model acquisition module 13 is configured to aggregate the feature value scores of the corresponding feature extraction layers and the received backbone model after secondary training based on a preset aggregation principle, to obtain a distributed model. The secondary training module 14 is configured to perform secondary training on the backbone model after first training according to pre-stored client sample data, to obtain a backbone model after secondary training, and obtain the feature value scores of the corresponding feature extraction layers in the backbone model after secondary training based on the probe model after first training. The second communication module 15 is configured to receive the backbone model and the probe model after first training distributed by the cloud, and upload the backbone model after secondary training to the cloud.

[0094] The specific limitations of the distributed training system of the model can be referred to the limitations of the distributed training method of the model in the foregoing, which will not be repeated here. Each module in the above distributed training system of the model can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. The computer program can include the processes of the above-mentioned embodiment methods. It can be understood that, in the present application, since the cloud and the client are involved, the storage medium can be placed in the cloud or the client, which is not limited here. In the embodiments provided in the present application, any reference to memory, storage, database or other medium can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various formats, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the system is divided into different functional units or modules to complete all or part of the functions described above.

[0097] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. Such modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A distributed training method for a model, characterized in that, The method includes: Sample data is acquired from the cloud, and the pre-constructed backbone model and probe model are trained for the first time based on the sample data; wherein, at least one feature extraction layer of the backbone model corresponds to the probe model; The cloud will distribute the backbone model and probe model after the initial training to each client associated with the backbone model; Each client performs secondary training on the backbone model after the first training based on the pre-stored client sample data, and obtains the backbone model after secondary training. Based on the probe model after the first training, the feature value scores of the corresponding feature extraction layer in the backbone model after secondary training are obtained. The cloud receives the secondary-trained backbone models uploaded by each client, and aggregates them according to the feature value scores of the corresponding feature extraction layers and the received secondary-trained backbone models based on the preset aggregation principle to obtain a distributed model. Before the cloud distributes the initial trained backbone model and probe model to each client associated with the backbone model, the method further includes: the cloud filtering each client based on a clustering model and associating each filtered client with the backbone model; wherein the clustering model is pre-trained.

2. The distributed training method for the model according to claim 1, characterized in that, Each feature extraction layer of the backbone model corresponds to a probe model.

3. The distributed training method for the model according to claim 1, characterized in that, The process of acquiring sample data from the cloud includes: inputting pre-stored cloud data into a pre-trained clustering model, clustering the cloud data, and obtaining sample data corresponding to the backbone model.

4. The distributed training method for the model according to claim 3, characterized in that, The clustering model was trained in the cloud using an unsupervised learning method.

5. The distributed training method for the model according to claim 1, characterized in that, The cloud platform filters each client based on the clustering model and associates each filtered client with the backbone model, including: The clustering model is distributed from the cloud to each client. Each client inputs pre-stored client data into the clustering model, clusters the client data, and if there is client sample data in the clustered data that corresponds to the backbone model, it initiates an association request to the cloud. The cloud filters the association requests uploaded by each client and associates the corresponding client with the backbone model.

6. The distributed training method for the model according to claim 1, characterized in that, The method for obtaining feature value scores of the corresponding feature extraction layer in the backbone model after secondary training, based on the probe model after initial training, includes: The client inputs client sample data into the backbone model after secondary training, and inputs multiple feature maps output by the feature extraction layer into the probe model corresponding to the feature extraction layer. Global average pooling is then performed on each feature map to obtain multiple first global means; each first global mean corresponds to one feature map. The client inputs each first global mean into the linear classifier of the probe model, and obtains the first score of the feature value of the current feature extraction layer based on the preset linear classification rules.

7. The distributed training method for the model according to claim 6, characterized in that, After the initial training of the pre-built backbone model and probe model based on the sample data, the process further includes: The cloud inputs the training data from the sample data into the backbone model after the first training, and inputs each feature map output by the feature extraction layer into the corresponding probe model. Global average pooling is then performed on each feature map to obtain multiple second global averages. The cloud inputs the global mean values ​​to the linear classifier to obtain the second score of the feature values ​​of the current feature extraction layer. The cloud inputs the test data from the sample data into the backbone model after the first training, and inputs each feature map output by the feature extraction layer into the corresponding probe model. Global average pooling is then performed on each feature map to obtain multiple third global means. The cloud inputs each third global mean into the linear classifier to obtain the third score of the feature value of the current feature extraction layer.

8. The distributed training method for the model according to claim 7, characterized in that, For the same feature extraction layer of the backbone model after secondary training: the aggregation based on the preset aggregation principle, according to the feature value scores of the corresponding feature extraction layer and the received backbone models after secondary training, includes: Based on the feature value scores of the same feature extraction layer in the backbone model after each secondary training, the weights of each feature extraction layer are obtained in the cloud. The cloud performs weighted aggregation of the weights corresponding to each feature extraction layer and the parameters of the corresponding feature extraction layer to obtain the parameters of the same feature extraction layer in the distributed model.

9. The distributed training method for the model according to claim 8, characterized in that, The cloud-based system obtains the weights corresponding to each feature extraction layer based on the feature value scores of the same feature extraction layer in the backbone model after each secondary training, including: S411. Select one of the feature extraction layers in the backbone model after secondary training; S412. Determine whether the first feature value score corresponding to the feature extraction layer is greater than or equal to the second feature value score; S413. If the first score of the feature value is greater than or equal to the second score of the feature value, then the weight of the feature extraction layer is set to the first value. S414. If the first feature value score is less than the second feature value score, then the weight of the feature extraction layer is set to a second value; wherein the second value is less than the first value. S415. Select the same feature extraction layer in another secondary training backbone model, and repeat steps S412 to S414 until the same feature extraction layer in each secondary training backbone model has been selected, and obtain the weights corresponding to the same feature extraction layer in each secondary training backbone model.

10. The distributed training method for the model according to claim 7, characterized in that, Each client performs secondary training on the backbone model after the first training based on the pre-stored client sample data. After obtaining the backbone model after secondary training, the client also performs compression pruning on the backbone model after secondary training based on a preset quantization pruning strategy.

11. The distributed training method for the model according to claim 10, characterized in that, For one feature extraction layer of the backbone model after secondary training on one of the clients: the compression pruning of the backbone model after secondary training based on a preset quantization pruning strategy includes: After sorting the first global means of each feature extraction layer in the backbone model after secondary training, the client filters the sorted first global means based on preset filtering rules to obtain a set of filtered first global means. After the initial training, the client sorts the second global means of the same feature extraction layer in the backbone model, and then filters the sorted second global means based on the filtering rules to obtain a set of filtered second global means. After the client sorts the third global means of the same feature extraction layer in the backbone model after the first training, it filters the sorted third global means based on the filtering rules to obtain a set of filtered third global means. The client uses the intersection of the filtered first global mean set, the filtered second global mean set, and the filtered third global mean set as the mask set; Based on the mask set, the client performs pruning on the current feature extraction layer of the backbone model after secondary training.

12. A distributed training system for a model, characterized in that, The distributed training method applied to the model of claim 1, wherein the system includes a cloud and various clients, and the cloud and each of the clients are communicatively connected: The cloud includes: The initial training module is used to acquire sample data and perform initial training on the pre-constructed backbone model and probe model based on the sample data; wherein, at least one feature extraction layer of the backbone model corresponds to the probe model; The first communication module is used to distribute the backbone model and probe model after the first training to each client associated with the backbone model, and to receive the backbone model after the second training uploaded by each client. The distributed model acquisition module is used to aggregate the feature value scores of the corresponding feature extraction layer and the received secondary training backbone models based on the preset aggregation principle to obtain a distributed model. Each client includes: The secondary training module is used to perform secondary training on the backbone model after the first training based on the pre-stored client sample data, to obtain the secondary trained backbone model, and based on the probe model after the first training, to obtain the feature value scores of the corresponding feature extraction layer in the secondary trained backbone model. The second communication module is used to receive the backbone model and probe model after the first training from the cloud, and to upload the backbone model after the second training to the cloud.

Images