A heterogeneous federated model adjustment method based on importance sampling

Abstract

The present application relates to a kind of importance sampling-based heterogeneous federal model adjustment method, belong to the field of computing model, method includes: S1, by similar perception layer analysis, the importance of layer is quantitatively analyzed, the importance score is obtained by analyzing each layer of central server, so as to obtain the importance distribution for layer;S2, probability sampling is carried out based on the hierarchical importance distribution obtained in S1, generate sub-model extraction binary mask suitable for heterogeneous client, to improve the matching degree of model structure and resource condition;S3, each client extracts sub-model by binary mask and completes local deployment;S4, introduce TSDL and SDL to carry out local training, and upload and aggregate lora parameter;S5, repeat the steps of S2-S4 until convergence.The present application introduces TSDL, which can alleviate the problem of uneven aggregation caused by local aggregation;Introducing SDL can alleviate the problem of data heterogeneity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computing models, in particular to a heterogeneous federated model adjustment method based on importance sampling. BACKGROUND

[0002] In practical applications, image classification is one of the core tasks of many intelligent terminals. In medical scenarios, medical image classification is widely used for auxiliary diagnosis and is an important part of intelligent medical institutions at all levels. Basic models (such as visual Transformer or CLIP, etc.) are usually pre-trained on large-scale general image datasets and have good generalization ability, which can be transferred to the medical image field and improve the performance in specific diagnosis tasks through fine-tuning.

[0003] However, in real medical deployment environments, medical image data is usually scattered in different hospitals, medical centers or imaging departments, with high privacy sensitivity and strict data compliance requirements. In order to protect patient privacy, each medical center will not upload to the central server for unified training when training the model through existing historical medical images to realize auxiliary diagnosis of diseases. Therefore, in order to enable the model to obtain better ability, under the premise of protecting privacy, a federated learning training framework needs to be used for model training. As a privacy-friendly distributed training framework, federated learning allows each medical institution to use its private medical image data for model training, and only uploads model parameters or gradient information, thereby realizing a collaborative training mechanism without sharing original image data. This mechanism not only protects patient privacy, but also realizes model optimization across institutions, especially for large models and edge small models in medical scenarios.

[0004] However, in the training of medical image recognition models using federated learning, traditional federated learning frameworks usually assume that all participants use the same model structure to align and aggregate parameters. However, in the federated fine-tuning framework of large models (basic models), large models often have a large number of parameters, which cannot be deployed on many hospital edge devices. Under the condition that edge devices cannot deploy consistent model structures with cloud, how to achieve efficient and compatible collaborative training has become a core problem that needs to be solved in cloud-edge collaborative intelligence. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a heterogeneous federated model adjustment method based on importance sampling, which solves the deficiencies of the prior art.

[0006] The purpose of the present application is achieved by the following technical solution: a heterogeneous federated model adjustment method based on importance sampling, the method comprising:

[0007] S1, quantitatively analyze the importance of the layer by similar perception layer analysis, analyze each layer of the center server to obtain an importance score, thereby obtaining an importance distribution for the layer;

[0008] S2, based on the layer importance distribution obtained in S1, perform probability sampling to generate a sub-model extraction binary mask suitable for a heterogeneous client, so as to improve the matching degree of the model structure and the resource condition;

[0009] S3, each client extracts a sub-model through the binary mask and completes local deployment;

[0010] S4, introduce TSDL and SDL for local training, and upload and aggregate the lora parameters;

[0011] S5, repeat steps S2-S4 until convergence.

[0012] The step S1 specifically comprises:

[0013] S101, rearrange the of the global model, remove the first and last two layers, move the l layer to be analyzed in advance, obtain , compare the output difference with the original model, thereby obtaining the importance score of the l layer, indicates the original model parameter, indicates the rearranged model parameter;

[0014] S102, use CKA Z , Z l ) as the importance score of the l layer, record the importance score of all layers as , obtain the probability distribution sampled in sub-model extraction as , wherein indicates the temperature, L is the total number of layers.

[0015] The step of obtaining the importance score of the l layer comprises:

[0016] Given an input batch , the outputs are , , wherein indicates the input, indicates X the intermediate representation after the original model, indicates the intermediate representation before the last layer after the rearranged model;

[0017] The first layer is selected by the CKA based on the independence criterion HSIC l The importance score of the layer is , represents the independence between and , represents the independence between and , represents the independence between and .

[0018] The step S2 specifically comprises:

[0019] Importance sampling is performed according to the probability distribution obtained in S1, and a submodel of the m layers is extracted for deployment on the client, and the probability of extracting the submodel is After sampling, a binary mask for the i-th client is obtained , represents a set of selected layers.

[0020] The step S3 specifically comprises:

[0021] Before the start of each round, the selected client i obtains the binary mask After that, the submodel needs to be obtained and deployed in the i-th client, d represents the dimension, represents the product operation.

[0022] The step S4 specifically comprises:

[0023] After the submodel is deployed in each selected client, local training is started, and TSDL is introduced to alleviate the impact of local aggregation. The logtis information of the previous round on the client is retained, and the method of knowledge distillation is used to guide the training of the current round. TSDL is defined as , wherein is the parameter of the client i in the previous round, is the parameter of the client i in the current round, represents the average confidence of the previous round model, x represents the input image information, σ represents softmax, D KL represents divergence, represents the expected value of sampled from the distribution x ;

[0024] S402, introduce SDL to alleviate the influence of data heterogeneity, use the model distributed before each round of training as a reference model, take a snapshot of the training data before training starts to generate snapshot logits, use the snapshot logits to guide the current round of training through the method of knowledge distillation, and the SDL is defined as wherein, indicates the average confidence of the logits generated by the reference model, indicates the model parameters distributed before each round of training;

[0025] S403, when performing local training, the model of the training client i is i The cross-entropy loss needs to be calculated first, and is defined as , and the final total loss is , y indicates the label information, α indicates the hyperparameter, β is a constant value;

[0026] S404, after the client completes the local training, the lora parameters are uploaded and the corresponding parameters are aggregated to obtain wherein, indicates the lora parameters of the jth layer in the ith client, indicates the jth binary mask in the ith client.

[0027] The application has the following advantages: a heterogeneous federated model adjustment method based on importance sampling, importance sampling is introduced, sub-models can be selected according to the contribution of each layer, that is, layers with higher contribution have a higher probability of being selected, and layers with low contribution are not completely discarded and also have a probability of being selected, solving the problem that only the first k layers and random layers are selected in the existing traditional technology without focusing on the problem; the introduction of TSDL can alleviate the problem of uneven aggregation caused by local aggregation; the introduction of SDL can alleviate the problem of data heterogeneity. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a structural schematic diagram of the application. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the detailed description of the embodiments of this application provided below with reference to the accompanying drawings is not intended to limit the scope of protection of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The present invention will be further described below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the present invention specifically relates to an adjustment method for a heterogeneous federated fine-tuning base model based on importance sampling. By using medical data from heterogeneous nodes, a medical image recognition model that can assist in diagnosis can be collaboratively trained while protecting privacy.

[0031] Specifically, it includes the following:

[0032] S1. The importance of the layers is quantitatively analyzed by similarity perception layer analysis (SALA). Each layer of the central server is analyzed to obtain an importance score, thereby obtaining the importance distribution of the layers.

[0033] S2. Based on the hierarchical importance distribution obtained in S1, perform probability sampling to generate a sub-model extraction binary mask suitable for heterogeneous clients, so as to improve the matching degree between model structure and resource conditions.

[0034] S3. Each client extracts the sub-model using a binary mask and completes local deployment.

[0035] S4. Introduce TSDL (Temporal Self-Distillation Loss) and SDL (Snapshot Distillation Loss) for local training, and upload and aggregate the lora (low-rank adaptive fine-tuning) parameters;

[0036] S5. Repeat steps S2-S4 until convergence.

[0037] Furthermore, the steps in S1 specifically include:

[0038] The S101 and SALA modules evaluate the contribution of each layer to the data based on Centered Kernel Alignment (CKA, used to measure the correlation between two vectors). To analyze the importance of layers, the global model is first... Rearrange the layers, removing the first and last layers, to obtain the layer that needs to be analyzed. l Layer advance, get By comparing the output with that of the original model, the first result is obtained. l Layer importance score, Represents the original model parameters. This represents the rearranged model parameters. L Indicates the number of layers in the model;

[0039] S102, using CKA ( Z , Z l As the first l The importance score of each layer is recorded as follows: The probability distribution of sampling during sub-model extraction is obtained as follows: ,in, Indicates temperature.

[0040] Furthermore, we obtain the first l The importance score of a layer includes:

[0041] Given an input batch The outputs (referring to the intermediate morphological information of the model) are as follows: , ,in, and Representing model parameters respectively and The determined model, in which, Indicates input, This represents X after passing through the original model, in the middle before the last layer. This indicates that X, after being rearranged in the model, is represented in the middle before the last layer;

[0042] The calculation of the first step using CKA based on the independence criterion (HSIC, a statistic used to measure whether two random variables are independent) is as follows: l The importance score of the layer is , express and Independence between them express and Independence between them express and The independence between them.

[0043] Furthermore, the steps in S2 specifically include:

[0044] Importance sampling is performed based on the probability distribution obtained from S1, and a sub-model of layer m is extracted for deployment on the client. The probability of extracting the sub-model is... After sampling, the binary mask for the i-th client is obtained. , This represents a set of selected layers. It is a probability calculation without repetition sampling, meaning based on probability. Calculate the non-repeating sampling and select a subset with a total of m layers. This forms a binary mask.

[0045] Furthermore, the steps in S3 specifically include:

[0046] Before the start of each round, the selected client i receives a binary mask. After that, a sub-model needs to be obtained. And deploy it in the i-th client. d Representing dimension, This indicates a product operation.

[0047] Furthermore, the steps in S4 specifically include:

[0048] S401. After the sub-model is deployed on each selected client, local training begins. TSDL is introduced to mitigate the impact of local aggregation. The logits (the raw information of the model before softmax; logits are the unnormalized scores output by the classification model, representing the relative confidence of each category) from the previous round of training on the client are retained. Knowledge distillation is used to guide the current round of training. TSDL is defined as follows: ,in, For the parameters of client i in the previous round, For the parameters of client i in this round, This represents the average confidence level of the previous model. x The input image information is represented by σ, which represents the softmax function (the activation function of the output layer of a multi-class classification model), and D is the input image information. KL Denotes divergence, Indicates the distribution from Mid-sampling x Expected value;

[0049] S402, introduce SDL to alleviate the influence of data heterogeneity, use the model distributed before each round of training as a reference model, take a snapshot of the training data before training starts to generate snapshot logits, use snapshot logits to guide the current round of training through the method of knowledge distillation, SDL is defined as wherein, represents the average confidence of the logits generated by the reference model, represents the model parameters distributed before each round of training starts;

[0050] S403, when performing local training, the model of the training client i is i The cross-entropy loss needs to be calculated first, defined as , the final total loss is , y represents the label information, α represents the hyperparameter, β is a constant value;

[0051] S404, after the client completes the local training, upload the lora parameters and aggregate the corresponding parameters to obtain wherein, represents the lora parameters of the jth layer in the ith client, represents the jth binary mask in the ith client.

[0052] The above only describes the preferred embodiments of the present application, it should be understood that the present application is not limited to the form disclosed herein, should not be considered as excluding other embodiments, and can be used in various other combinations, modifications and improvements, and can be changed within the scope of the concept described herein, through the above teaching or related art or knowledge. The changes and variations made by those skilled in the art without departing from the spirit and scope of the present application shall be within the scope of protection of the appended claims of the present application.

Claims

1. A method for regulating heterogeneous federated models based on importance sampling, characterized in that: The method includes: S1. The importance of each layer is quantitatively analyzed through similarity perception layer analysis. Each layer of the central server is analyzed to obtain an importance score, thereby obtaining the importance distribution of each layer. S2. Based on the hierarchical importance distribution obtained in S1, perform probability sampling to generate a sub-model extraction binary mask suitable for heterogeneous clients, so as to improve the matching degree between model structure and resource conditions. S3. Each client extracts the sub-model using a binary mask and completes local deployment. S4. Introduce TSDL and SDL for local training, and upload and aggregate lora parameters; S5. Repeat steps S2-S4 until convergence; The steps in S1 specifically include: S101, Regarding the global model Rearrange the layers, remove the first and last layers, and move the l-th layer that needs to be analyzed to the front, to obtain... By comparing the output with that of the original model, the importance score of the l-th layer is obtained. Represents the original model parameters. Indicates the rearranged model parameters; S102, using CKA(Z, Z) l As the importance score for layer l, the importance scores for all layers are recorded as follows: The probability distribution of sampling during sub-model extraction is obtained as follows: ,in, Indicates temperature, L is the total number of layers; The importance score obtained for the l-th layer includes: Given an input batch The outputs are respectively , ,in, Indicates input, This represents X after passing through the original model, in the middle before the last layer. This indicates that X, after being rearranged in the model, is represented in the middle before the last layer; The importance score of layer l was calculated using CKA based on the independence criterion and HSIC. , express and Independence between them express and Independence between them express and Independence between them; The federated model processes image data.

2. The heterogeneous federated model adjustment method based on importance sampling according to claim 1, characterized in that: The steps in S2 specifically include: Importance sampling is performed based on the probability distribution obtained from S1, and a sub-model of layer m is extracted for deployment on the client. The probability of extracting the sub-model is... After sampling, the binary mask for the i-th client is obtained. , This represents a set of selected layers.

3. The heterogeneous federated model adjustment method based on importance sampling according to claim 1, characterized in that: The steps in S3 specifically include: Before the start of each round, the selected client i receives a binary mask. After that, a sub-model needs to be obtained. And deploy it in the i-th client, where d represents the dimension. This indicates a product operation.

4. The heterogeneous federated model adjustment method based on importance sampling according to claim 1, characterized in that: The steps in S4 specifically include: S401. After the sub-model is deployed on each selected client, local training begins. TSDL is introduced to mitigate the impact of local aggregation, retaining the previous round's logits information on the client and using knowledge distillation to guide the current round of training. TSDL is defined as follows: ,in, For the parameters of client i in the previous round, For the parameters of client i in this round, Let σ represent the average confidence score of the previous model, x represent the input image information, σ represent the softmax function, and D represent the average confidence score of the previous model. KL Denotes divergence, Indicates the distribution from The expected value of x sampled in the middle; S402. Introducing SDL to mitigate the impact of data heterogeneity involves using the model distributed before each training round as a reference model. A snapshot of the training data is taken before training begins, generating snapshot logits. Knowledge distillation is then used to guide the current training round with these snapshot logits. SDL is defined as follows: ,in, This represents the average confidence level of the logits generated by the reference model. This represents the model parameters distributed before the start of each round of training; S403. During local training, train model θ for client i. i We need to calculate the cross-entropy loss first, defined as follows: The final total loss is y represents the label information, α represents the hyperparameter, and β is a constant. S404. After the client completes local training, it uploads the LoRa parameters and aggregates the corresponding parameters to obtain... ,in, This represents the lora parameter in the j-th layer of the i-th client. This represents the j-th binary mask in the i-th client.

Images