A method for text classification based on a federated learning large model pre-trained with shallow features.

Abstract

This invention discloses a method for text classification based on a federated learning large model pre-trained with shallow features, comprising: the server obtaining a global model; the client obtaining the initial parameters of the global model and constructing a local model based on the initial parameters of the global model and the processing layer number l specified by the server for training, and training the local model; the client uploading the updated parameters of the l-th processing layer and the output layer parameters to the server for aggregation to obtain updated parameters, updating the global model on the server, and distributing the updated parameters to each client for a new round of federated learning training; the client obtaining the updated parameters and the newly specified processing layer number l' from the server and repeating the federated learning; after completing the federated learning, the server replacing the corresponding parameters of the global model with the parameters aggregated by each client to obtain a further pre-trained global model, which is then fine-tuned for text classification in a specific domain.

Description

Technical Field

[0001] This invention belongs to the field of federated learning, and in particular relates to a method for text classification based on a large federated learning model pre-trained with shallow features. Background Technology

[0002] Federated learning is a technology that allows multiple clients to collaboratively train a model without exposing their own data. Currently, with the development of artificial intelligence, neural network models are becoming increasingly large, and training these models requires ever-growing amounts of data. Traditional neural network training processes require centralized collection and processing of the necessary data. However, this centralized data collection and processing often raises privacy concerns, especially for data from specific professional fields. Federated learning technology can effectively protect user data privacy while simultaneously training the neural network model.

[0003] In federated learning, the central server that receives the training model is called the server. It possesses the complete neural network model and a small amount of task data for model performance testing. The device that completes the model training task using local data is called the client. This can consist of a mobile phone, tablet, personal computer, or a device from a small company (organization). It holds the data needed for neural network model training locally, but its computing power and bandwidth are limited. In each round of federated learning, the server distributes the required model parameters to all clients. After the clients complete the model training on their local data, they upload the model parameters back to the server, thus preventing the leakage of local data. The server aggregates the parameters uploaded by each client to obtain a new model, and then distributes this model parameters back to the clients for a new round of federated learning until a fixed number of federated learning rounds are completed or the desired model is obtained.

[0004] In federated learning, the limited computing power and bandwidth of clients prevent them from training large models, especially large pre-trained language models used in Natural Language Processing (NLP). To address this issue, proposed techniques can be broadly categorized into three areas: information compression, model pruning, and knowledge distillation. Information compression uses fewer bits to represent the parameters that the client needs to transmit, saving communication overhead but not computational overhead. Model pruning and knowledge distillation can produce a smaller model than the original to accelerate inference, but they do not effectively reduce the computational cost of training large models. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a method for text classification based on a federated learning large model pre-trained with shallow features.

[0006] To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0007] According to a first aspect of the present invention, a method for text classification based on a federated learning large model pre-trained with shallow features is provided, the method comprising:

[0008] Step S1: The server obtains the global model and collects test data for text classification tasks in the professional field; the client collects unlabeled pre-trained data in the professional field.

[0009] Step S2: The client obtains the initial parameters of the global model and constructs a local model based on the initial parameters of the global model and the processing layer number l to be trained specified by the server. The local model is then trained using unlabeled pre-training data from the professional field.

[0010] Step S3: The client uploads the updated parameters of the l-th processing layer and the output layer to the server for aggregation, obtains the updated parameters, updates the global model on the server, and sends the updated parameters to each client for a new round of federated learning training.

[0011] Step S4: The client obtains the updated parameters from the server and the newly assigned processing layer number l' to be trained from the server, and repeats the federated learning process.

[0012] Step S5: After the federated learning training reaches the preset number of rounds or the global model on the server converges, the server uses the parameters obtained from the aggregation of various clients to replace the corresponding parameters of the global model, thereby obtaining the trained global model. The global model is then used to perform text classification on the test data of the professional domain text classification task.

[0013] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory being coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described method for text classification based on a federated learning large model pre-trained with shallow features.

[0014] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for text classification based on a federated learning large model pre-trained with shallow features.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for text classification based on a federated learning large model pre-trained with shallow features. In a federated learning scenario, this method can train the global model on a resource-constrained client by training only some layer parameters of the local model, thus achieving both user privacy protection and successful training of the global model in federated learning. Compared with traditional federated learning, this method achieves accuracy close to that of traditional federated learning while significantly reducing client training and communication overhead, thereby improving the accuracy and efficiency of text classification. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the method of the present invention;

[0018] Figure 2 This is an overall framework diagram of federated learning provided by the present invention;

[0019] Figure 3 This is a schematic diagram of the client building a local small model according to the present invention;

[0020] Figure 4 This is a schematic diagram of an electronic device provided by the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0023] like Figure 1 As shown, this embodiment of the invention provides a method for text classification based on a large federated learning model pre-trained with shallow features, specifically including the following steps:

[0024] Step S1: The server obtains the global model and collects test data for text classification tasks in the professional domain; the client collects unlabeled pre-trained data in the professional domain.

[0025] Further, in step S1, the global model is a pre-trained language model consisting of an embedding layer, L (L>1) processing layers (Transformer-Layer, T-Layer), and an output layer. The processing layers are selected from Transformer, CNN, LSTM, or GRU neural networks. In this example, the global model can be BERT, RoBERTa, etc.

[0026] It should be noted that the global model obtained by the server has been trained on a general corpus and has certain knowledge. The client has corpus knowledge in a professional field. In this example, the professional field corpus includes relevant corpus in professional fields such as biology, medicine, chemistry, and computer science. Due to privacy protection issues, these corpora cannot be collected centrally for training the global model. Therefore, federated learning method is required to complete the training of the global model.

[0027] In step S2, the client obtains the initial parameters of the global model and constructs a local model based on the initial parameters of the global model and the processing layer number l to be trained specified by the server. The local model is then trained using unlabeled pre-training data from the professional domain.

[0028] Specifically, step S2 includes the following sub-steps:

[0029] Step S201: The client obtains the initial parameters of the global model.

[0030] In step S202, the client constructs a local model based on the initial parameters of the global model and the processing layer number l to be trained specified by the server.

[0031] Specifically, in step S202, the client constructs a local model using the initial parameters of the global model and the T-Layer number l specified by the server that needs to be trained. Client C k The constructed local model consists of an Embedding layer, L k (1≤L k The local model consists of a processing layer (T-Layer) and an output layer. The local model has the same network structure as the global model, but it has fewer processing layers (T-Layer), so the local model is smaller in size.

[0032] The local model built by the client is smaller than the global model on the server and does not participate in the final downstream text classification task test. It is only used to assist in training some neural network layers of the global model.

[0033] The T-Layer number l specified by the server for the current training phase includes:

[0034] Step S20201: Set the current federated training iteration round to i and the total federated training iteration round to I;

[0035] Step S20202: When i = 0, set the T-Layer number to be trained to l = 0, that is, specify the client to train the 0th T-Layer. Shallow T-Layers play a more important role in the pre-training process of the global model.

[0036] Step S20203: The process is complete after half of the remaining federated learning iterations have been completed. After each federated learning iteration, update the T-Layer number to be trained, l = l + 1, and the remaining federated learning iterations are...

[0037] In step S20204, repeat step S20203. After completing half of the remaining federated learning iterations, set l = l + 1 so that the client prioritizes training the shallow T-Layer during the model training process.

[0038] The process of building a local model on the client side includes the following steps:

[0039] Step S20205: Assume the T-Layer number to be trained is l=2, the server's global model has L=12 T-Layer layers, and the client C... k The local model has L k =6-layer T-Layer;

[0040] Step S20206, for client C k The parameters of the local model's Embedding layer, output layer, and T-Layer layer (l=0,1,2) are consistent with those of the server's global model. The parameters of the local model are obtained using direct mapping.

[0041] Step S20207, for client C k The parameters of the T-Layer layers above layer l in the local model are randomly sampled from the remaining high-level T-Layer numbers in the global model. For example, three numbers are sampled from the numbers (3…11) in an ordered manner. For example, if index = (7,7,9), then the T-Layer layer parameters of the local model l = 3, 4, 5 are respectively taken from the T-Layer layer parameters corresponding to index in the global model.

[0042] Step S20208: After the local model is built on the client, the local model is trained on the available local data.

[0043] It should be noted that the client first obtains the parameters of the global model from step S1 in the initial stage of the overall federated learning process, constructs a local parameter pool for later use in building the local model, and then obtains the processing layer number to be trained from the server, and combines it with the parameters of the global model to construct the local model for training. The processing layers below the training layer in the local model maintain the same parameters as the corresponding global model, while the processing layers above the training layer in the local model are sampled from the corresponding upper layers of the global model to assist the local model in obtaining sufficient knowledge for single-layer training. During training, only the parameters of the training layer are trained, while the remaining parameters in the local model remain unchanged. By reducing the model size and the number of training parameters, the computational overhead during the local model training process is reduced. After training, the client only sends the parameters of the training layer to the server for aggregation, thereby reducing the communication overhead in federated learning and improving efficiency.

[0044] Step S203: Train the local model, update the parameters of the l-th T-Layer and the output layer, and send them to the server. This includes the following sub-steps:

[0045] In step S20301, the client first processes the unlabeled corpus of its local professional domain, processing all the corpus into individual sentences and segmenting each sentence to obtain fields; then, 15% of all fields in the sentences are selected for processing. Of these 15% of fields, 80% are replaced with [mask], 10% are replaced with a random field, and the remaining 10% are left unchanged. These fields are then used for training the local model to create a Mask Language Model (MLM).

[0046] In step S20302, the client fixes all the layer parameters of the local model except for the T-Layer of the l-th layer and the output layer. Only the T-Layer of the l-th layer and the output layer will participate in both forward propagation and backpropagation to update the parameters. The other parameters will only participate in forward propagation to assist training.

[0047] In step S20303, the client performs a fixed number of rounds of MLM training on the local model on the local corpus, and updates the parameters of the l-th T-Layer and the output layer parameters.

[0048] It should be noted that the parameters of the local model correspond to those of the embedding and output layers of the server-side global model. The parameters of the T-Layer layer and the T-Layer layers below it specified for training on the server side are consistent with the parameters of the server-side global model. The parameters above the specified T-Layer layer are obtained by sampling and mapping the parameters of the corresponding interval of the server-side global model. The client performs MLM training on the constructed local model on the domain-specific corpus stored locally, and only updates the l-th T-Layer processing layer specified by the server in the local model layer. The parameters of the other layers remain fixed. After the local training is completed, only the trained and updated l-th T-Layer processing layer and output layer parameters are sent to the server for aggregation.

[0049] In step S3, the client uploads the updated parameters of the l-th T-Layer processing layer and the output layer parameters to the server for aggregation, obtains the updated parameters, updates the global model on the server, and distributes the aggregated parameters to each client for a new round of federated learning training.

[0050] Specifically, the client uploads the updated parameters of the l-th T-Layer processing layer and output layer to the server for aggregation. The server then aggregates these parameters, denoted as W. k For the parameters uploaded by client k, if the server uses the FedAvg() algorithm to aggregate the parameters, and if there are a total of n clients participating in federated learning, then the parameters aggregated by the server are:

[0051]

[0052] Step S4: The client obtains the updated parameters from the server and the newly assigned processing layer number l' to be trained from the server, and repeats the federated learning process.

[0053] Specifically, in step S4, the client updates the parameters in the local parameter pool each time it receives new parameters, ensuring that the latest parameters are used when building the local model based on the parameter pool. The training layer number l' obtained by the client is obtained by the server in a progressive manner. Training starts from the lowest layer and then increments by one for each half of the remaining federated learning rounds, so that the client's local model focuses on training more important parameters.

[0054] The number of the processing layer trained by the client is progressively increased by the server. Resources are prioritized to train the shallower T-layers that are more important in the local model. After completing half of the remaining federated learning rounds, the number of the trained T-layer is incremented so that the higher layers of the local model can also receive some training. In addition to the T-layers specified by the server, the client also trains the output layer of the local model. The output layer and the specified T-layer together constitute the training and update layer of the local model. Other parameters are not updated. After training is completed, the updated parameters are uploaded to the server.

[0055] Step S5: After the federated learning training reaches the preset number of rounds or the global model on the server converges, the server uses the parameters obtained from the aggregation of various clients to replace the corresponding parameters of the global model, thereby obtaining a further pre-trained global model, which is then used for text classification.

[0056] It should be noted that the server uses some parameters obtained from the client's training to update its global model. At this point, the global model has acquired the knowledge contained in the client's data without viewing the client's data, thus protecting the client's privacy. Afterward, the server uses the updated global model to fine-tune and test it on the downstream text classification task, thereby completing the overall federated learning task.

[0057] Example 1:

[0058] In this example, taking text classification in the biological field as an example, the method of text classification based on shallow feature pre-training federated learning large model proposed in this invention is further clarified.

[0059] Step S1: The server obtains the global model and collects test data in the biological field; the client collects training data in the biological field.

[0060] Furthermore, this example selects the bert-base-uncased model, which contains an embedding layer, 12 transformer (T-layer) layers, and a single output layer. The number of parameters in the output layer is negligible compared to the number of parameters in the rest of the model. In this example, the server holds downstream tasks in the biological domain that require text classification testing, while each client holds a varying amount of unlabeled corpus data from the biological domain.

[0061] In step S2, the client first downloads the parameters of the global model during the federated learning initialization phase and stores them locally as a parameter pool for subsequent local model construction. In this example, the local model built by the local client contains an embedding layer, six transformer (T-Layer) layers, and one output layer. During the initial stage of federated learning, the T-Layer number to be trained and updated is l=0. Therefore, during the local model construction process, the parameters of the embedding layer, the 0th T-Layer, and the output layer of the local model are consistent with the parameters of the global model. Then, five ordered random numbers are selected from (1...11), such as (2,5,5,9,10). The parameters of the 1st to 5th T-Layer layers of the local model are then the corresponding T-Layer parameters specified by the preceding random number sequence in the global model. Then, the local model fixes all parameters except for the T-Layer 0 and the output layer, and performs a fixed number of MLM training rounds using the local unlabeled biological domain corpus. After training, the parameters of the T-Layer 0 and the output layer are sent to the server for aggregation.

[0062] Step S3: In this example, the server performs FedAVG aggregation on the parameters uploaded by each client, that is, it calculates the average value of the parameters corresponding to all clients based on the number of clients, and then uses the aggregated parameters to update the global model of the server, and distributes the aggregated parameters to each client for a new round of federated learning.

[0063] In step S4, the client receives new parameters from the server. First, it updates its local parameter pool for building the local model in the new round. Then, it builds a new local model based on the T-Layer number l to be trained. The number l is progressively built by the server. Every time half of the remaining federated learning training rounds are completed, l is incremented. In this example, if a total of ten rounds of federated learning training are required, then l = 0 in rounds 1-5, l = 1 in rounds 6-8, l = 2 in round 9, and l = 3 in round 10.

[0064] In step S5, after federated learning training is complete, the server uses the aggregated parameters to replace the corresponding parameters of the global model, enabling the global model to acquire the domain-specific knowledge contained in the client without infringing on the client's privacy. The server will then use this large model for fine-tuning on a biological domain text classification task (Downstream Task), utilizing the global model to perform the biological domain text classification task.

[0065] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described above for text classification using a federated learning large model based on shallow feature pre-training. Figure 4 The diagram shown illustrates a hardware structure of any device with data processing capabilities, where the text classification method based on a shallow feature pre-trained federated learning model provided in this embodiment of the invention is located. (Except for...) Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0066] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the method for text classification based on a federated learning large model pre-trained with shallow features as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the principles and scope of the present invention. The protection scope of the present invention should be determined by the claims.

Claims

1. A method for text classification based on a federated learning large model pre-trained with shallow features, characterized in that, The method includes: Step S1: The server obtains the global model and collects test data for text classification tasks in the professional domain; the client collects unlabeled pre-trained data in the professional domain. Step S2: The client obtains the initial parameters of the global model and constructs a local model based on the initial parameters of the global model and the processing layer number l to be trained specified by the server. The local model is then trained using unlabeled pre-training data from the professional field. Step S3: The client uploads the updated parameters of the l-th processing layer and the output layer to the server for aggregation, obtains the updated parameters, updates the global model on the server, and sends the updated parameters to each client for a new round of federated learning training. Step S4: The client obtains the updated parameters from the server and the newly assigned processing layer number l' to be trained from the server, and repeats the federated learning process. Step S5: After the federated learning training reaches the preset number of rounds or the global model on the server converges, the server uses the parameters obtained from the aggregation of each client to replace the corresponding parameters of the global model, and then uses the global model to perform text classification on the test data of the professional domain text classification task. Building a local model includes: Based on the processing layer number l specified by the server, the parameters of processing layers 0 to l of the global model are directly mapped to obtain the parameters of processing layers 0 to l of the local model. The parameters of processing layers l+1 to L of the global model are randomly sampled to obtain the parameters of processing layers l+1 to L of the local model. k Layer processing layer parameters; The server-specified layer number l to be trained includes: Let i be the current iteration round of federated training and I be the total number of federated training iteration rounds; When i = 0, the T-Layer number to be trained is set to l = 0; Once half of the remaining federated learning iterations have been completed, the process is complete. After each federated learning iteration, update the T-Layer number to be trained, l = l + 1, and the remaining federated learning iterations are... After completing half of the remaining federated learning iterations, update the T-Layer number l = l + 1 that needs to be trained.

2. The method for text classification based on a federated learning large model pre-trained with shallow features according to claim 1, characterized in that, The global model is a pre-trained language model consisting of an embedding layer, an L-layer processing layer, and an output layer.

3. The method for text classification based on a federated learning large model pre-trained with shallow features according to claim 2, characterized in that, The local model consists of an embedding layer, L k It consists of a layer processing layer and an output layer, where 1 ≤ L k <L。 4. The method for text classification based on a federated learning large model pre-trained with shallow features according to claim 1, characterized in that, Training a local model using unlabeled pre-trained data from a specialized domain includes: Using unlabeled pre-trained data from a specialized field, a masked language model is trained on the local model. The client fixes all the parameters of the local model except for the l-th processing layer and the output layer, and only participates in forward propagation to assist training; the l-th processing layer and the output layer participate in forward propagation and back propagation to update parameters.

5. The method for text classification based on a federated learning large model pre-trained with shallow features according to claim 1, characterized in that, Step S3 includes: The client uploads the updated parameters of the l-th processing layer and the output layer to the server, and the server uses the FedAvg() algorithm to aggregate the parameters.

6. The method for text classification based on a federated learning large model pre-trained with shallow features according to claim 1, characterized in that, Step S4 includes: Each time the client receives updated parameters, it first updates the parameters in its local parameter pool. The number l' of the processing layer to be trained obtained by the client is incrementally increased by the server; training starts from the lowest processing layer, and then the number of the processing layer to be trained is incremented by one every time half of the remaining federated learning rounds are completed.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method for text classification based on a federated learning large model pre-trained with shallow features as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method for text classification based on a federated learning large model pre-trained with shallow features as described in any of claims 1-6.

Images