A text prediction method and device based on horizontal federated learning

Abstract

The present disclosure provides a text prediction method and apparatus based on horizontal federated learning, which are applied to participants and include: obtaining a text to be predicted; inputting the text to be predicted into a local federated model to obtain federated prediction data output by the local federated model, and inputting the federated prediction data into a private model of a downstream task to obtain a prediction result output by the private model; wherein the local federated model forms a horizontal federated learning network with a public-private key server and a summarization server; the local federated model is trained based on shared parameter data of the horizontal federated learning network and training labels of the shared parameter data; the private model is trained based on local privatized text training data and text training data labels. Each participant forms a horizontal federated learning network with the public-private key server and the summarization server, and uses the local federated learning model of federated learning for the downstream task of text prediction, which improves the security of data network transmission while effectively protecting local private data.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing technology, and in particular, to a text prediction method and device based on horizontal federated learning. Background Art

[0002] Federated learning originated from the distributed training scheme of the language model of virtual keyboards, using logistic regression for the ranking of the next word and the language model for the prediction of the next word. In some scenarios, due to the long text, to achieve better results, a larger model and more samples are required. However, each participant does not always have the willingness to share their data with each other. Federated learning allows each participant to jointly model without sharing their data locally and only transmitting parameters.

[0003] In the prior art, methods based on federated distillation, such as FedED, or methods based on federated transfer learning, such as FedNER, show that joint modeling by all parties has significant performance advantages compared to single-party modeling. However, there is only one central server in the existing horizontal federated learning network, and data loss will occur during the process of the central server decrypting and then aggregating the gradients, which also affects the security of data transmission. Summary of the Invention

[0004] The present disclosure provides a text prediction method and device based on horizontal federated learning to solve the security problem of federated learning in the prior art.

[0005] The present disclosure provides a text prediction method based on horizontal federated learning, which is applied to a participant. The prediction method includes:

[0006] Obtain the text to be predicted;

[0007] Input the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model, and input the federated prediction data into the private model of the downstream task to obtain the prediction result output by the private model;

[0008] Wherein, the local federated model, the public-private key server, and the aggregation server form a horizontal federated learning network;

[0009] The local federated model is trained based on the shared parameter data and the training labels of the shared parameter data of the horizontal federated learning network;

[0010] The private model is trained based on the local privatized text training data and the text training data labels.

[0011] According to a text prediction method based on horizontal federated learning provided by the present disclosure, the horizontal federated learning network includes multiple participants;

[0012] The training process of the local federal model for each participant specifically includes:

[0013] Based on the local privatized text training data and text training data labels, train the local federal model to obtain shared parameter gradients;

[0014] Receive the public key generated by the public-private key server;

[0015] Homomorphically encrypt the shared parameter gradients according to the public key and send them to the aggregation server;

[0016] Receive the updated shared parameters sent by the public-private key server; wherein, the updated shared parameters are obtained by the aggregation server receiving the shared parameter gradients sent by each participant, performing gradient mean processing, and sending the processed mean data to the public-private key server; the public-private key server decrypts the received mean data with the private key;

[0017] Update the local federal model with the updated shared parameters to complete one round of iteration;

[0018] Loop through at least one round of iteration until the loss function of the local federal model converges.

[0019] According to a text prediction method based on horizontal federated learning provided by the present disclosure, the local federal model includes an embedding layer and at least one encoding layer; the process of inputting the text to be predicted into the local federal model to obtain the federated prediction data output by the local federal model specifically includes:

[0020] Input the text to be predicted into the embedding layer, and obtain the dimensionality-reduced feature vector output by the embedding layer through initial mapping of the text;

[0021] Input the dimensionality-reduced feature vector into at least one of the encoding layers in sequence, and obtain the federated prediction data output by the encoding layer through high-order feature extraction.

[0022] According to a text prediction method based on horizontal federated learning provided by the present disclosure, the task types of the downstream tasks include: text classification, sequence labeling, text generation, and sentence pair classification, and the private model processes to obtain corresponding prediction results according to the task types of the downstream tasks.

[0023] The present disclosure also provides a text prediction device based on horizontal federated learning, including:

[0024] An acquisition unit, configured to acquire the text to be predicted;

[0025] A text prediction unit, configured to input the text to be predicted into a local federated model, obtain federated prediction data output by the local federated model, and input the federated prediction data into a private model of a downstream task to obtain a prediction result output by the private model;

[0026] Wherein, the local federated model, a public-private key server, and a summary server form a horizontal federated learning network;

[0027] The local federated model is trained based on shared parameter data of the horizontal federated learning network and training labels of the shared parameter data; the private model is trained based on local privatized text training data and text training data labels.

[0028] According to a text prediction device based on horizontal federated learning provided by the present disclosure, the text prediction unit includes a model training unit, and the model training unit is configured to perform model training on the local federated model.

[0029] According to a text prediction device based on horizontal federated learning provided by the present disclosure, the horizontal federated learning network includes multiple participants; for each participant, the model training unit includes:

[0030] A shared parameter gradient acquisition sub-unit, configured to perform local federated model training based on local privatized text training data and text training data labels to obtain a shared parameter gradient;

[0031] A public key receiving sub-unit, configured to receive a public key generated by the public-private key server;

[0032] A gradient encryption sending sub-unit, configured to homomorphically encrypt the shared parameter gradient according to the public key and send it to the summary server;

[0033] An updated shared parameter receiving sub-unit, configured to receive updated shared parameters sent by the public-private key server; wherein, the updated shared parameters are obtained by the summary server receiving the shared parameter gradients sent by each participant, performing gradient mean processing, and sending the processed mean data to the public-private key server; the public-private key server decrypts the received mean data with a private key;

[0034] A model update sub-unit, configured to update the updated shared parameters to the local federated model to complete one round of iteration;

[0035] A loop iteration sub-unit, configured to loop through at least one round of iteration until the loss function of the local federated model converges.

[0036] A text prediction device based on horizontal federated learning provided by the present disclosure, wherein the local federated model includes an embedding layer and at least one encoding layer; the text prediction unit includes a mapping subunit and a feature extraction subunit;

[0037] The mapping subunit is configured to input the text to be predicted into the embedding layer, and obtain a dimensionality-reduced feature vector output by the embedding layer through initial mapping of the text;

[0038] The feature extraction subunit is configured to sequentially input the dimensionality-reduced feature vector into at least one of the encoding layers, and obtain federated prediction data output by the encoding layer through high-order feature extraction.

[0039] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the text prediction method based on horizontal federated learning as described in any one of the above are implemented.

[0040] The present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the text prediction method based on horizontal federated learning as described in any one of the above are implemented.

[0041] The text prediction method and device based on horizontal federated learning provided by the present disclosure enable each participating party to form a horizontal federated learning network with a public-private key server and an aggregation server, and use the local federated learning model of federated learning for downstream tasks of text prediction, improving the security of data network transmission while effectively protecting local private data. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 is a schematic flowchart of the text prediction method based on horizontal federated learning provided by the present disclosure;

[0044] Figure 2 is a schematic flowchart of the training process of the local federated model of each participating party in the text prediction method based on horizontal federated learning provided by the present disclosure;

[0045] Figure 3 is a schematic structural diagram of the training of the local federated model of each participating party in the text prediction method based on horizontal federated learning provided by the present disclosure;

[0046] Figure 4 It is a schematic structural diagram of the connection between the local federated model and the private model of the text prediction method based on horizontal federated learning provided by the present disclosure;

[0047] Figure 5 It is a schematic structural diagram of the text prediction device based on horizontal federated learning provided by the present disclosure;

[0048] Figure 6 It is a schematic structural diagram of the electronic device provided by the present disclosure; Detailed implementation manners

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some, but not all, of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts shall fall within the scope of protection of the embodiments of the present disclosure.

[0050] Before describing the embodiments of the present disclosure, a brief description of the federated learning technology in the prior art will be given first.

[0051] Federated learning is essentially a distributed machine learning technology or machine learning framework. The goal of federated learning is to achieve joint modeling and improve the performance of the AI model on the basis of ensuring data privacy, security, and compliance with laws and regulations. Horizontal Federated Learning, also known as Homogenous Federated Learning, refers to a type of federated learning in which each participating party has the same feature space but different sample spaces. In the prior art, the local federated model of the FedED method for federated distillation is a bert model. Due to the huge number of parameters, based on the method of federated distillation, the central server needs to have private data, and each participating party needs to align the annotation standards.

[0052] The FedNER method for federated transfer is limited to the NER (Named Entity Recognition) task in downstream tasks, and the local federated model is a language model based on word2vec and elmo. It cannot make good use of the context and requires each party to align the vocabulary, so there is a risk of gradient privacy leakage. Once the gradient information is leaked, it is easy to expose the original text information.

[0053] In the existing centralized solution, both the operation of aggregating encrypted gradients and decryption are performed on the same server, which will pose a security risk, and adding noise to the gradients will affect the modeling effect.

[0054] The following combinationFigure 1 Describe the text prediction method based on federated learning according to the embodiments of the present disclosure, including:

[0055] Step 110: Obtain the text to be predicted;

[0056] The text to be predicted in this step is determined according to the downstream task.

[0057] Step 120: Input the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model, and input the federated prediction data into the private model of the downstream task to obtain the prediction result output by the private model;

[0058] Among them, the local federated model forms a horizontal federated learning network with the public-private key server and the aggregation server;

[0059] The local federated model is trained based on the shared parameter data of the horizontal federated learning network and the training labels of the shared parameter data;

[0060] The private model is trained based on the local privatized text training data and the text training data labels.

[0061] In the embodiments of the present disclosure, the local federated model includes an Embedding layer and at least one Transformer layer.

[0062] In the embodiments of the present disclosure, the horizontal federated learning network includes multiple participants.

[0063] The embodiments of the present disclosure provide a method for decentralized horizontal federated modeling, which can not only prevent the leakage of gradients during network transmission, but also prevent the central server from decrypting and then aggregating the gradients, achieving theoretically lossless data.

[0064] Step 120 specifically includes:

[0065] Input the text to be predicted into the Embedding layer, and obtain the dimensionality-reduced feature vector output by the Embedding layer after performing an initial mapping on the text;

[0066] Input the dimensionality-reduced feature vector into at least one of the Transformer layers in sequence, and obtain the federated prediction data output by the Transformer layer after performing high-order feature extraction.

[0067] Since the text data of each participating party is not shared, the labeled tags are not exactly the same, and the data needs to be protected for privacy, so joint modeling is required. However, in the existing technologies, the pre-trained models such as bert commonly used in the federated distillation method have too large parameter scales and too large network communication overheads. And the shared models based on word2vec and elmo commonly used in federated transfer have limited ability to capture contexts. Therefore, in the embodiments of the present disclosure, preferably, the albert model is applied as the local federated model, and transfer learning tasks are performed thereon.

[0068] In order to achieve a better modeling effect, in other preferred embodiments, other pre-trained models such as the distilbert model can also be used as the local federated model. Although the number of parameters may be larger, it can reduce the parameter transfer process.

[0069] In the embodiments of the present disclosure, the downstream tasks are mainly for (Natural Language Processing) nlp tasks. The types of downstream tasks mainly include: text classification, sequence labeling, text generation, and sentence pair classification. The private model processes to obtain corresponding prediction results according to the task type of the downstream task.

[0070] In the embodiments of the present disclosure, various networks can be used for the downstream tasks. For example, when the downstream task is text classification, only the [CLS] vector or the [CLS] vector connected to another linear layer and / or an external multinomial logistic regression (softmax) layer can be used for the downstream task. In the embodiments of the present disclosure, in order to ensure the prediction effect, the downstream tasks of each participating party should be the same and the goals should be as close as possible, but it is not required that the training data labels are exactly the same.

[0071] In the embodiments of the present disclosure, the underlying local federated model is a shared model, and each participating party shares the parameters, while the parameters of the upper-layer private model are private.

[0072] In the embodiments of the present disclosure, in step 120, the training process of the local federated model for each participating party is as Figure 2 and Figure 3 shown, and specifically includes:

[0073] Step 210: Train the local federated model based on the local privatized text training data and the text training data labels to obtain the shared parameter gradients;

[0074] In this step, the data types of the local privatized text training data of each participating party are the same.

[0075] Each participating party trains and applies the same model as the local federated model. The parameters of the local federated model in the first round of local federated model training are the model parameters downloaded from the public-private key server, and iterative training starts.

[0076] Step 220: Receive the public key generated by the public-private key server;

[0077] In the embodiments of the present disclosure, in a horizontal federated learning network, two servers are randomly selected from the servers of each participating party as the public-private key server and the aggregation server respectively. As Figure 3 shown, platform k is the public-private key server and platform j is the aggregation server. In the embodiments of the present disclosure, the purpose of random selection is that before the task is carried out, it is not known which participating party the public-private key server and the aggregation server are respectively, and the participating parties to which the two selected servers belong themselves do not know that they are selected, so as to avoid collusion between the two selected participating parties.

[0078] In this step, the public-private key server generates a public key and a private key, and sends the public key to each of the participating parties;

[0079] Step 230: Homomorphically encrypt the shared parameter gradients according to the public key and send them to the aggregation server;

[0080] Existing NLP federated learning methods have the risk of gradient leakage. In particular, the leakage of gradients in the embedding layer will lead to the leakage of tokens of the original text. Therefore, in the embodiments of the present disclosure, homomorphic encryption technology is used to protect the gradients, so as to prevent gradient leakage and ensure the security of network transmission. The gradients of the local federated model are encrypted by the Paillier algorithm and sent to the aggregation server.

[0081] In this step, preferably, the albert model is used as the local federated model. This method based on the pre-trained model can make more full use of the context information of the large-scale pre-trained corpus compared with the shared models of word2vec and elmo in the fedNER federated migration method. At the same time, the albert model adopts the methods of parameter low-rank decomposition and cross-layer parameter sharing, and the number of parameters is only 1 / 12 of the original bert parameters. Therefore, it preferably solves the problems in the prior art that the number of parameters in the federated learning joint modeling is huge and the network transmission consumption is large. As Figure 4 shown, the albert model contains 1 Embedding layer and 12 Transformer encoder layers.

[0082] Using the albert model as the local federated model for horizontal federated transfer learning can greatly reduce the number of parameters, thereby reducing the communication overhead, and has certain advantages in effect compared with non-pre-trained models.

[0083] Step 240: Receive the updated shared parameters sent by the public-private key server; wherein, the updated shared parameters are obtained by the aggregation server receiving the shared parameter gradients sent by each participating party, performing gradient mean processing, and sending the processed mean data to the public-private key server; the public-private key server decrypts the received mean data using the private key;

[0084] In this step, the aggregation server performs ciphertext operations, avoiding obtaining the specific gradient values of each participating party, and then transmitting the calculation result to the public-private key server.

[0085] In this step, the gradient data is first calculated on the aggregation server and then decrypted on the public-private key server, thus preventing gradient leakage and achieving lossless processing of gradient data.

[0086] In this step, since the updated shared parameters are the calculation results and there is no problem of gradient leakage, the updated shared parameters can be transmitted in plaintext.

[0087] In this step, after decrypting the gradient mean calculated in the aggregation server, the mean gradient is updated to the model parameters of the local federated model, that is, the updated shared parameters are obtained. In the embodiments of the present disclosure, since it is a horizontal federated learning network, the model parameters of the local federated models of each participating party are the same. Therefore, the processed mean gradient can be directly updated to the model parameters, so that the public-private key server finally sends the same updated shared parameters to each participating party.

[0088] Step 250: Update the local federated model with the updated shared parameters to complete one round of iteration;

[0089] In the embodiments of the present disclosure, the underlying local federated models of each participating party are shared models, and each participating party shares parameters, while the parameters of the upper-layer private models are private. During the backpropagation process, the private parameters are directly updated based on the gradients.

[0090] Step 260: Loop through at least one round of iteration until the loss function of the local federated model converges.

[0091] In the embodiments of the present disclosure, steps 210-250 are repeated until the loss functions of the local federated models of each participating party converge. Preferably, during each round of iteration in steps 210-250, the content transmitted, sent, and received each time is uploaded to the chain, thereby enhancing data security and privacy protection.

[0092] After step 260, the local federated models of each participating party are enhanced and can be directly used for the prediction of their respective downstream tasks. After training, each participating party makes full use of the data of other participating parties when training its own downstream tasks.

[0093] Most of the existing horizontal federated learning homomorphic encryption technologies perform decryption and gradient aggregation on the same central server, or rely on a third party to perform decryption. The former cannot ensure that the central server aggregates gradients first and then decrypts. It can only decrypt first and then aggregate, which cannot guarantee data security. The latter is difficult to find a reliable third party. Therefore, the embodiments of the present disclosure propose a decentralized method of separating decryption and gradient aggregation on two servers to ensure calculation first and then decryption, thereby realizing the protection of gradient data.

[0094] The following uses specific application scenario examples to illustrate the text prediction method based on horizontal federated learning in the embodiments of the present disclosure:

[0095] Application scenario: Multiple hospitals in the same or similar departments conduct federated learning based on case texts for downstream tasks of text generation: achieving parallel expectations for medical conversations, automatic consultations, and chat conversations based on multiple hospitals, and training a chatbot for automatic assisted consultations.

[0096] Since the data of hospital patients is often private, joint modeling by multiple parties can better improve the effect of the model and achieve the purpose of protecting patient privacy.

[0097] This example includes two downstream tasks: annotating entities and entity reasoning.

[0098] The application process is as follows:

[0099] Annotated entities are obtained based on the content of the patient's case text after passing through the local federated model and the private model. For example:

[0100] The content of the first conversation is the patient's case text, which is "Stomach flatulence sometimes causes acid reflux and often makes a gurgling sound". According to the case text, after extracting the disease symptom features through the trained local federated model, and then through the NER (Named Entity Recognition) downstream task, the structured text of the disease symptoms "Stomach distension, Acid reflux" is automatically formed;

[0101] The content of the second conversation is the doctor's consultation text, which is "Have you had abdominal pain or hiccups during this period? How is your digestion?" According to the consultation text, after extracting the disease symptom features through the trained local federated model, and then through the NER (Named Entity Recognition) downstream task, the structured text of the disease symptoms "Abdominal pain, Hiccups" is automatically formed;

[0102] Article 3 The content of the conversation is the patient's case text, which reads "Sometimes there is abdominal pain, but only a few hiccups, and sometimes I feel indigested". According to the case text, after extracting the disease symptom features through the trained local federated model, and then through the downstream task of NER (Named Entity Recognition), the structured text of the disease symptoms "abdominal pain, hiccups, indigestion" is automatically formed;

[0103] According to the structured text of the disease symptoms extracted by the three downstream tasks, through the downstream task of text generation, the disease symptom of this patient is obtained as "indigestion", and the corresponding drug treatment plan is "omeprazole, domperidone". Furthermore, through the downstream task of text generation, the disease symptom of this patient is "indigestion", and the corresponding drug treatment plan is "omeprazole, domperidone" to automatically form structured speech, and output "Well, this situation is caused by indigestion. It is recommended to take omeprazole plus domperidone orally for treatment", thus forming an automatic assisted consultation chatbot.

[0104] Correspondingly, the training process of the local federated model and the above application process are corresponding processes. According to each case text and consultation text, entity annotation labels and text generation labels are obtained respectively, and are used to train the local federated model with the training data and labels, and the trained local federated model is used in the actual assisted consultation system.

[0105] As Figure 5 shown, the present disclosure also provides a text prediction device based on horizontal federated learning, including:

[0106] An acquisition unit 510, configured to acquire the text to be predicted;

[0107] A text prediction unit 520, configured to input the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model, and input the federated prediction data into the private model of the downstream task to obtain the prediction result output by the private model;

[0108] Wherein, the local federated model forms a horizontal federated learning network with the public-private key server and the aggregation server;

[0109] The local federated model is trained based on the shared parameter data of the horizontal federated learning network and the training labels of the shared parameter data;;

[0110] The private model is trained based on the local privatized text training data and the text training data labels.

[0111] In the embodiments of the present disclosure, the horizontal federated learning network includes multiple participants. In the embodiments of the present disclosure, the local federated model includes an Embedding layer and at least one Transformer encoder layer. Since the text data of each participant is not shared, the labeled tags are not exactly the same, and the data needs to be protected for privacy, so joint modeling is required. However, in the prior art, the parameter scale of pre-trained models such as bert commonly used in the federated distillation method is too large, and the network communication overhead is too large. And the shared models based on word2vec and elmo commonly used in federated transfer have limited context capture ability. Therefore, in the embodiments of the present disclosure, preferably, the albert model is applied as the local federated model, and transfer learning tasks are performed thereon. In order to achieve a better modeling effect, in other preferred embodiments, other pre-trained models such as the distilbert model can also be used as the local federated model, but the number of parameters may be larger, but the parameter transfer process can be reduced. Taking the albert model as the local federated model, this method based on the pre-trained model can make more full use of the context information of the large-scale pre-trained corpus compared with the shared models of word2vec and elmo in the fedNER federated transfer method. At the same time, the albert model adopts the methods of parameter low-rank decomposition and cross-layer parameter sharing, and the number of parameters is only 1 / 12 of the original bert parameter number. Therefore, it preferably solves the problems of huge parameter number and large network transmission consumption in the joint modeling of federated learning in the prior art.

[0112] In the embodiments of the present disclosure, the downstream tasks are mainly for natural language processing (NLP) tasks. The types of downstream tasks mainly include: text classification, sequence labeling, text generation, and sentence pair classification. The private model processes to obtain corresponding prediction results according to the task type of the downstream task.

[0113] In the embodiments of the present disclosure, various networks can be used for the downstream tasks. For example, when the downstream task is text classification, only the [CLS] vector or the [CLS] vector connected with a linear layer and / or an external softmax layer can be used for the downstream task. In the embodiments of the present disclosure, in order to ensure the prediction effect, the downstream tasks of each participant should be the same and the goals should be as close as possible, but it is not required that the training data labels are exactly the same.

[0114] The embodiments of the present disclosure provide a method for decentralized horizontal federated modeling, which can not only prevent the leakage of gradients during network transmission, but also prevent the central server from decrypting and then aggregating the gradients, realizing theoretically lossless data.

[0115] In the embodiments of the present disclosure, the local federated model includes an embedding layer and at least one encoding layer; specifically, the text prediction unit 520 includes:

[0116] Mapping sub - unit: It is used to input the text to be predicted into the embedding layer, and obtain the reduced - dimensional feature vector output by the embedding layer through initial mapping of the text.

[0117] Feature extraction sub - unit: It is used to input the reduced - dimensional feature vector into at least one of the encoding layers in sequence, and obtain the federated prediction data output by the encoding layer through high - order feature extraction.

[0118] In the embodiments of the present disclosure, the horizontal federated learning network includes multiple participants; the model training unit for each participant includes:

[0119] Shared parameter gradient acquisition sub - unit: It is used to perform local federated model training based on local privatized text training data and text training data labels, and obtain shared parameter gradients.

[0120] In the embodiments of the present disclosure, using the albert model as the local federated model for horizontal federated transfer learning can greatly reduce the number of parameters, thereby reducing communication overhead, and has certain advantages in effect compared with non - pre - trained models.

[0121] The data feature types of the local privatized text training data of each participant are the same. Each participant trains and applies the same model as the local federated model. The first - round local federated model parameters during local federated model training are the model parameters downloaded by the public - private key server, and iterative training starts.

[0122] Public key receiving sub - unit: It is used to receive the public key generated by the public - private key server.

[0123] In the embodiments of the present disclosure, the public - private key server and the summary server are two randomly selected servers among the respective servers of each participant. The purpose of random selection is that before the task, it is not known which participant the public - private key server and the summary server are respectively, and the participants to which the two selected servers belong do not know that they are selected themselves. This can avoid the two selected participants colluding and cheating.

[0124] Gradient encryption sending sub - unit: It is used to homomorphically encrypt the shared parameter gradients according to the public key and send them to the summary server.

[0125] Updated shared parameter receiving sub - unit: It is used to receive the updated shared parameters sent by the public - private key server; wherein, the updated shared parameters are the shared parameter gradients sent by each participant received by the summary server, and gradient mean processing is performed, and the processed mean data is sent to the public - private key server; the public - private key server decrypts the received mean data with the private key.

[0126] Model update sub - unit: It is used to update the updated shared parameters into the local federated model to complete one round of iteration.

[0127] A loop iteration sub-unit for performing at least one round of iteration in a loop until the loss function of the local federated model converges.

[0128] Figure 6 An example of the entity structure diagram of an electronic device is shown as Figure 6 shown. The electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute a text prediction method based on horizontal federated learning. The method includes: obtaining a text to be predicted; inputting the text to be predicted into a local federated model to obtain federated prediction data output by the local federated model, and inputting the federated prediction data into a private model of a downstream task to obtain a prediction result output by the private model; wherein, the local federated model forms a horizontal federated learning network with a public-private key server and an aggregation server; the local federated model is trained based on the shared parameter data of the horizontal federated learning network and the training labels of the shared parameter data; the private model is trained based on local privatized text training data and text training data labels.

[0129] In addition, when the logical instructions in the above-mentioned memory 630 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present disclosure, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0130] On the other hand, the present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the text prediction method based on horizontal federated learning provided by each of the above methods. The method includes: obtaining the text to be predicted; inputting the text to be predicted into a local federated model to obtain federated prediction data output by the local federated model, and inputting the federated prediction data into a private model of a downstream task to obtain a prediction result output by the private model; wherein the local federated model forms a horizontal federated learning network with a public-private key server and an aggregation server; the local federated model is trained based on shared parameter data of the horizontal federated learning network and training labels of the shared parameter data; the private model is trained based on local privatized text training data and text training data labels.

[0131] In another aspect, the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the text prediction method based on horizontal federated learning provided by each of the above. The method includes: obtaining the text to be predicted; inputting the text to be predicted into a local federated model to obtain federated prediction data output by the local federated model, and inputting the federated prediction data into a private model of a downstream task to obtain a prediction result output by the private model; wherein the local federated model forms a horizontal federated learning network with a public-private key server and an aggregation server; the local federated model is trained based on shared parameter data of the horizontal federated learning network and training labels of the shared parameter data; the private model is trained based on local privatized text training data and text training data labels.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0133] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A text prediction method based on horizontal federated learning, characterized in that Applied to a participant, the prediction method includes: Obtain the text to be predicted; Input the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model, and input the federated prediction data into the private model of the downstream task to obtain the prediction result output by the private model; Among them, the local federated model forms a horizontal federated learning network with the public-private key server and the aggregation server; The local federated model is trained based on the shared parameter data of the horizontal federated learning network and the training labels of the shared parameter data; The private model is trained based on the local privatized text training data and the text training data labels.

2. The text prediction method based on horizontal federated learning according to claim 1, wherein The horizontal federated learning network includes multiple participants; For the training process of the local federated model of each participant, it specifically includes: Perform local federated model training based on the local privatized text training data and the text training data labels to obtain the shared parameter gradient; Receive the public key generated by the public-private key server; Homomorphically encrypt the shared parameter gradient according to the public key and send it to the aggregation server; Receive the updated shared parameters sent by the public-private key server; among them, the updated shared parameters are the shared parameter gradients sent by each participant received by the aggregation server, and perform gradient mean processing, and send the processed mean data to the public-private key server; the public-private key server decrypts the received mean data with the private key; Update the updated shared parameters to the local federated model to complete one round of iteration; Loop through at least one round of iteration until the loss function of the local federated model converges.

3. The text prediction method based on horizontal federated learning according to claim 1, characterized in that The local federated model includes an embedding layer and at least one encoding layer; the input of the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model specifically includes: Input the text to be predicted into the embedding layer, and obtain the reduced-dimensional feature vector output by the embedding layer after performing an initial mapping on the text; Input the reduced-dimensional feature vector into at least one of the encoding layers in sequence, and obtain the federated prediction data output by the encoding layer after performing high-order feature extraction.

4. The text prediction method based on horizontal federated learning according to claim 1, characterized in that The task types of the downstream tasks include: text classification, sequence labeling, text generation, and sentence pair classification, and the private model processes to obtain the corresponding prediction results according to the task types of the downstream tasks.

5. A text prediction device based on horizontal federated learning, characterized in that, Applied to a participant, it includes: An acquisition unit for obtaining the text to be predicted; A text prediction unit for inputting the text to be predicted into the local federated model to obtain the federated prediction data output by the local federated model, and inputting the federated prediction data into the private model of the downstream task to obtain the prediction result output by the private model; Among them, the local federated model forms a horizontal federated learning network with the public-private key server and the aggregation server; The local federated model is trained based on the shared parameter data of the horizontal federated learning network and the training labels of the shared parameter data; The private model is trained based on the local privatized text training data and the text training data labels.

6. The text prediction device based on horizontal federated learning according to claim 5, characterized in that, The text prediction unit includes a model training unit for performing model training on the local federated model.

7. The text prediction device based on horizontal federated learning according to claim 6, wherein The horizontal federated learning network includes multiple participants; The model training unit for each participant includes: A shared parameter gradient acquisition subunit, configured to perform local federated model training based on local privatized text training data and text training data labels to obtain shared parameter gradients; A public key receiving subunit, configured to receive the public key generated by the public-private key server; A gradient encryption sending subunit, configured to homomorphically encrypt the shared parameter gradients according to the public key and send them to the aggregation server; An updated shared parameter receiving subunit, configured to receive the updated shared parameters sent by the public-private key server; wherein, the updated shared parameters are obtained by the aggregation server receiving the shared parameter gradients sent by each participant, performing gradient mean processing, and sending the processed mean data to the public-private key server; the public-private key server decrypts the received mean data with the private key; A model update subunit, configured to update the updated shared parameters into the local federated model to complete one round of iteration; A loop iteration subunit, configured to loop through at least one round of iteration until the loss function of the local federated model converges.

8. The text prediction device based on horizontal federated learning according to claim 7, wherein The local federated model includes an embedding layer and at least one encoding layer; the text prediction unit includes a mapping subunit and a feature extraction subunit; The mapping subunit is configured to input the text to be predicted into the embedding layer, and obtain a reduced-dimensional feature vector output by the embedding layer after performing an initial mapping on the text; The feature extraction subunit is configured to sequentially input the reduced-dimensional feature vector into at least one of the encoding layers, and obtain federated prediction data output by the encoding layer after performing high-order feature extraction.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the text prediction method based on horizontal federated learning according to any one of claims 1 to 4.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the text prediction method based on horizontal federated learning according to any one of claims 1 to 4.

Images