A federated relation extraction method under data imbalance conditions

Abstract

The present invention discloses a method for extracting federated relations under data imbalance conditions, comprising: inputting terminal local training data at a terminal input layer; using a pre-trained language model at a terminal encoding layer to symbolically mark the head and tail entities involved in the relation, and concatenating the hidden layer vectors corresponding to the marks as a sentence encoding vector; at a terminal output layer, classifying the relations according to the feature vectors output by the terminal encoding layer, and optimizing the model through cross-entropy loss; at a terminal credibility calculation layer, the terminal first downloads a public validation set from a server, fixes the model parameters after the previous optimization training, and calculates the prediction credibility of each type of relation on the validation set; at a terminal prediction layer, the terminal predicts unlabeled data, multiplies the prediction result by the prediction credibility, obtains a final prediction result, and uploads it to a server; the server obtains a weak annotation result of the unlabeled data set, and sends the data set with the weak label to each terminal for local training.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing in artificial intelligence, and relates to a method for extracting federated relations under data imbalance conditions. Background Art

[0002] Relation extraction is a key task in natural language processing, serving a range of downstream applications such as knowledge graph construction and information retrieval. This task is defined as selecting appropriate relationship category labels for a pair of candidate entities from a set of candidate relationship categories based on the contextual description of the pair in the text. Recent developments in deep learning have sparked interest in using neural networks for supervised learning for relationship extraction. In practice, these methods often require collecting annotations from different platforms for centralized training, which violates some data privacy protection regulations. Therefore, how to keep the data local to the platform while combining different platforms for relationship extraction has become a difficult challenge.

[0003] To solve this problem, a federated relation extraction method under data imbalance conditions is proposed. Federated relation extraction means that multiple relation extraction platforms participate in the process. These platforms share knowledge without sharing local data and help each other improve the accuracy of relation extraction.

[0004] In this type of research, early studies mainly involved uploading local models to a server through each platform (i.e., terminal), averaging or weighting the model parameters on the server to integrate them, and then sending the integrated model to each platform to achieve knowledge sharing between terminals. However, this method brings huge time overhead, so this solution uses a knowledge distillation method to aggregate the model knowledge of each platform. The knowledge aggregation method of knowledge distillation is to place a shareable unlabeled dataset on the server side, and each platform predicts the unlabeled data and uploads the prediction results to the server. The server is responsible for aggregating the prediction results. The aggregated data and prediction results are sent to each platform for local training. In this way, knowledge sharing is carried out, and knowledge exchange can be carried out by only uploading and downloading the prediction results, which greatly reduces communication overhead.

[0005] Some previous studies have used knowledge distillation to federate the knowledge of terminals, but they treated each terminal equally. However, this solution takes into account that the data distribution of each terminal is uneven, and each terminal cannot be treated equally. The imbalance here means that the data distribution of each terminal is not independent and identically distributed, that is, their data distribution is not the same. Therefore, the prediction confidence of each terminal for different relationship categories is different. After the local model of each terminal platform is trained and the result is predicted for the unlabeled data set in the server, the prediction result needs to be corrected according to the terminal's prediction confidence for different relationship categories. The corrected prediction result is a more reliable prediction result. Therefore, the server cannot simply add up the predictions of each terminal directly, but needs to add up the more reliable terminal results after correction as the weak label of the final unlabeled data set, and then send it to each terminal for further local training. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to propose a method for federated relationship extraction under data imbalance conditions. The method proposes a knowledge aggregation method in which the terminal adjusts the prediction results according to its credibility on different relationships. First, the terminal fine-tunes based on local data to obtain a local model with relationship extraction capabilities; second, the local model parameters are fixed, and a small-scale validation set is downloaded from the server to verify its credibility on different relationships. The credibility is measured in the way that the smaller the prediction cross entropy loss on a certain relationship category, the higher its credibility; then, with the parameters fixed, the local model downloads a large-scale unlabeled dataset from the server and predicts these unlabeled data. The prediction results are corrected according to the credibility of the previous step and uploaded to the server; finally, when the server receives the prediction results from all terminals, it aggregates these results and forms weak labels for the unlabeled dataset. The unlabeled data and these weak labels are then sent to all terminals for local training to achieve the purpose of knowledge sharing and mutual learning.

[0007] Based on the above purpose, a federated relation extraction method under data imbalance conditions includes the following steps:

[0008] Step 1: In the terminal input layer, input the terminal local training data;

[0009] Step 2: At the terminal encoding layer, use the pre-trained language model to symbolize the head and tail entities involved in the relationship, and concatenate the hidden layer vectors corresponding to the labels as the sentence encoding vector;

[0010] Step 3: At the terminal output layer, the relationship is classified according to the feature vector output by the terminal encoding layer, and the model is optimized through cross entropy loss;

[0011] Step 4: In the terminal credibility calculation layer, the terminal first downloads the public validation set from the server, fixes the model parameters optimized and trained in step 3, and calculates the prediction credibility of each type of relationship on the validation set;

[0012] Step 5: In the terminal prediction layer, the terminal downloads the unlabeled data set from the server and predicts the unlabeled data. The prediction result is multiplied by the prediction credibility to obtain the final prediction result and upload it to the server.

[0013] In step 6, after receiving the prediction results from all terminals, the server adds up the terminal prediction results to obtain the weakly labeled results of the unlabeled dataset, and sends the dataset with weak labels to each terminal for local training.

[0014] Specifically, in step 2, the terminal encoding layer adds identifiers [E1], [ / E1], [E2], [ / E2] before and after the head and tail entities of the input text sentence, indicating the start and end of the head entity and the start and end of the tail entity respectively. The sentence is represented as:

[0015] s′={[CLS],w1,...,w i-2 ,[E1],w i ,...,w j ,[ / E1],...,[E2],w k ,...,w l ,[ / E2],...,w n ,[SEP]}

[0016] Among them, the symbols [CLS] and [SEP] indicate the beginning and end of the sentence, w i Represents the i-th word of the text, inputs the sentence into the pre-trained language model, and obtains the hidden layer vector sequence of the sentence:

[0017]

[0018] The hidden layer vectors corresponding to the identifiers [E1] and [E2] are serially input into a fully connected layer, h [CLs] , h [SEP] Indicates the hidden layer state corresponding to the characters [CLS] and [SEP], h i Representation word w i The corresponding hidden layer state, Represent the hidden layer states corresponding to the start and end markers of the head and tail entities, and obtain the final sentence feature vector:

[0019]

[0020] Where W and b are trainable parameters, [:] represents a concatenation operation, setting the vector dimension of the feature vector s to d;

[0021] Furthermore, in step 3, the terminal output layer determines the relationship category according to the sentence feature vector s through a multi-classifier:

[0022] p(r|s,Θ)=softmax(W o s+b o )

[0023] Among them, W o and b o is a trainable parameter, Θ is the set of all training parameters, and p(r|s,Θ) represents the probability that s belongs to category r, r∈R, where R is the set of all relation categories, including N relation categories. The r with the highest probability value is the category of s.

[0024] Furthermore, the loss function in the output layer is defined as the cross entropy loss function:

[0025]

[0026] Where, I(r=r * ) represents an indicator function, when r is The true category r * It is 1 when , otherwise it is 0, and D represents the entire local training set.

[0027] Preferably, in step 4, after training the local data, the terminal needs to evaluate the performance of the local model on each relationship category to obtain the credibility of the terminal model on different relationship categories. The terminal first downloads the public verification set from the service. The verification set is small in size and does not consume too much resources for data annotation. For a certain relationship r i , and its verification data is expressed as The cross entropy loss is used to represent its credibility. The smaller the loss, the higher its credibility. The loss is expressed as:

[0028]

[0029] The model parameters are fixed during the calculation process. Based on the loss, we can obtain the confidence distribution of the terminal model on each relationship:

[0030]

[0031] The i-th element c i Indicates that the terminal is in relation r i The credibility of the predictions.

[0032] Preferably, in step 5, the terminal prediction layer first downloads a large-scale unlabeled dataset U from the server. For each data in the unlabeled dataset, the terminal model performs prediction under fixed parameters. The prediction process is expressed as:

[0033] f(s;Θ)=softmax(W o s+b o )

[0034] Among them, s is the sentence feature vector obtained by the terminal encoding layer of the data in the unlabeled dataset U, is the probability distribution of the terminal model on all relationship category sets R. According to the terminal credibility, the prediction results of the unlabeled data are updated again to obtain:

[0035]

[0036] After the terminal obtains the prediction result, it uploads the prediction result to the server. It represents the predicted probability distribution after confidence correction, and ⊙ represents the bitwise multiplication operation of the elements.

[0037] Furthermore, in step 6, after receiving the prediction results of all terminals, the server aggregates the prediction results of all terminals. Assuming that there are M terminals in total, for the jth terminal, the prediction result represents The prediction results after aggregation are expressed as:

[0038]

[0039] After summing the relationship probability distributions for all terminals, the index with the highest probability value is selected as the weakly labeled relationship label. After assigning weak labels to the unlabeled dataset, the weak labels are distributed to all terminals, and local model training is performed according to steps 2 to 4. By aggregating and distributing prediction results, terminals can learn from other terminals. To ensure that the model achieves better relationship extraction results, steps 1 to 6 are iterated multiple times until all terminal models converge to optimal relationship extraction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the overall model structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0043] An embodiment of the present invention provides a method for extracting federated relations under data imbalance conditions. The method includes two parts: several terminals and a server. Each terminal includes five parts: an input layer, an encoding layer, an output layer, a credibility calculation layer, and a prediction layer. In the input layer, local training data of the terminal is input, and the input training data is encoded in the encoding layer. In the output layer, relations are classified according to the feature vector output by the encoding layer, and the model is optimized using cross-entropy loss. In the credibility calculation layer, the terminal first downloads a public small-scale validation set from the server. The parameters of the local model trained above are fixed, and the prediction credibility of each type of relationship is calculated on the validation set. In the prediction layer, the terminal downloads a large-scale unlabeled data set from the server and predicts the unlabeled data. The prediction result is multiplied by the credibility to obtain the final prediction result and upload it to the server. The server always maintains a public small-scale validation set and a large-scale unlabeled data set. After receiving the prediction results of all terminals, the terminal prediction results are added to obtain a weakly labeled result of the unlabeled data set. The data set with weak labels is sent to each terminal for local training. The server aggregates the knowledge of all terminals using an unlabeled dataset, allowing each terminal to learn from the others. Furthermore, because the data distribution of each terminal is uneven (non-IID), the confidence levels of their predictions for different relationships vary. Therefore, a confidence calculation layer is designed to calculate the confidence levels of different terminals on different relationship predictions, making the overall knowledge aggregation of each terminal more accurate. This method enables each terminal to learn from each other more accurately, improving the overall accuracy of relationship extraction.

[0044] like Figure 1 As shown in FIG, a federated relation extraction method under data imbalance conditions includes the following steps:

[0045] Step 1: In the terminal input layer, input the terminal local training data;

[0046] Step 2: At the terminal encoding layer, use the pre-trained language model to symbolize the head and tail entities involved in the relationship, and concatenate the hidden layer vectors corresponding to the labels as the sentence encoding vector;

[0047] Step 3: At the terminal output layer, the relationship is classified according to the feature vector output by the encoding layer, and the model is optimized through cross entropy loss;

[0048] Step 4: In the terminal credibility calculation layer, the terminal first downloads a public small-scale validation set from the server. The parameters of the previously trained local model are fixed, and the prediction credibility of each type of relationship is calculated on the validation set.

[0049] Step 5: At the terminal prediction layer, the terminal downloads a large-scale unlabeled data set from the server and makes predictions on the unlabeled data. The prediction result is multiplied by the aforementioned credibility to obtain the final prediction result and upload it to the server.

[0050] In step 6, after receiving the prediction results from all terminals, the server adds up the terminal prediction results to obtain the weakly labeled results of the unlabeled dataset, and sends the dataset with weak labels to each terminal for local training.

[0051] The overall model structure diagram of the embodiment of the present invention is as follows Figure 2 shown.

[0052] Specifically, in step 2, the terminal encoding layer adds identifiers [E1], [ / E1], [E2], [ / E2] before and after the head and tail entities of the input text sentence, indicating the start and end of the head entity and the start and end of the tail entity respectively. The sentence can be represented as:

[0053] s'={[CLS],w1,...,w i_ 2,[E1],w i ,...,w j ,[ / E1],...,[E2],w k ,...,w l ,[ / E2],...,w n ,[SEP]}

[0054] The symbols [CLS] and [SEP] represent the beginning and end of a sentence. The sentence is input into the pre-trained language model to obtain the hidden layer vector sequence of the sentence:

[0055]

[0056] The hidden layer vectors corresponding to the identifiers [E1] and [E2] are serially input into a fully connected layer to obtain the final sentence feature vector:

[0057]

[0058] Among them, W and b are trainable parameters, [:] represents a splicing operation, and sets the vector dimension of the feature vector s to d.

[0059] Furthermore, in step 3, the terminal output layer determines the relationship category according to the sentence feature vector s through a multi-classifier:

[0060] p(r|s,Θ)=softmax(W o s+b o )

[0061] Among them, W o and b ois a trainable parameter, Θ is the set of all training parameters, and p(r|s,Θ) represents the probability that s belongs to category r, r∈R, where R is the set of all relation categories, including N relation categories. The r with the highest probability value is the category of s.

[0062] Furthermore, the loss function in the output layer is defined as the cross entropy loss function:

[0063]

[0064] Where, I(r=r * ) represents an indicator function, when r is The true category r * It is 1 when , otherwise it is 0, and D represents the entire local training set.

[0065] Preferably, in step 4, after training the local data, the terminal needs to evaluate the performance of the local model on each relationship category to obtain the credibility of the terminal model on different relationship categories. The terminal first downloads the public verification set from the service. The verification set is small in size and does not consume too much resources for data annotation. For a certain relationship r i , and its verification data is expressed as The cross entropy loss is used to represent its credibility. The smaller the loss, the higher its credibility. The loss is expressed as:

[0066]

[0067] The model parameters are fixed during the calculation process. Based on the loss, we can obtain the confidence distribution of the terminal model on each relationship:

[0068]

[0069] The i-th element c i Indicates that the terminal is in relation r i The credibility of the predictions.

[0070] Preferably, in step 5, the terminal prediction layer first downloads a large-scale unlabeled dataset U from the server. For each data in the unlabeled dataset, the terminal model performs prediction under fixed parameters. The prediction process is expressed as:

[0071] f(s;Θ)=softmax(W o s+b o )

[0072] Among them, s is the sentence feature vector obtained by the terminal encoding layer of the data in the unlabeled dataset U, is the probability distribution of the terminal model on all relationship category sets R. According to the terminal credibility, the prediction results of the unlabeled data are updated again to obtain:

[0073]

[0074] After the terminal obtains the prediction result, it uploads the prediction result to the server.

[0075] Furthermore, in step 6, after receiving the prediction results of all terminals, the server aggregates the prediction results of all terminals. Assuming that there are M terminals in total, for the jth terminal, its prediction result is represented by The prediction results after aggregation are expressed as:

[0076]

[0077] After summing the relationship probability distributions for all terminals, the index with the highest probability value is selected as the weakly labeled relationship label. After assigning weak labels to the unlabeled dataset, the weak labels are distributed to all terminals, and local model training is performed according to steps 2 through 4. By aggregating and distributing prediction results, terminals can learn from other terminals. To ensure optimal relationship extraction, steps 1 through 6 can be repeated multiple times until all terminal models converge to optimal relationship extraction performance.

[0078] The above embodiment is an implementation method of the method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be considered equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for extracting federated relations under data imbalance conditions, characterized by: The following steps are involved: Step 1: In the terminal input layer, input the terminal local training data; Step 2: At the terminal encoding layer, use the pre-trained language model to symbolize the head and tail entities involved in the relationship, and concatenate the hidden layer vectors corresponding to the labels as the sentence encoding vector; the terminal encoding layer adds markers [E1], [ / E1], [E2], [ / E2] before and after the head and tail entities of the input text sentence, indicating the start and end of the head entity and the start and end of the tail entity respectively. The sentence is represented as: s′={[CLS],w1,…,w i-2 ,[E1],w i ,…,w j ,[ / E1],...,[E2],w k ,...,w l ,[ / E2],…,w n ,[SEP]} Among them, the symbols [CLS] and [SEP] indicate the beginning and end of the sentence, w i Represents the i-th word of the text, inputs the sentence into the pre-trained language model, and obtains the hidden layer vector sequence of the sentence: The hidden layer vectors corresponding to the identifiers [E1] and [E2] are serially input into a fully connected layer, h [CLS] , h [SEP] Indicates the hidden layer state corresponding to the characters [CLS] and [SEP], h i Representation word w i The corresponding hidden layer state, Represent the hidden layer states corresponding to the start and end markers of the head and tail entities, and obtain the final sentence feature vector: Among them, W and b are trainable parameters, [:] represents a splicing operation, setting the vector dimension of the feature vector s to d Step 3: At the terminal output layer, the relationship is classified according to the feature vector output by the terminal encoding layer, and the model is optimized through cross entropy loss; Step 4: In the terminal credibility calculation layer, the terminal first downloads the public validation set from the server, fixes the model parameters optimized and trained in step 3, and calculates the prediction credibility of each type of relationship on the validation set; Step 5: In the terminal prediction layer, the terminal downloads the unlabeled data set from the server and predicts the unlabeled data. The prediction result is multiplied by the prediction credibility to obtain the final prediction result and upload it to the server. In step 6, after receiving the prediction results from all terminals, the server adds up the terminal prediction results to obtain the weakly labeled results of the unlabeled dataset, and sends the dataset with weak labels to each terminal for local training.

2. The method for extracting federated relations under data imbalance conditions according to claim 1, characterized in that: In step 3, the terminal output layer determines the relationship category according to the sentence feature vector s through a multi-classifier: p(r|s,Θ)=softmax(W o s+b o ) Among them, W o and b o is a trainable parameter, Θ is the set of all training parameters, p(r|s,Θ) represents the probability that s belongs to category r, r∈R, R is the set of all relation categories, including N relation categories.

3. The method for extracting federated relations under data imbalance conditions according to claim 2, characterized in that: The loss function is defined as the cross entropy loss function in the output layer: Where, I(r=r * ) represents an indicator function, when r is The true category r * It is 1 when , otherwise it is 0, and D represents the entire local training set.

4. The method for extracting federated relations under data imbalance conditions according to claim 2, characterized in that: In step 4, after training the local data, the terminal needs to evaluate the performance of the local model on each relationship category to obtain the credibility of the terminal model on different relationship categories. The terminal first downloads the public verification set from the service; for a certain relationship r i , and its verification data is expressed as The cross entropy loss is used to represent its credibility. The smaller the loss, the higher its credibility. The loss is expressed as: The model parameters are fixed during the calculation process, and the confidence distribution of the terminal model on each relationship can be obtained based on the loss: The i-th element c i Indicates that the terminal is in relation r i The credibility of the predictions.

5. The method for extracting federated relations under data imbalance conditions according to claim 4, characterized in that: In step 5, the terminal prediction layer first downloads the unlabeled dataset U from the server. For each data in the unlabeled dataset, the terminal model performs prediction under fixed parameters. The prediction process is expressed as: f(s;Θ)=softmax(W o s+b o ) Among them, s is the sentence feature vector obtained by the terminal encoding layer of the data in the unlabeled dataset U, is the probability distribution of the terminal model on all relationship category sets R. According to the terminal credibility, the prediction results of the unlabeled data are updated again to obtain: After the terminal obtains the prediction result, it uploads the prediction result to the server. It represents the predicted probability distribution after confidence correction, and ⊙ represents the bitwise multiplication operation of the elements.

6. The method for extracting federated relations under data imbalance conditions according to claim 5, characterized in that: In step 6, after receiving the prediction results of all terminals, the server aggregates the prediction results of all terminals. Assuming there are M terminals in total, for the jth terminal, the prediction result is expressed as The prediction results after aggregation are expressed as: After summing up the relationship probability distributions of all terminals, the index with the largest probability value is selected as its weakly labeled relationship label. After weakly labeling the unlabeled dataset, it is sent to all terminals and local model training is performed according to steps 2 to 4. By aggregating and sending the prediction results, the terminals can learn from other terminals.

7. The method for extracting federated relations under data imbalance conditions according to claim 1, characterized in that: Steps 1 to 6 are iterated multiple times until all terminal models converge to the best relation extraction performance.

Images