Industrial named entity identification method based on privacy protection

Abstract

The invention relates to an industrial named entity recognition method based on privacy protection, and belongs to the technical field of natural language processing, the industrial named entity recognition method is composed of a plurality of local clients and a global model, and the industrial named entity recognition method comprises the following steps: step 1, obtaining an industrial fault detection data set, dividing a training set and distributing labels; 2, establishing an initial model; 3, each local client downloads global model parameters from the server; 4, after each local client sets an iteration round, the private data is used for training in the initial model, and model parameters are uploaded to the server; 5, the server aggregates the model parameters and updates the global model; step 6, repeating the step 3 to the step 5 until a set iteration round is finished; and 7, inputting data needing entity recognition into the trained model, and extracting a test set entity. According to the method, the data privacy of each client is protected while the performance of the local model is improved, and the accuracy of named entity recognition is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to an industrial named entity recognition method based on privacy protection. Background Art

[0002] Named Entity Recognition (NER), also known as entity recognition or entity extraction, aims to identify entities with specific meanings in text and classify them into predefined categories. It provides an effective means of data processing and utilization for the industrial field, which can help transform unstructured data into structured knowledge for further analysis and application. Named entity recognition is the basis for building industrial knowledge graphs, and its accuracy directly affects the quality of knowledge graphs and the subsequent application effects of knowledge graphs.

[0003] In the current NER models, most of the classic methods are based on sequence tagging, but they perform poorly in identifying discontinuous entities. In addition, span-based methods perform well in nested NER problems, but are still limited by the maximum span length, which increases the complexity when dealing with longer entities. Recently, grid tagging-based methods have achieved good results in NER, which defines the discontinuous NER task as a word-to-word relationship recognition problem. However, since the text information in the industrial field contains a large number of domain-specific terms and technical terms, the underlying BERT model does not have a good semantic representation. Moreover, due to the uniqueness of technical terms, existing NER models are prone to boundary recognition errors when identifying these terms, which in turn affects the classification results of entity types.

[0004] In order to build a powerful entity extraction model, large-scale text data is required. Traditional machine learning algorithms need to concentrate all training data on a central server for training, but due to the particularity of the industrial field, its data involves commercial competition and security and privacy issues within the industry, so these data cannot be directly shared to train industrial entity extraction models.

[0005] In summary, existing NER methods face the problem of inaccurate boundary identification when identifying professional terms and technical terms in industrial texts due to the limitations of underlying semantic understanding; the privacy and security requirements of industrial data restrict data sharing. For factories or departments with limited data, it is difficult to effectively train high-quality NER models. Summary of the invention

[0006] The purpose of the present invention is to provide an industrial named entity recognition method based on privacy protection to address the problems existing in the prior art. No data or parameter exchange occurs between clients, and there is no need to directly share sensitive original data. Named entity recognition is performed on industrial data under the premise of protecting the privacy of all parties.

[0007] The technical solution is as follows:

[0008] An industrial named entity recognition method based on privacy protection consists of two parts: multiple local clients and a global model. Different clients represent different companies or departments, including the following steps:

[0009] Step 1: Obtain the industrial fault detection dataset, divide it into training sets and assign labels;

[0010] Step 2: Establish a grid named entity recognition (NER) model based on a dual-axis attention mechanism as the initial model;

[0011] Step 3: Each local client downloads the global model parameters from the server;

[0012] Step 4: After each local client sets the iteration round, it uses its private data to perform local named entity recognition training in the initial model and uploads the trained model parameters to the server;

[0013] Step 5: The server uses an algorithm to aggregate the model parameters trained by each local client and update the global model;

[0014] Step 6: Repeat steps 3 to 5 until the set iteration round ends, that is, the training of the local grid named entity recognition model is completed;

[0015] Step 7: Input the data required for entity recognition as a test set into the trained local grid named entity recognition model to extract the test set entities.

[0016] Furthermore, the grid named entity recognition model based on the dual-axis attention mechanism in step 2 includes an encoder module, a character pair representation module, a dual-axis attention module and a predictor module.

[0017] Furthermore, the encoder module is composed of BERT and Bi-LSTM. After the sentence is input into BERT, the context representation of each word is obtained as H = {h1, h2, ..., h N},in represents the representation of the i-th character, d h The embedding dimensions of the characters are then passed through a Bi-LSTM layer to further enhance the context encoding.

[0018] Furthermore, the character pair representation module includes combining the relative character position information between the original character pairs, the character pair position information distinguishing the upper and lower triangle areas, and the character pair representation information, and passing them through a multi-layer perceptron to obtain the position-aware representation of the character pairs. Specifically, the conditional layer normalization CLN mechanism is used to generate the grid representation V of the character pairs. ij , calculate the relative character position information between character pairs Distinguish the character pair position information of the upper and lower triangle areas The three are passed through a multi-layer perceptron MLP to obtain a position-aware representation of the character pair. R=MLP(V;E p ; E q ).

[0019] Furthermore, the dual-axis attention module defines a single-axis attention ij , represents the attention weight of the character pair relationship at position (i, j), which is obtained by calculating the attention of rows and columns respectively: a ij =MultiHead(R ij , row i , row j )+MultiHead(R ij ,col i ,col j ), where MultiHead, row i and col j Respectively represent multi-head attention, character pair represents R ij , and then calculate the i-th row and j-th column of R ij The corresponding symmetrical character pair R ji Perform axial focus and calculate a ji , and compare them with R ij The concatenation operation is performed to capture the structural information and relevance of the text while retaining its semantic representation, thereby improving the performance and effectiveness of the model in the grid named entity recognition task.

[0020] Furthermore, the predictor module constructs the context representation M: Biaffine predictor and MLP predictor are used to calculate character pairs (h i ,h j ) and combine them as the final prediction y ij =Softmax(y′ ij +y″ ij ), where the NER loss for each sentence X is The calculation is: N is the length of the sentence, and Represents character pairs (w i , w j )’s predicted label probability and target label, Represents the rth relationship set in the predefined The loss function of the kth client is: Where |D k | represents local private data Dk The number of sentences in , θ represents all the parameters of the local model.

[0021] Furthermore, the local named entity recognition training in step 4 includes jointly identifying all possible entities through four predefined character pair labels and character pair boundary labels, with the row characters of the grid as the subject and the column characters as the object, and defining the following labels: the next adjacent word NNW indicates that the subsequent character in the row is the column character; the tail word THW-* indicates that the row character is the tail character of the current entity, and the column character is the head character of the current entity, and * represents the type of the current entity; the previous adjacent word PNW indicates that the previous character in the row is the column character; the head and tail word HTW indicates that the row character is the head character of the current entity, and the column character is the tail character of the current entity; *-S / *-E indicates that the type of the entity starting / ending with the current character is *.

[0022] Furthermore, the aggregation algorithm of the server in step 5 is FedAvg, which is used to update the global model parameters Beneficial effects:

[0023] 1) In the present invention, there is no exchange of data or parameters between clients, and there is no need to directly share sensitive original data, so that local models with less data and poorer performance can learn knowledge and experience from other clients, thereby improving the performance of the local model while protecting the data privacy of each client.

[0024] 2) The grid-based dual-axis attention module can better capture the correlation of character pair relationships within the grid. The labeling system fully considers entity boundaries, internal character pair relationships and relationship types, so that more complex character-character relationships can be predicted, enhancing the accuracy of named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an overall schematic diagram of the present invention.

[0026] Figure 2 Schematic diagram of predefined character pair labels and character pair boundary labels.

[0027] Figure 3 Flowchart of the steps of the named entity recognition method. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. The directions or positional relationships indicated by the terms "upper", "lower", "front", "back", "left", "right", "bottom", "inside", "outside", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention:

[0029] like Figure 1 The method for industrial named entity recognition based on privacy protection is shown, which consists of two parts: multiple local clients and a global model, and includes the following steps:

[0030] Step 1: Obtain the industrial fault detection dataset, divide it into training sets and assign labels;

[0031] Step 2: Establish a grid named entity recognition (NER) model based on a dual-axis attention mechanism as the initial model;

[0032] Step 3: Each local client downloads the global model parameters from the server;

[0033] Step 4: After each local client sets the iteration round, it uses its private data to perform local named entity recognition training in the initial model and uploads the trained model parameters to the server;

[0034] Step 5: The server uses an algorithm to aggregate the model parameters trained by each local client and update the global model;

[0035] Step 6: Repeat steps 3 to 5 until the set iteration round ends, that is, the training of the local grid named entity recognition model is completed;

[0036] Step 7: Input the data required for entity recognition as a test set into the trained local grid named entity recognition model to extract the test set entities.

[0037] The grid named entity recognition model based on the dual-axis attention mechanism in step 2 includes an encoder module, a character pair representation module, a dual-axis attention module, and a predictor module. The encoder module consists of BERT and Bi-LSTM. After the sentence is input into BERT, the context representation of each word is obtained as H = {h1, h2, ..., h N},in represents the representation of the i-th character, d hThe embedding dimension of the characters is represented, and these representations are passed through the Bi-LSTM layer to further enhance the context encoding. The character pair representation module includes combining the relative character position information between the original character pairs, the character pair position information that distinguishes the upper and lower triangle areas, and the character pair representation information, and passing them through a multi-layer perceptron to obtain the position-aware representation of the character pairs. Specifically, the conditional layer normalization CLN mechanism is used to generate the grid representation V of the character pairs. ij , calculate the relative character position information between character pairs Distinguish the character pair position information of the upper and lower triangle areas The three are passed through a multi-layer perceptron MLP to obtain a position-aware representation of the character pair. R=MLP(V;E p ; E q ). The dual-axis attention module defines a single-axis attention ij , represents the attention weight of the character pair relationship at position (i, j), which is obtained by calculating the attention of rows and columns respectively: a ij =MultiHead(R ij , row i , row j )+MultiHead(R ij ,col i ,col j ), where MultiHead, row i and col j Respectively represent multi-head attention, character pair represents R ij , and then calculate the i-th row and j-th column of R ij The corresponding symmetrical character pair R ji Perform axial focus and calculate a ji , and compare them with R ij The concatenation operation captures the structural information and relevance of the text while retaining its semantic representation, thereby improving the performance and effectiveness of the model in the grid named entity recognition task. The predictor module builds the context representation M: Biaffine predictor and MLP predictor are used to calculate character pairs (h i ,h j ) and combine them as the final prediction y ij =Softmax(y′ ij +y″ ij ), where the NER loss for each sentence X is The calculation is: N is the length of the sentence, and Represents character pairs (w i , wj )’s predicted label probability and target label, Represents the rth relationship set in the predefined The loss function of the kth client is: Where |D k | represents local private data D k The number of sentences in , θ represents all the parameters of the local model.

[0038] The local named entity recognition training in step 4 includes jointly identifying all possible entities through four predefined character pair labels and character pair boundary labels, taking the row characters of the grid as the subject and the column characters as the object, and defining the following labels: the next adjacent word NNW indicates that the subsequent character in the row is the column character; the tail word THW-* indicates that the row character is the tail character of the current entity, and the column character is the head character of the current entity, and * represents the type of the current entity; the previous adjacent word PNW indicates that the previous character in the row is the column character; the head and tail word HTW indicates that the row character is the head character of the current entity, and the column character is the tail character of the current entity; *-S / *-E indicates that the type of the entity starting / ending with the current character is *.

[0039] The aggregation algorithm of the server in step 5 is FedAvg, which is used to update the global model parameters

[0040] Example 1: The model of the present invention is implemented based on PyTorch, and the machine graphics card configuration is GTX4090. Bert-base-Chinese is used as the training language model. It includes two parts: a grid NER based on a dual-axis attention mechanism as a local client and a global model. Different clients represent different companies or departments, and perform the following steps respectively:

[0041] (1) Obtain an industrial fault detection dataset, assign labels, and partition the dataset;

[0042] (2) The grid NER model based on the dual-axis attention mechanism is used as the initial model;

[0043] (3) Each client first downloads the global model parameters;

[0044] (4) Each local client performs local NER training on its private data and only uploads the model parameters to the server. No data or parameter exchange occurs between clients.

[0045] In the local training phase, all possible entities are first jointly identified by four predefined character pair labels and character pair boundary labels.

[0046] The row characters of the grid are taken as the subject and the column characters are taken as the object. The following tags are defined:

[0047] a. Next Neighboring Word (NNW): Indicates that the subsequent character in the row is the column character.

[0048] b. Tail and head word (THW-*): Indicates that the row characters are the tail characters of the current entity, the column characters are the head characters of the current entity, and (*) represents the type of the current entity.

[0049] c. Previous Neighboring Word (PNW): Indicates that the previous character in the row is the column character.

[0050] d. Head-Tail Word (HTW): Indicates that the row character is the head character of the current entity, and the column character is the tail character of the current entity.

[0051] e.*-S / *-E: Indicates that the type of the entity starting / ending with the current character is *.

[0052] The grid-based NER model consists of four modules: encoder module, character pair representation module, biaxial attention module, and predictor module. The encoder module consists of BERT and Bi-LSTM. Given an input sample, it is input into the pre-trained BERT to obtain the contextual representation of each word. These representations are then passed through the Bi-LSTM layer to further enhance the contextual encoding. After the input sentence is processed by the encoder module, the sentence representation is H = {h1, h2, ..., h N},in, represents the representation of the i-th character, d h Represents the embedding dimension of the character.

[0053] Use the conditional layer normalization (CLN) mechanism to generate grid representations V of character pairs. ij , calculate the relative character position information between character pairs Distinguish the character pair position information of the upper and lower triangle areas The three are passed through a multi-layer perceptron MLP to obtain a position-aware representation of the character pair. R=MLP(V;E p ; E q )

[0054] In the dual-axis attention module, define a single-axis attention a ij As follows, it represents the attention weight of the character pair relationship at position (i, j), which is obtained by calculating the attention for rows and columns separately:

[0055] a ij =MultiHead(R ij , row i , row j )+MultiHead(Rij ,col i ,col j ), where MultiHead, row i and col j Respectively represent multi-head attention, character pair represents R ij Then, for R ij The corresponding symmetrical character pair R ji Perform axial focus and calculate a ji . and compare them with R ij The concatenation operation is performed to capture the structural information and relevance of the text while retaining its semantic representation, thereby improving the performance and effectiveness of the model in the NER task.

[0056] Construct the context representation M:

[0057] Biaffine predictor and MLP predictor are used to calculate character pairs (h i ,h j ) and combine them as the final prediction y ij =Softmax(y′ ij +y″ ij ).

[0058] Among them, the NER loss of each sentence X is The calculation of is as follows: N is the length of the sentence, and Represents character pairs (w i , w j )’s predicted label probability and target label, Represents the rth relationship set in the predefined

[0059] (5) The server uses the FedAvg algorithm to aggregate the received local model parameters and update the global model. The global model parameters are updated as follows: Among them, θ m For each local client parameter, |D m |、|D n | are the number of sample sentences for each local client;

[0060] (6) Repeat (3)-(5) until the iteration round ends. The loss function of the kth client is: Where |D k | represents local private data D k The number of sentences in , θ represents all parameters of the local model;

[0061] (7) Obtain the test set data, input it into the trained local NER model, and extract the test set entities.

[0062] In a local centralized training environment, experiments were conducted on fault detection data of two industrial datasets A and B, and the NER model of the present invention was compared with 6 other baseline NER models. The results are shown in Table 1. It can be seen that the NER performance of the present invention on a single platform is better than other baseline models. The grid labeling-based method is better at annotating information and is more suitable for processing complex NER scenarios. The label representation embedding module designed by the present invention not only considers the word-word relationship, but also considers the relationship between word-label and label-label through finer-grained labels. In addition, a dual-axis attention mechanism is used to better capture the correlation of these relationships.

[0063] Table 1

[0064]

[0065] As shown in Table 2, by comparing the performance of each local client model on dataset A with and without federated training, the results show that for NER tasks in both IID and non-IID settings, the model trained using federated learning outperforms each local model without federated training. Due to differences in data between clients, some clients may have advantages in specific indicators, but the comprehensive F1 score of the model after federated training significantly exceeds the F1 score of a single platform. This shows that federated training allows clients to learn knowledge and experience from each other by passing model parameters without directly sharing sensitive raw data, thereby enhancing the overall performance of each client model.

[0066] Table 2

[0067]

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the principles and spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A privacy-preserving industrial named entity recognition method, characterized by: It consists of two parts: multiple local clients and a global model, including the following steps: Step 1: Obtain the industrial fault detection dataset, divide it into training sets and assign labels; Step 2: Establish a grid named entity recognition model based on a dual-axis attention mechanism as the initial model; Step 3: Each local client downloads the global model parameters from the server; Step 4: After each local client sets the iteration round, it uses its private data to perform local named entity recognition training in the initial model and uploads the trained model parameters to the server; Step 5: The server uses an algorithm to aggregate the model parameters trained by each local client and update the global model; Step 6: Repeat steps 3 to 5 until the set iteration round ends, that is, the training of the local grid named entity recognition model is completed; Step 7: Input the data required for entity recognition as a test set into the trained local grid named entity recognition model to extract the test set entities.

2. The privacy-preserving industrial named entity recognition method according to claim 1, characterized in that: The grid named entity recognition model based on the dual-axis attention mechanism in step 2 includes an encoder module, a character pair representation module, a dual-axis attention module and a predictor module.

3. The privacy-preserving industrial named entity recognition method according to claim 2, characterized in that: The encoder module is composed of BERT and Bi-LSTM. After the sentence is input into BERT, the context representation of each word is obtained as H = {h1, h2, ..., h N },in represents the representation of the i-th character, d h Represents the embedding dimension of the characters, and then passes these representations through the Bi-LSTM layer.

4. The privacy-preserving industrial named entity recognition method according to claim 2, characterized in that: The character pair representation module comprises combining the relative character position information between the original character pairs, the character pair position information distinguishing the upper and lower triangle areas, and the character pair representation information, and passing them through a multi-layer perceptron to obtain the position perception representation of the character pairs, specifically using the conditional layer normalization CLN mechanism to generate the grid representation V of the character pairs. ij , calculate the relative character position information between character pairs Distinguish the character pair position information of the upper and lower triangle areas The three are passed through a multi-layer perceptron MLP to obtain a position-aware representation of the character pair. R=MLP(V;F p ; E q ).

5. The privacy-preserving industrial named entity recognition method according to claim 2, characterized in that: The dual-axis attention module defines a single-axis attention a ij , represents the attention weight of the character pair relationship at position (i, j), which is obtained by calculating the attention of rows and columns respectively: a ij =MultiHea(R ij , row i , row j )+MultiHead(R ij ,col i ,col j ), where MultiHead, row i and col j Respectively represent multi-head attention, character pair represents R ij , and then calculate the i-th row and j-th column of R ij The corresponding symmetrical character pair R ji Perform axial focus and calculate a ji , and compare them with R ij The concatenation operation is performed to capture the structural information and relevance of the text while preserving its semantic representation.

6. The privacy-preserving industrial named entity recognition method according to claim 2, characterized in that: The predictor module constructs the context representation M: Biaffine predictor and MLP predictor are used to calculate character pairs (h i ,h j ) and combine them as the final prediction y ij =Softmax(y′ ij +y″ ij ), where the NER loss for each sentence X is The calculation is: N is the length of the sentence, and Respectively represent character pairs (w i , w j )’s predicted label probability and target label, Represents the rth relationship set in the predefined The loss function of the kth client is: Where |D k | represents local private data D k The number of sentences in , θ represents all the parameters of the local model.

7. The privacy-preserving industrial named entity recognition method according to claim 1, characterized in that: The local named entity recognition training in step 4 includes jointly identifying all possible entities through four predefined character pair labels and character pair boundary labels, taking the row characters of the grid as the subject and the column characters as the object, and defining the following labels: the next adjacent word NNW indicates that the subsequent character in the row is the column character; the tail word THW-* indicates that the row character is the tail character of the current entity, and the column character is the head character of the current entity, and * represents the type of the current entity; the previous adjacent word PNW indicates that the previous character in the row is the column character; the head and tail word HTW indicates that the row character is the head character of the current entity, and the column character is the tail character of the current entity; *-S / *-E indicates that the type of the entity starting / ending with the current character is *.

8. The privacy-preserving industrial named entity recognition method according to claim 1, characterized in that: The aggregation algorithm of the server in step 5 is FedAvg, which is used to update the global model parameters

Images