A Chinese named entity recognition method for special equipment based on pre-training model
Through the Chinese named entity recognition method of special equipment based on pre-trained models, combined with BERT, BiLSTM and CRF, the problem of insufficient labeling data in the special equipment field is solved, the accuracy of naming entity recognition is improved, and the automatic identification and knowledge of special equipment entities is realized The construction of the map.
Patent Information
- Application Number
- CN202110893472.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-04
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-08-04
AI Technical Summary
The lack of labeled data sets and insufficient sample feature information in the field of special equipment has made it difficult for existing models to effectively identify named entities and the entity recognition accuracy is not high.
A Chinese named entity recognition method for special equipment based on pre-trained models is adopted, word vector representation is obtained through the BERT model, and entity annotation and decoding are performed in combination with BiLSTM and CRF conditional random fields, and the optimal label sequence is obtained using the Viterbi algorithm.
Effectively learn text feature information in the field of special equipment, improve the accuracy of naming entity recognition, solve the problems of small data scale and insufficient sample characteristics, and realize the automatic recognition of special equipment entities, which facilitates the construction of knowledge graphs.
Smart Images

Figure CN113609859B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of knowledge graphs, deep learning, special equipment, entity recognition, etc., and particularly provides a Chinese named entity recognition method for special equipment based on a pre-training model. Background Art
[0002] Knowledge graph is a semantic network that reveals the relationship between entities. It can formally describe things in the real world and their relationships. It is now used to refer to various large-scale knowledge bases. Knowledge graph is a structured semantic knowledge base that is used to describe concepts and their relationships in the physical world in symbolic form. Its basic components are (entity, relationship, entity) triples, as well as entities and their related attribute value pairs. Entities are connected to each other through relationships to form a network of knowledge structures. Knowledge graphs are widely used in intelligent search, intelligent question and answer, personalized recommendations, and other aspects.
[0003] Entity recognition is a fundamental task in the process of building a knowledge graph, which aims to extract entities with specific meanings and types from unstructured text. Today, the main problem facing Chinese entity recognition is the different language characteristics of Chinese and English. Chinese has no spaces and the basic unit is a character, so it is very difficult to understand a sentence at the character level.
[0004] Entity recognition models based on deep learning mainly include entity recognition methods based on recurrent neural networks, long short-term memory networks, and convolutional neural networks. BiLSTM-CRF (Computer Science, 2015) first applied the bidirectional long short-term memory network model to entity recognition. BiLSTM can use past and future input features, and CRF can use sentence-level annotation information. LSTM-CNNs-CRF (Association for Computational Linguistics, 2016) uses CNN to encode the character information of words into character-level features and then inputs BiLSTM with word embedding to construct the contextual features of each character. Finally, CRF uses the contextual label information to decode the label of the entire sentence. Lattice-structured LSTM (Association for Computational Linguistics, 2018) encodes the character information and potential vocabulary information of the input in the sequence. Compared with the previous input representation based only on a single word, here the vocabulary information can be explicitly used and the entity segmentation error is avoided. WC-LSTM (Association for Computational Linguistics, 2019) uses four different strategies to encode word information into fixed-size vectors, so that it can be trained in batches and adapted to various application scenarios. Multi-digraph Gazetteers (Association for Computational Linguistics, 2019) uses dictionaries and graph neural networks to incorporate word information into the NER system. LR-CNN (International Joint Conference on Artificial Intelligence) proposes a CNN network with a rethinking mechanism, which processes the words in sentences and lexicons through CNN and resolves lexicon conflicts through the rethinking mechanism. CAN-NER (North American Association for Computational Linguistics, 2019) proposes a convolutional neural network architecture based on an attention mechanism for Chinese entity recognition.
[0005] Entity recognition in the field of special equipment is a key step in building a knowledge graph of special equipment. Its task is to extract specific types of named entities from unstructured data of special equipment. At present, there is no public annotated dataset in the field of special equipment, the scale of domain text data is small, and the sample feature information is insufficient. These problems make it difficult for existing models to effectively identify named entities in the field of special equipment. Summary of the invention
[0006] In view of the problems in the field of special equipment, such as the small number of annotations and low entity recognition accuracy, the present invention proposes a special equipment entity recognition model based on a pre-training model. The model can more fully learn the text feature information in the field of special equipment, extract special equipment named entities, and thus construct a knowledge graph.
[0007] In order to solve the technical problem, the present invention provides the following technical solution:
[0008] A method for recognizing Chinese named entities of special equipment based on a pre-trained model comprises the following steps:
[0009] 1) Chinese named entity dataset annotation:
[0010] 1.1) Data acquisition:
[0011] A large amount of data related to the field of special equipment is extracted from documents, books, literature, and web pages, and the data is cleaned and deduplicated as the data source for Chinese entity recognition.
[0012] 1.2) Chinese named entity labeling strategy:
[0013] Define the Chinese named entity annotation strategy BIEOS, the annotation categories include:
[0014] B indicates that the character is the start position of the entity;
[0015] I means that the character is in the middle of the entity;
[0016] E indicates the end position of the character entity;
[0017] O means that the character entity does not belong to any entity type;
[0018] S means that the character alone constitutes an entity;
[0019] x indicates the type of entity in which the character resides;
[0020] 1.3) Entity category labeling strategy:
[0021] The types of named entities in the field of special equipment are defined, including four categories: C, P, F and A, which represent composition, parameters, faults and accidents, respectively, as shown in Table 1 below:
[0022] Table 1 Entity types in the field of special equipment
[0023] Marking symbols Entity Class Category Definition Example C composition Equipment components Traction machine, landing door, etc. P parameter Equipment parameters Running speed, load capacity, etc. F Fault Equipment failure Stop running, inaccurate leveling, etc. A ACCIDENT Elevator Accidents Pier bottom, top, etc.
[0024] 1.3) Training data set annotation:
[0025] The entities are labeled according to the entity labeling strategy and entity type, and the training set is labeled. The labeling method is shown in Table 2 below:
[0026] Table 2 Labeling strategy
[0027] Entity Class Start tag Middle Label Conclusion composition BC IC EC parameter BP IP EP Fault BF IF EF ACCIDENT BA I A EA
[0028] 2) Word vector representation based on BERT pre-trained model:
[0029] BERT is a new language model that pre-trains deep bidirectional representations by jointly adjusting the bidirectional Transformers in all layers. The BERT structure is constructed using BERT's BertModel class. The input Chinese character sequence for special equipment is then input into the BERT feature representation layer, and each Chinese character in the text is encoded to obtain a vector representation of the Chinese character:
[0030] x=(x1,x2,x3…x n ) (1)
[0031] 3) biLSTM context encoding:
[0032] biLSTM is a bidirectional long short-term memory network, which is suitable for temporal data modeling and can better capture long-distance bidirectional semantic dependencies;
[0033] 3.1) Input of biLSTM:
[0034] The biLSTM receives the vector representation x of the Chinese character sequence output from the BERT layer, inputs it into two biLSTM units, and initializes the hidden state h0;
[0035] 3.2) Bidirectional encoding of word vector sequences based on biLSTM:
[0036] The output representation formula of the biLSTM hidden layer is as follows:
[0037] i t =σ(W ix x t +W ih h t-1 +b i ) (2)
[0038] f t =σ(Wfx x t +W fh h t-1 +b f ) (3)
[0039] o t =σ(W ox x t +W oh h t-1 +b o ) (4)
[0040]
[0041]
[0042]
[0043] Among them, W and b represent the weight matrix and bias vector connecting the two layers respectively, σ is the sigmoid activation function, is the dot multiplication operation, x t is the input vector, i t , f t and t They are the input gate, forget gate and output gate at time t respectively. represents the state at time t, h t Then it is the output of the hidden layer at time t;
[0044] After calculating the input sequence in order and in reverse order, two different hidden layer representations h are obtained: 前 and h 后 , and then the final hidden layer representation is obtained by vector concatenation, as shown below:
[0045] h i =[h i前 +h i后 ] (8)
[0046] where h i前 ,h i后 They represent the forward and backward hidden layer representations corresponding to the i-th character, [] represents phasor concatenation, and h i represents the final hidden layer representation.
[0047] The obtained hidden layer representation is mapped into a vector with a dimension equal to the number of output labels through a linear layer, and then normalized through the Softmax function to output the probability score P of each label. The formula is as follows:
[0048] P = log(Softmax(h i )) (9)
[0049] Thus, the probability value of each label is obtained;
[0050] 3.3) Minimize the loss function:
[0051] The NLLLoss loss function is used to calculate the difference between the probability value of each label and the target result. The NLLLoss function receives two tensors. The first tensor is the probability value of the label, and the second is the target result. According to the value of the target result, the elements at the corresponding positions in the probability value of the label are found, and the sum is taken to get the average value. The loss function value is minimized by the stochastic gradient descent method. The formula is as follows:
[0052] Loss = NLLLoss (output, target) (10) where output is the probability value of the label and target is the target result;
[0053] 4) Decode label sequence based on CRF conditional random field:
[0054] 4.1) Learning state matrix and transfer matrix:
[0055] The CRF (Conditional Random Field) conditional random field is used to learn the context label information. The CRF algorithm involves the state matrix and the transfer matrix.
[0056] In the conditional random field, for an input sequence x = (x1, x2, x3…x n ), the probability of the i-th character in the sequence corresponding to k specific tags constitutes a state matrix, which is the probability score of the current character corresponding to each tag, expressed as:
[0057]
[0058] The higher the score, the higher the probability that the word corresponds to the label. n represents the length of the sequence, i represents the i-th character in the sequence, and y i Then it represents the label corresponding to the character, and k represents the number of labels; Represents a matrix of dimension n×k;
[0059] In the conditional random field, for a specific k labels, the transfer score from label i to label i+1 constitutes a transfer matrix, which is used to represent the transfer relationship between labels, expressed as:
[0060]
[0061] Among them, the higher the score, the more likely the label y is. i Move to next label y i+1 The higher the probability;
[0062] 4.2) Calculate the maximum probability of sequence label:
[0063] According to the state matrix and the transfer matrix, for an input sequence x = (x1, x2, x3...x n ), the corresponding label sequence is y = (y1, y2, y3…y n ) is scored as follows:
[0064]
[0065] The probability of the y sequence label is obtained by normalizing with the Softmax function. The formula is as follows:
[0066]
[0067] in, Represents the real path mark value of the sequence, Y x represents the set of all possible path labels; e is an exponential function;
[0068] 4.3) Minimize the loss function:
[0069] During the training process, the following loss function is used to maximize the likelihood probability of the correct label, and the value of the loss function is minimized by the stochastic gradient descent method. The calculation formula is:
[0070]
[0071] 4.4) Decode the final tag sequence based on the Viterbi algorithm:
[0072] After the Chinese named entity recognition model is trained, all possible label sequences for each Chinese character are obtained. Then, the Viterbi algorithm is used to decode the label sequence, that is, to calculate the optimal sequence in the label sequence. The algorithm records the optimal sequence of each observed label at the current moment each time, and uses the dynamic programming method to reduce unnecessary repeated calculations. At the end, the optimal value at the last moment is traced back to the starting position. After the backtracking is completed, the path from the beginning to the end is the optimal.
[0073] For a set of unknown label sequences x, the Viterbi algorithm is used to obtain the label sequence with the highest total prediction score among all label sequences. The calculation formula is as follows:
[0074]
[0075] Finally, output the final entity label sequence y corresponding to the Chinese character sequence x * .
[0076] The beneficial effects of the present invention are as follows: the entity recognition method applied to the field of special equipment provided by the present invention integrates a pre-trained language model, a long short-term memory network and a conditional random field. Its advantage is that the pre-trained model is trained from a large-scale unlabeled corpus in an unsupervised manner, which fully considers the characteristic relationship between characters, words, sentences and sentences, and enhances the semantic representation of word vectors, which can effectively solve the problem of small data scale and insufficient sample feature information in the field of special equipment. BiLSTM is effective in capturing long-distance information and solving the gradient vanishing problem for named entity recognition tasks. The conditional random field enables the model to learn the constraint information between labels, and ensures the rationality of the final predicted label by considering the relationship between adjacent labels. The entity recognition model finally trained can be used for automatic entity recognition of special equipment, which is convenient for the subsequent construction of the special equipment knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] Figure 1 is the pre-trained model graph;
[0078] Figure 2 It is the entity recognition annotation result. DETAILED DESCRIPTION
[0079] Taking typical special equipment of elevator as an example, combined with Figure 1 and Figure 2 The present invention is further described.
[0080] A method for Chinese entity recognition of special equipment based on a pre-trained training model comprises the following steps:
[0081] 1) Chinese named entity dataset annotation:
[0082] 1.1) Data acquisition:
[0083] Extract a large amount of data related to the field of special equipment from documents, books, literature, and web pages, clean and deduplicate the data, and use it as the data source for Chinese entity recognition;
[0084] 1.2) Chinese entity labeling strategy:
[0085] The Chinese named entity annotation strategy and entity category annotation strategy are used to annotate the text. For example, the annotation results of the text sequence "caused problems with the traction machine" are as follows: Figure 2 As shown;
[0086] 2) Word vector representation based on BERT pre-trained language model:
[0087] The text sequence "caused a problem with the tractor" is input into the BERT pre-training layer to obtain the vector representation of each character: x = (x1, x2, ... x9, x 10 ),like Figure 1 As shown;
[0088] 3) Bidirectional encoding of word vector sequences based on biLSTM:
[0089] 3.1) Input of biLSTM:
[0090] The biLSTM receives the vector representation x of the Chinese character sequence output from the BERT layer, inputs it into two biLSTM units, and randomly initializes the hidden state h0;
[0091] 3.2) Bidirectional encoding of word vector sequences based on biLSTM:
[0092] The vector x is input into two bidirectional LSTM units, and the forward and reverse hidden layer outputs h corresponding to each character are calculated using formulas (2)(3)(4)(5)(6)(7) i前 and h i后 , output h of the forward and reverse hidden layers of each character i前 ,h i后 The final hidden layer representation h is concatenated using formula (8) i , is mapped into a vector with the dimension of the number of output labels through a linear layer, and normalized by the Softmax function using formula (9), and the corresponding probability score P of each label is output;
[0093] 3.3) Minimize the loss function:
[0094] Finally, according to formula (10), the NLLLoss loss function is used to calculate the difference between the probability value of each label and the target result. The loss function value is minimized by the stochastic gradient descent method until convergence;
[0095] 4) Label sequence prediction based on CRF conditional random field, the process is as follows:
[0096] 4.1) Learning state matrix and transfer matrix:
[0097] 1) State matrix: State matrix The output P of step 3) is obtained as shown in the following table, P 1,B-C =0.9 means the probability that the character w1 is labeled as "BC" is 0.9;
[0098] BC 0.9 0.4 0.4 0.2 IC 0.5 0.8 0.5 0.1 EC 0.1 0.6 0.9 0.3 O 0.5 0.2 0.1 0.9 … 0.3 0.1 0.1 0.4 <![CDATA[w1]]> <![CDATA[w2]]> <![CDATA[w3]]> <![CDATA[w4]]>
[0099] 2) Transfer matrix: Before training the model, the transfer score matrix is randomly initialized. During the training process, all random scores in this matrix will be updated, and finally the transfer score matrix is obtained. During the training process, in order to make the transfer score matrix more robust, two additional labels are added to the data: "START" and "END". "START" indicates the beginning of a sentence. Note that this does not refer to the first word of the sentence. The first word comes after "START". Similarly, "END" represents the end of the sentence. The transfer score matrix is shown in the following table. B-C,I-C =0.9 indicates the score of label “BC” transferred to “IC”, that is, the label after “BC” is likely to be “IC”;
[0100] START BC IC EC O END START 0 0.8 0.007 0.005 0.9 0.08 BC 0 0.4 0.9 0.8 0.1 0.009 IC 0.001 0.4 0.75 0.78 0.12 0.005 EC 0.002 0.6 0.2 0.3 0.78 0.6 O 0 0.65 0.008 0.07 0.85 0.7 END 0 0 0 0 0 0
[0101] 4.2) Calculate sequence label probability:
[0102] According to the state matrix and the transfer matrix, for the input sequence x=(x1, x2, ..., x9, x 10 ), the corresponding label sequence is y = (y1, y2, ..., y9, y 10 ), the score of the label sequence corresponding to the input sequence is obtained by formula (11) as Score, and then the probability P of the y sequence label is obtained by formula (12);
[0103] 4.3) Minimize the loss function:
[0104] According to the loss function of formula (13), during the training process, the model will be continuously updated and the value of the loss function will be minimized by the stochastic gradient descent method until convergence;
[0105] 4.4) Decode the final tag sequence based on the Vibit algorithm:
[0106] After the model training is completed, given a set of unknown label sequences x, the Viterbi algorithm, i.e., formula (14), is used to obtain the label sequence with the highest total score among all label sequences as the final annotation result y for named entity recognition in the field of special equipment. * ;
[0107] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.
Claims
1. A method for recognizing Chinese named entities of special equipment based on a pre-training model, characterized in that: The following steps are involved: 1) Chinese named entity dataset annotation: 1.1) Data acquisition: Extract text data related to the field of special equipment, clean and deduplicate the data, use it as a data set for Chinese entity recognition, and divide the data set into a training set and a test set; 1.2) Chinese named entity labeling strategy: Define the Chinese named entity annotation strategy BIEOS, the annotation categories include: B indicates that the character is the start position of the entity; I means that the character is in the middle of the entity; E indicates the end position of the character entity; O means that the character entity does not belong to any entity type; S means that the character alone constitutes an entity; x indicates the type of entity in which the character resides; 1.3) Entity category labeling strategy: Define the types of named entities in the field of special equipment, including C, P, F and A, which represent the composition, parameters, faults and accidents of special equipment respectively; 1.3) Training data set annotation: Label the entities according to the entity labeling strategy and entity type, and label the training set in the dataset; 2) Word vector representation based on BERT pre-trained model: The BERT pre-training model pre-trains deep bidirectional representation by jointly adjusting the bidirectional Transformer in all layers, constructs the BERT structure using BERT's BertModel class, and then inputs the special equipment Chinese character sequence into the BERT feature representation layer, encoding each Chinese character in the text to obtain the vector representation of the Chinese character: x=(x1,x2,x3…x n ) (1) 3) biLSTM context encoding: 3.1) Input of biLSTM: The biLSTM receives the vector representation x of the Chinese character sequence output from the BERT layer, inputs it into two biLSTM units, and initializes the hidden state h0; 3.2) Bidirectional encoding of word vector sequences based on biLSTM: The output representation formula of the biLSTM hidden layer is as follows: i t =σ(W ix x t +W ih h t-1 +b i ) (2) f t =σ(W fx x t +W fh h t-1 +b f ) (3) o t =σ(W ox x t +W oh h t-1 +b o ) (4) Among them, W and b represent the weight matrix and bias vector connecting the two layers respectively, σ is the sigmoid activation function, is the dot multiplication operation, x t is the input at time t, c t is the cell state at time t, is the temporary cell state at time t, h t represents the hidden layer state at time t, f t 、i t and t Represents the forget gate, memory gate and output gate at time t; The input sequence is calculated in order and in reverse order to obtain two different hidden layer representations h 前 and h 后 , and then the final hidden layer representation is obtained by vector concatenation. The formula is as follows: h i =[h i前 +h i后 ] (8) where h i前 ,h i后 They represent the forward and backward hidden layer representations corresponding to the i-th character, [] represents phasor concatenation, and h i represents the final hidden layer representation; The obtained hidden layer representation is mapped into a vector with a dimension of the number of output labels through a linear layer, and then normalized by the Sofimax function to obtain the probability P of each label. The calculation formula is as follows: P=log(Softmax(h i )) (9) 3.3) Minimize the loss function: The NLLLoss loss function is used to calculate the difference between the probability value of each label and the target result. The NLLLoss function receives two tensors. The first tensor is the probability value of the label, and the second is the target result. According to the value of the target result, the elements at the corresponding positions in the probability value of the label are found, and the sum is taken to get the average value. The loss function value is minimized by the stochastic gradient descent method. The formula is as follows: Loss two NLLLoss(output, target) (10) Among them, output is the probability value of the label, and target is the target result; 4) Decode label sequence based on CRF conditional random field: 4.1) Learning state matrix and transfer matrix: The CRF conditional random field is used to learn the label information of the context. The CRF algorithm involves the state matrix and the transition matrix. In the conditional random field, for an input sequence x = (x1, x2, x3…x n ), the probability of the i-th character in the sequence corresponding to k specific tags constitutes a state matrix, which is the probability score of the current character corresponding to each tag, expressed as: The higher the score, the higher the probability that the word corresponds to the label. n represents the length of the sequence, i represents the i-th character in the sequence, and y i It indicates the label corresponding to the character, k indicates the number of labels, Represents a matrix of dimension n×k; In the conditional random field, for a specific k labels, the transfer score from label i to label i+1 constitutes a transfer matrix, which is used to represent the transfer relationship between labels, expressed as: Among them, the higher the score, the more likely the label y is. i Move to next label y i+1 The higher the probability; 4.2) Calculate the maximum probability of sequence label: According to the state matrix and the transfer matrix, for an input sequence x = (x1, x2, x3...x n ), the corresponding label sequence is y = (y1, y2, y3…y n ) is scored as follows: The probability of the y sequence label is obtained by normalizing with the Softmax function. The formula is as follows: in, Represents the real path mark value of the sequence, Y x represents the set of all possible path labels, and e is an exponential function; 4.3) Minimize the loss function: During the training process, the following loss function is used to maximize the likelihood probability of the correct label, and the value of the loss function is minimized by the stochastic gradient descent method. The calculation formula is: 4.4) Decode the final tag sequence based on the Viterbi algorithm: After the Chinese named entity recognition model training is completed, all possible label sequences for each Chinese character are obtained; then, the Viterbi algorithm is used to decode the label sequence, that is, to calculate the optimal sequence in the label sequence. The algorithm records the optimal sequence of each observed label at the current moment each time, and uses the dynamic programming method to reduce unnecessary repeated calculations. At the end, the optimal value at the last moment is traced back to the starting position. After the backtracking is completed, the path from the beginning to the end is the optimal; For a set of unknown label sequences x, the Viterbi algorithm is used to obtain the label sequence with the highest total prediction score among all label sequences. The calculation formula is as follows: Finally, output the final entity label sequence y corresponding to the Chinese character sequence x * .
