Fault text named entity recognition device and method based on field BERT

Abstract

The invention discloses a fault text named entity recognition device and method based on the field BERT, and belongs to the field of named entity recognition, the fault text named entity recognition device comprises a processor and a memory, a model program is stored in the memory, and when the model program is loaded by the processor, fault text named entity recognition based on the field BERT is operated, the system specifically comprises a character presentation layer module, a sequence modeling layer module and a label reasoning layer module. According to the method, the entity identification accuracy in the fault text data is improved.

Description

Technical Field

[0001] The present invention relates to the field of named entity recognition, and more specifically, to a device and method for fault text named entity recognition based on domain BERT. Background Art

[0002] Given the complexity of electronic information equipment maintenance work, high experience dependence, and lack of advanced fault diagnosis tools, there are currently many difficulties and pain points. The urgent need for enterprises and users to quickly resolve faults further highlights the limitations of the existing maintenance system. In China, although fault diagnosis technology has made great progress, research on the use of knowledge graphs for fault diagnosis is still relatively scarce. By constructing a maintenance process knowledge graph, traditional maintenance experience is combined with modern information technology to promote the digital transformation of electronic equipment maintenance business. This not only helps to improve the level of intelligence in maintenance work, but also brings more efficient and convenient maintenance service experience to enterprises and users. One of the core technologies for constructing a maintenance process fault knowledge graph is named entity recognition technology. Due to the high professionalism of electronic information equipment fault texts, as well as the frequent repeated character combinations and the small number of character types, character vectors are sparse in high-dimensional space, resulting in low accuracy of entity recognition in fault texts. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a device and method for fault text named entity recognition based on domain BERT, thereby improving the accuracy of entity recognition in fault text data.

[0004] The object of the present invention is achieved through the following solutions:

[0005] A device for named entity recognition of fault text based on domain BERT, comprising: a processor and a memory, wherein a model program is stored in the memory, and when the model program is loaded by the processor, the device runs the named entity recognition of fault text based on domain BERT, specifically comprising a character representation layer module, a sequence modeling layer module and a label inference layer module;

[0006] The character representation layer module is based on the BERT model, and a fully connected layer is added after the output layer of the BERT model and a convolution operation is performed, which is recorded as the DBERT model character representation layer module. The DBERT model is used to perform domain pre-training on character features in text data;

[0007] The sequence modeling layer module is based on the BiLSTM model. In the BiLSTM model, the time series is simultaneously input into the forward LSTM and the backward LSTM to obtain the hidden state outputs of the forward and reverse sequences; then the hidden state output sequences in these two directions are merged to form a comprehensive hidden state sequence;

[0008] The label inference layer module is based on the CRF model and uses the CRF model to calculate the probabilities of all possible label sequences and select the most likely sequence as the final prediction.

[0009] Furthermore, it also includes: a training data input module, in which the text is first segmented before word vector embedding, so that the input of the DBERT model is the text token after segmentation, and each token generates a corresponding embedding vector through the DBERT model, and these vectors capture the semantic information of the token in its context;

[0010] H=DBERT(tokens) (1)

[0011] Where H is the hidden state matrix output by DBERT, and each row corresponds to the embedding vector of a token in the input sequence.

[0012] Furthermore, a single character is used as a delimiter and special markers are introduced at the beginning and end of each sentence to comply with the model input specification.

[0013] Furthermore, in the BiLSTM model, the time series is simultaneously input into the forward LSTM and the backward LSTM, thereby obtaining the hidden state outputs of the forward and reverse sequences; then the hidden state output sequences in these two directions are merged to form a comprehensive hidden state sequence, specifically including:

[0014] At each time step, the BiLSTM model computes the output in the following way:

[0015] ① Forward LSTM processing sequence:

[0016]

[0017] ②Backward LSTM processing sequence:

[0018]

[0019] ②Merge the hidden states in two directions:

[0020]

[0021] in, express, Indicates that x t express, express, Indicates that h t express;

[0022] In this way, BiLSTM is able to capture contextual information.

[0023] Furthermore, the method of using the CRF model to calculate the probabilities of all possible label sequences and selecting the most likely sequence as the final prediction specifically includes the following sub-steps:

[0024] In the CRF model, given an input sequence x, the goal is to predict the most likely output label sequence y; CRF achieves this by defining a joint probability distribution over the entire label sequence, rather than modeling each label separately;

[0025] The core is to calculate the conditional probability P(y|x), which represents the probability of output sequence y given the input sequence x. The calculation method of this probability is shown in formula (5):

[0026]

[0027] Among them, Z(x) is the normalization factor to ensure that the sum of probabilities is 1. The calculation method is shown in formula (6):

[0028]

[0029] f k (y t-1 ,y t ,x,t) is a feature function, which usually depends on the input feature of the current position, the current label and the previous label; λ k is the weight of the feature function, which is learned through training data;

[0030] Use CRF to calculate the probability of all possible label sequences and select the most likely sequence as the final prediction;

[0031] y * = arg max y p(y|O) (7)

[0032] Among them, y * is the optimal label sequence predicted by the model, O is the given observation sequence, that is, the output of the BiLSTM layer; the model searches for the one with the highest probability of appearing in the entire label sequence under the condition of a given observation sequence.

[0033] A method for named entity recognition of fault text based on domain BERT, based on the device as described in any one of the above items, further comprising the following steps:

[0034] S1, the DBERT model achieves effective dimensionality reduction and refinement of fault text features through feature compression strategy and learns and adapts to the unique features and expertise of fault text through domain pre-training;

[0035] S2, extracts context-related features of characters in the text through the DBERT model, combines these features with the specific representation of the characters after weighted operations, and uses BiLSTM to encode the features bidirectionally;

[0036] S3, outputs the label sequence with the highest probability through CRF as the final prediction result.

[0037] The beneficial effects of the present invention include:

[0038] The method of the present invention proposes a DBERT model, which specifically achieves effective dimensionality reduction and refinement of fault text features by introducing a feature compression strategy, and fully learns and adapts to the unique features and professionalism of fault texts through domain-specific pre-training. Afterwards, the context-related features of the characters in the text are extracted through the DBERT model, and these features are combined with the specific representation of the characters after weighted operations. At the same time, the method combining the BiLSTM and CRF models is used to perform named entity recognition, and the features are bidirectionally encoded using BiLSTM. Finally, the label sequence with the highest probability is output through CRF as the final prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of a model architecture of a device according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram for DBERT-based embedding vector generation. DETAILED DESCRIPTION

[0042] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0043] The specific implementation process of the present invention is as follows:

[0044] The present invention proposes a device for fault text named entity recognition based on domain BERT, comprising a processor and a memory, wherein a model program is stored in the memory, and when the model program is loaded by the processor, a fault text named entity recognition process is executed. Figure 1 As shown in the figure, the model includes: DBERT character representation layer, BiLSTM sequence modeling layer and CRF label inference layer.

[0045] In the design concept of the character representation layer, BERT is a deep bidirectional pre-trained language representation model based on the Transformer model. Compared with traditional word embedding methods, the BERT model can better capture the semantic and contextual information in sentences. Since repeated character combinations often appear in fault text data and there are relatively few character types, the generated character vectors show excessive sparsity in high-dimensional space. Sparse data in high-dimensional space may negatively affect the training effect and model performance. To address this problem, the present invention improves the Chinese BERT model by adding a fully connected layer after the output layer of BERT and performing a convolution operation, which helps to reduce the feature dimension, thereby achieving compression and accuracy of the feature representation. Finally, the representation of each character is converted from the 768-dimensional high-dimensional vector output by the original BERT model to a more compact 512-dimensional vector. This improvement not only reduces the sparsity of the vector, but also helps the model to more effectively process and identify key information in the fault text. The improved model is named DBERT model in the present invention. The structure of the DBERT model is as follows Figure 2 shown.

[0046] In the present invention, fault text data such as "SKJK-1601, intermediate frequency digital exchange module, AD2.895.00030GC, module DSP2 locked fault, device D92 was found to be damaged, and device D92 was replaced" contains rich context information. Usually, the part after "found to be" describes the cause object, and the content after "replace" involves the measure object. It can be found that the fault text data has relatively obvious contextual features. Therefore, the present invention uses the DBERT model to perform domain-specific pre-training on the character features in the text data, which can fully capture and express the context of the text to improve the accuracy of entity recognition in the fault text data.

[0047] Unlike languages ​​like English that use spaces to separate words, Chinese has no obvious word boundary separators, and the entire sentence is a continuous sequence of characters, which makes it difficult for the model to directly learn effective language rules and vocabulary boundaries. In order to more accurately obtain the vocabulary features in the fault text, before embedding the word vector, the present invention contemplates first segmenting the text, and specifically uses the precise segmentation mode in the Jieba segmentation tool library to process the fault text data.

[0048] The input of the DBERT model is the text token after word segmentation. Each token generates a corresponding embedding vector through the DBERT model, which captures the semantic information of the token in its context.

[0049] H=DBERT(tokens) (1)

[0050] Where H is the hidden state matrix output by DBERT, and each row corresponds to the embedding vector of a token in the input sequence. In order to use the DBERT model to obtain the characteristics of each character, a single character is used as a separator, and special markers ([CLS] and [SEP]) are introduced at the beginning and end of each sentence to meet the model input specifications. For example, for the text "[CLS] User reported abnormal noise problem of rear axle [SEP]". When the model is output, the special characters at the beginning and end are omitted, only the vectors of the middle characters are retained, and a certain proportion of the vocabulary is randomly replaced with a special [mask] marker.

[0051] In the design concept of the sequence modeling layer, the traditional long short-term memory network (LSTM) can only process information from the past to the present moment due to its unidirectionality, thus limiting the model's performance in processing sequence data. To overcome this limitation, the BiLSTM model deploys both forward and reverse LSTM networks on the time series, allowing the model to integrate historical and future information, thereby capturing contextual features more comprehensively. In the BiLSTM model, the time series is simultaneously input into the forward LSTM and the backward LSTM to obtain the hidden state outputs of the forward and reverse sequences. Subsequently, the hidden state output sequences in these two directions are merged to form a comprehensive hidden state sequence. At each time step, the BiLSTM calculates the output in the following way:

[0052] ① Forward LSTM processing sequence:

[0053]

[0054] ②Backward LSTM processing sequence:

[0055]

[0056] ②Merge the hidden states in two directions:

[0057]

[0058] In this way, BiLSTM is able to capture contextual information, thereby improving the model's overall understanding of the data.

[0059] In the design concept of the label inference layer, in the named entity recognition task, the BiLSTM sequence processing model has the ability to handle long-distance dependency information. However, BiLSTM mainly focuses on extracting features from text, but fails to fully consider the constraints between labels. For example, in entity labeling, the beginning of an entity is usually marked as "B" to indicate the beginning, and "I" to indicate the inside of the entity. BiLSTM may not be able to correctly handle this dependency of label sequences without additional mechanisms.

[0060] At this point, the addition of the CRF layer is particularly important for the NER model. The CRF layer can use the transition matrix to define the transition probability between labels, thereby enforcing the legality rules in the label sequence. For example, the CRF layer can prevent illogical label conversions through learning, such as directly going from "O" to "I" or from "B" to another "B" without "I" in between. In this way, the CRF layer ensures that the model can take into account the correlation between labels when predicting labels, significantly enhancing the accuracy of the model's recognition of entity boundaries. The CRF layer not only uses the distribution predicted by the model for each time step, but also uses the transition probability of labels between adjacent time steps. It can be regarded as adding a global normalization when predicting sequence annotations, ensuring that the final predicted label sequence is optimal overall.

[0061] In the CRF model, given an input sequence x, the goal is to predict the most likely output label sequence y. CRF achieves this by defining a joint probability distribution over the entire label sequence, rather than modeling each label separately.

[0062] The core is to calculate the conditional probability P(y|x), which represents the probability of output sequence y given the input sequence x. The calculation method of this probability is shown in formula (5):

[0063]

[0064] Among them, Z(x) is the normalization factor to ensure that the sum of probabilities is 1. The calculation method is shown in formula (6):

[0065]

[0066] f k (y t-1 ,y t ,x,t) is a feature function, which usually depends on the input features at the current position, the current label, and the previous label. k is the weight of the feature function, which is learned through training data.

[0067] Use CRF to calculate the probabilities of all possible label sequences and select the most likely sequence as the final prediction.

[0068] y * = arg max y p(y|O) (7)

[0069] Among them, y * is the optimal label sequence predicted by the model, and O is the given observation sequence, that is, the output of the BiLSTM layer. The model looks for the one with the highest probability of appearing in the entire label sequence under the condition of a given observation sequence.

[0070] In summary, the present invention proposes a method for named entity recognition based on Domain BERT (DBERT). First, the DBERT model achieves effective dimensionality reduction and refinement of fault text features through feature compression strategy and fully learns and adapts to the unique features and professionalism of fault text through domain-specific pre-training. After that, the context-related features of characters in the text are extracted through the DBERT model, and these features are combined with the specific representation of the characters after weighted operation, and the features are bidirectionally encoded using BiLSTM, and finally the label sequence with the highest probability is output as the final prediction result through CRF.

[0071] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0072] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0073] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. A fault text named entity recognition device based on domain BERT, characterized in that: include: A processor and a memory, wherein the model program is stored in the memory, and when the model program is loaded by the processor, the fault text named entity recognition based on the domain BERT is executed, specifically including a character representation layer module, a sequence modeling layer module, and a label inference layer module; The character representation layer module is based on the BERT model, and a fully connected layer is added after the output layer of the BERT model and a convolution operation is performed, which is recorded as the DBERT model character representation layer module. The DBERT model is used to perform domain pre-training on character features in text data; The sequence modeling layer module is based on the BiLSTM model. In the BiLSTM model, the time series is simultaneously input into the forward LSTM and the backward LSTM to obtain the hidden state output of the forward and reverse sequences; Then the hidden state output sequences in these two directions are merged to form a comprehensive hidden state sequence; The label inference layer module is based on the CRF model and uses the CRF model to calculate the probabilities of all possible label sequences and select the most likely sequence as the final prediction.

2. The fault text named entity recognition device based on domain BERT according to claim 1 is characterized in that: Also includes: Training data input module: Before embedding the word vectors in the training data input module, the text is first segmented so that the input of the DBERT model is the text token after segmentation. Each token generates a corresponding embedding vector through the DBERT model. These vectors capture the semantic information of the token in its context. H=DBERT(tokens) (1) Where H is the hidden state matrix output by DBERT, and each row corresponds to the embedding vector of a token in the input sequence.

3. The fault text named entity recognition device based on domain BERT according to claim 2 is characterized in that: Single characters are used as delimiters and special tokens are introduced at the beginning and end of each sentence to meet the model input specifications.

4. The fault text named entity recognition device based on domain BERT according to claim 1 is characterized in that: In the BiLSTM model, the time series is simultaneously input into the forward LSTM and the backward LSTM, thereby obtaining the hidden state outputs of the forward and reverse sequences; The hidden state output sequences in these two directions are then merged to form a comprehensive hidden state sequence, which includes: At each time step, the BiLSTM model computes the output in the following way: ①Forward LSTM processing sequence: ②Backward LSTM processing sequence: ②Merge the hidden states in two directions: in, express, Indicates that x t express, express, Indicates that h t express; In this way, BiLSTM is able to capture contextual information.

5. The fault text named entity recognition device based on domain BERT according to claim 1 is characterized in that: The CRF model is used to calculate the probabilities of all possible label sequences and select the most likely sequence as the final prediction, which specifically includes the following sub-steps: In the CRF model, given an input sequence x, the goal is to predict the most likely output label sequence y; CRF achieves this by defining a joint probability distribution over the entire label sequence, rather than modeling each label separately; The core is to calculate the conditional probability P(y|x), which represents the probability of output sequence y given the input sequence x. The calculation method of this probability is shown in formula (5): Among them, Z(x) is the normalization factor to ensure that the sum of probabilities is 1. The calculation method is shown in formula (6): f k (y t-1 ,y t ,x,t) is a feature function, which usually depends on the input feature of the current position, the current label and the previous label; λ k is the weight of the feature function, which is learned through training data; Use CRF to calculate the probability of all possible label sequences and select the most likely sequence as the final prediction; and * =arg max y p(y|O) (7) Among them, y * is the optimal label sequence predicted by the model, O is the given observation sequence, that is, the output of the BiLSTM layer; the model searches for the one with the highest probability of appearing in the entire label sequence under the condition of a given observation sequence.

6. A method for named entity recognition of fault text based on domain BERT, characterized in that: The device according to any one of claims 1 to 5 further comprises the following steps: S1, the DBERT model achieves effective dimensionality reduction and refinement of fault text features through feature compression strategy and learns and adapts to the unique features and expertise of fault text through domain pre-training; S2, extracts context-related features of characters in the text through the DBERT model, combines these features with the specific representation of the characters after weighted operations, and uses BiLSTM to encode the features bidirectionally; S3, outputs the label sequence with the highest probability through CRF as the final prediction result.

Images