Transform-based boiler operation domain entity extraction method

Abstract

The invention discloses a method for extracting entities in a boiler operation field based on Transform. Comprising the following steps: 1, data collection: creating a data set by adopting a BIO annotation system; 2, preprocessing: cleaning and enhancing terminologies, and defining definitions and labeling standards of various entities; step 3, field adaptive BERT pre-training; step 4, time sequence feature extraction: learning context information is extracted through time sequence features; 5, performing global constraint decoding by using a CRF model: inputting a feature vector output by the network extracted from the time sequence features into a CRF layer for sequence labeling; and step 6, carrying out model training to obtain a knowledge graph of the data set. According to the method, key entities such as equipment parts, operation parameters and fault types can be automatically extracted, bottom-layer technical support is provided for boiler fault intelligent diagnosis, knowledge graph construction and operation and maintenance decision support, and the manual sorting cost is reduced.

Description

Technical Field

[0001] This invention belongs to the field of electronic data processing technology in the field of coal-fired power generation, specifically a method for entity extraction in the field of boiler operation based on Transformer. Background Technology

[0002] In the field of boiler operation, there is often a large amount of unstructured text data, such as operation logs, maintenance reports, and technical documents. These texts contain a wealth of key information related to boiler operation, such as equipment component names, operating instructions, operating parameters, and fault diagnoses. Extracting truly effective knowledge from these texts requires a significant investment of time and effort. The boiler field has a large number of specialized terms that are highly professional and domain-specific. With the increase in international projects, boiler terminology often involves a mix of Chinese and English (such as "PID regulation" and "PLC control"), or differences in terminology due to different sources of technology (such as "waterwall" corresponding to "water-cooled wall" or "water wall"). Some older documents may also retain historical usage (such as the interchangeable use of "firebed combustion" and "laminar combustion"), resulting in the same concept being expressed in diverse ways in different texts. Normalization can be achieved through terminology mapping or ontology construction. Furthermore, to simplify recording, abbreviations are frequently used in operation logs and reports, such as "DCS" (Distributed Control System), "SCR" (Selective Catalytic Reduction Denitrification Unit), and "MFT" (Main Fuel Trip). Without standardized specifications or contextual explanations, these abbreviations can easily lead to misunderstandings. Given these entity characteristics, traditional manual sorting methods are insufficient for efficient knowledge extraction, necessitating the development of a systematic solution incorporating technological means. Traditional entity extraction methods are primarily based on rules and statistical models. Rule-based methods also require the manual creation of numerous complex rules, resulting in a large workload and poor adaptability, making it difficult to handle constantly changing text data and new domain knowledge. Statistical methods, such as Hidden Markov Models (HMMs) and Conditional Random Fields (CRFs), rely on manually extracted features. The quality of feature extraction significantly impacts recognition performance and fails to fully utilize the contextual semantic information of the text. These shortcomings force boiler domain knowledge extraction to shift towards a data-driven, automated learning-based technological approach.

[0003] With the development of deep learning technology, significant progress has been made in entity extraction methods based on neural networks. Among them, the Transformer architecture (as shown in the attached image) has achieved remarkable results. Figure 4As shown in the diagram, the Transformer architecture is the most popular model structure among NLP researchers. The self-attention mechanism, as the core of the Transformer, not only helps the current node focus on the current word but also on the context, enabling the model to better acquire semantic information from the surrounding text. While the Transformer architecture demonstrates powerful contextual modeling capabilities in general NLP tasks thanks to its self-attention mechanism, it still faces many challenges in specialized fields like boiler operation. Its global self-attention-based computational model, while capable of capturing long-distance semantic dependencies, leads to a quadratic increase in model parameters and computational complexity with text length, making it difficult to efficiently handle long texts of several thousand characters commonly found in the boiler field (such as fault analysis reports and maintenance procedures). More importantly, the boiler field's unique terminology system—including abbreviation mappings (e.g., "DCS" corresponds to "distributed control system") and cross-linguistic technical terms (e.g., the semantic equivalence of "waterwall" and "water-cooled wall")—cannot be automatically learned by a purely data-driven Transformer model, resulting in a significant decrease in the accuracy of out-of-vocabulary (OOV) term recognition. Summary of the Invention

[0004] To address the issues of insufficient coverage of boiler-related terminology and low entity extraction accuracy in traditional general datasets, this invention provides a Transformer-based entity extraction method for boiler operation. It innovatively integrates domain-adaptive pre-training and dynamic feature enhancement techniques by constructing a labeling system encompassing four entity classes: COMPONENT, MEASURE, SYMPTOM, and PARAMETER. The method employs a BERT model for semantic representation of boiler-related terminology, cleans log data using dynamic vocabulary matching to enhance the labeling coverage of entities such as "water-cooled wall leakage" and "abnormal drum water level," captures the temporal features of equipment parameters using a BiLSTM network, and finally achieves global optimization decoding of the label sequence through a CRF layer. Experimental results show that this method exhibits excellent performance in token-level accuracy, F1 score, and recall, demonstrating potential for practical application in power plant DCS systems. It can improve fault diagnosis efficiency and provide structured knowledge support for intelligent boiler operation. The technical solution includes:

[0005] Step 1, Data Collection: Create the dataset using the BIO annotation system;

[0006] Step 2, Preprocessing: Cleaning and enhancement of technical terms, clarifying the definitions and labeling standards of various entities;

[0007] Step 3, Domain Adaptive BERT Pre-training: Use BertTokenizer to segment the text data of the training set, validation set, and test set; where the maximum sentence truncation length is set according to the maximum number of characters in the text, the training batch is set, and the number of sentences included in each training batch is set.

[0008] Step 4, Temporal Feature Extraction: Learn contextual information by extracting temporal features;

[0009] Step 5: Use the CRF model for global constraint decoding: Input the feature vector output by the temporal feature extraction network into the CRF layer for sequence labeling;

[0010] Step 6: Train the model to obtain the knowledge graph of the dataset;

[0011] During model training, the objective function is to minimize the negative likelihood function of the score of the correct label sequence, and the model's loss function Lr is:

[0012]

[0013] In the formula: N is the total number of training samples, yi is the true label sequence of the i-th sample, xi is the input sequence of the i-th sample, and P(yi|xi) is the conditional probability of the model predicting the true label yi corresponding to the input xi.

[0014] In step 1, the data is cleaned using Python's regular expression library. The regular expression library is imported; each collected text file is traversed and its contents are read; for different types of invalid characters, corresponding regular expression patterns are written for matching and replacement.

[0015] The dataset identifies four types of entities: equipment, fault handling, fault phenomena, and operating parameters. The dataset contains a total of nine labels: “O”, “B-device”, “I-device”, “B-fault handling”, “I-fault handling”, “B-fault phenomena”, “I-fault phenomena”, “B-operating parameters”, and “I-operating parameters”.

[0016] After performing network processing for temporal feature extraction, the feature vectors output by the forward and backward LSTMs are concatenated to obtain a feature vector sequence with a dimension of 256.

[0017] The CRF model selects the optimal label sequence by maximizing the conditional probability of the label sequence. The output of the encoding module is used as the input to the CRF layer, and the Viterbi algorithm is used to decode the sequence that outputs the highest score as the predicted label sequence. The calculation formula is as follows:

[0018]

[0019] Where S is the total score of the labeled sequence, A is the transition matrix, and n is the length of the input sentence. yi,yi+1 Let P represent the score for transforming label i into label i+1, where P is the performance matrix obtained by passing the output of the BiLSTM-Attention layer through a multilayer perceptron, p(y|x) is the conditional probability that the labeled sequence is y given the input sequence x, and s(x,y) is the matching score between the input sequence x and the labeled sequence y. i,yi This represents the score given by the i-th word in the sentence to the i-th label.

[0020] The CRF model consists of two parts: feature functions and transition features. The feature functions are functions of the input sequence and the output label, used to capture the local relationships between the input sequence and the output label. The transition features capture the global transition relationships between adjacent labels.

[0021] The beneficial effects of this invention are as follows:

[0022] 1. This paper presents an entity extraction method that integrates BERT pre-trained model, bidirectional long short-term memory neural network (BiLSTM) and conditional random field (CRF). This method can automatically extract key entities such as equipment components, operating parameters, and fault types, providing underlying technical support for intelligent boiler fault diagnosis, knowledge graph construction and operation and maintenance decision support, and reducing the cost of manual sorting.

[0023] 2. This invention organically combines BERT, BiLSTM, and CRF, fully leveraging BERT's powerful semantic understanding capabilities, BiLSTM's ability to capture contextual information, and CRF's ability to optimize label sequences, thereby improving the accuracy and performance of entity extraction in the boiler operation domain. Through the collection, annotation, and training of a large amount of textual data from the boiler operation domain, the model can better adapt to the characteristics of the domain, accurately identify various types of named entities, and provide reliable data support for intelligent management and decision-making in boiler operation. The method and system of this invention have good versatility and scalability, can be easily applied to different types of boiler operation scenarios, and can be continuously improved by further increasing training data and optimizing model parameters. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating an embodiment of a boiler operation entity extraction method based on Transformer according to the present invention.

[0025] Figure 2 This is a comparison chart of model accuracy after training in an embodiment of the present invention.

[0026] Figure 3 This is a comparison chart of model recall rates after training in an embodiment of the present invention.

[0027] Figure 4 This is a comparison chart of the F1 scores of the model after training in an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings.

[0029] like Figure 1 The embodiment of the present invention shown includes:

[0030] Step 1, Data Collection: Create the dataset using the BIO annotation system;

[0031] Step 2, Preprocessing: Cleaning and enhancement of technical terms, clarifying the definitions and labeling standards of various entities;

[0032] Step 3, Domain Adaptive BERT Pre-training: Use BertTokenizer to segment the text data of the training set, validation set, and test set; where the maximum sentence truncation length is set according to the maximum number of characters in the text, the training batch is set, and the number of sentences included in each training batch is set.

[0033] Step 4, Temporal Feature Extraction (BiLSTM): Learn contextual information through BiLSTM;

[0034] Step 5: Use the CRF model for global constraint decoding: Input the feature vector output by the BiLSTM network into the CRF layer for sequence labeling.

[0035] Step 6: Train the model to obtain the knowledge graph of the dataset; and compare the model recognition based on accuracy, recall, and F1 score.

[0036] Specifically, it is divided into:

[0037] (I) Data Collection and Preprocessing

[0038] To obtain comprehensive and representative boiler operation data, approximately 200 text files (complete paragraphs of Chinese and / or English sentences from the operation manual) were collected from a large power plant over the past year. These files came from various sources and contained a large amount of invalid information, such as garbled characters, HTML tags, and advertising information unrelated to boiler operation, which would interfere with subsequent data analysis and model training. Therefore, Python's regular expression library was used to clean the data. The `re` library was imported to prepare for subsequent text matching and replacement operations. Each collected text file was iterated through, and its content was read. Corresponding regular expression patterns were written for matching and replacement of different types of invalid characters. `re.sub(r'<.?>',",text)` was used to remove HTML tags; `re.sub(r'[^\x00\x7F]+',",text)` was used to remove garbled characters.

[0039] The BIO annotation system is adopted, which can clearly identify the boundaries and types of each entity in the text. The definitions and annotation standards for each type of entity are clearly defined. This dataset identifies four types of entities: COMPONENT, MEASURE, SYMPTOM, and PARAMETER. The dataset labels contain a total of nine types: "O", "B-Equipment", "I-Equipment", "B-Fault Handling", "I-Fault Handling", "B-Fault Phenomenon", "I-Fault Phenomenon", "B-Operating Parameter", and "I-Operating Parameter". The boiler operation dataset is specifically annotated in BIO format as shown in the table below.

[0040] Table 1 BIO Labeling Table

[0041]

[0042] The labeled data is divided into training, validation, and test sets in an 8:1:1 ratio. The `train_test_split` function from the `sklearn.model_selection` library is then used for data splitting.

[0043] (II) Model Training

[0044] BertTokenizer was used to segment the text data in the training, validation, and test sets. During segmentation, the text was converted into a format suitable for the BERT model input, including adding special tokens (such as [CLS] and [SEP]) and chunking. These vectors contain the semantic information of each character in the text within its context, providing rich feature representations for subsequent model learning.

[0045] In this boiler fault entity recognition task, meticulous parameter settings were implemented at each stage. Before setting hyperparameters, sentence length statistics were performed on all labeled texts. The statistics showed that the maximum text length was 79 characters, the minimum was 32 characters, and the average was 52.82 characters. No texts exceeded 256 characters. Therefore, during the text input stage, the maximum sentence truncation length was set to 256 characters. Simultaneously, to balance training stability and efficiency, each training batch contained 16 sentences. For the word vector representation stage, the pre-trained model bert-base-chinese was selected. It performs excellently in Chinese text processing, effectively capturing the semantic information of Chinese boiler fault data vocabulary. Its default vector dimension is 768.

[0046] In the semantic encoding stage, the default 12-layer Transformer encoder of BERT is used. This encoder has powerful feature extraction and semantic understanding capabilities, and can deeply mine the semantic relationships in boiler fault data. In the model training stage, the loss rate is set to 0.1 to control the difference between the model's prediction and the true label; the learning rate is set to 4e-5 to ensure that the model can converge stably during training; and the number of training epochs is set to 50 to give the model sufficient opportunities to learn the feature patterns in the boiler text dataset.

[0047] By learning contextual information using BiLSTM, a BiLSTM network is constructed to learn the contextual information of the text. The LSTM has three gates to maintain and adjust the cell state: the forget gate, the input gate, and the output gate. The forget gate receives h... t-1 and x t The formula (1) outputs a value f between 0 and 1. t This value will apply to the previous Cell state C. t-1 , where f t A value of 1 indicates "complete preservation", f t A value of 0 indicates "complete forgetting"; the input gate receives h. t-1 and x t Formula (2) outputs a value between 0 and 1 to control the current candidate state i. t What percentage of information needs to be retained or lost; candidate states A new candidate value vector is created from the tanh layer using formula (3); then, based on the previous cell state C... t-1 and forgetting value f t Candidate status and input value i t The Cell state is updated by formula (4); the output gate receives h. t-1 and x tFormula (5) outputs a value O between 0 and 1. t Finally, formula (6) determines the current state C. t How much information needs to be output?

[0048] f t =δ(w f [h t-1 ,x t ]+b f (1)

[0049] i t =δ(w i [h t-1 ,x t ]+b i (2)

[0050]

[0051] O t =δ(W o [h t-1 ,x t ]+b o (5)

[0052] H t =O t *tanh(C t (6)

[0053] Among them, b f For the bias term of the forget gate, b i δ is the bias term of the input gate, δ is the sigmoid activation function, and w f It is the weight matrix of the forget gate, w i It is the weight matrix of the input gate; w c It is to calculate the candidate cell state C t At that time, for the input [h t-1 ,x t The weight matrix for linear transformation; b c It is to calculate the candidate cell state C t The bias term; w0 is the calculation of the output gate O. t At that time, for the input [h t-1 ,x t The weight matrix for the linear transformation; b0 is the output gate O. t The bias term.

[0054] After processing by the BiLSTM network, the feature vectors output by the forward and backward LSTMs are concatenated to obtain a feature vector sequence with a dimension of 256.

[0055] At this point, the CRF layer sequence labeling inputs the feature vector output by the BiLSTM network into the CRF layer for sequence labeling. The CRF model consists of two parts: feature functions and transition features. The feature functions are functions of the input sequence and output label, used to capture the local relationship between the input sequence and output label; the transition features capture the global transition relationship between adjacent labels. The CRF model uses a linear combination of these feature functions as the logarithmic form of the conditional probability, and models the relationship between the input sequence and output label by learning the weights of the feature functions. The CRF model training uses the maximum likelihood estimation method, learning the weights of the feature functions by maximizing the log-likelihood function of the training data. In the prediction phase, the CRF model uses a dynamic programming algorithm to find the optimal solution for the output sequence, i.e., the label sequence with the maximum conditional probability. The model in this paper uses the CRF model to select the best label sequence by maximizing the conditional probability of the label sequence. The output of the encoding module is used as the input of the CRF layer, and the Viterbi algorithm is used to decode the sequence with the highest output score as the predicted label sequence. The calculation formula is as follows:

[0056]

[0057] Where S is the total score of the labeled sequence, A is the transition matrix, and n is the length of the input sentence. yi,yi+1 Let P represent the score for transforming label i into label i+1, where P is the performance matrix obtained by passing the output of the BiLSTM-Attention layer through a multilayer perceptron, p(y|x) is the conditional probability that the labeled sequence is y given the input sequence x, and s(x,y) is the matching score between the input sequence x and the labeled sequence y. i,yi This represents the score of the i-th word in the sentence based on the i-th label;

[0058] During model training, the objective function is to minimize the negative likelihood function of the score of the correct label sequence. The specific loss function calculation is shown in formula (10).

[0059]

[0060] In the formula: Lr is the loss function of the model, N is the total number of training samples, yi is the true label sequence of the i-th sample, xi is the input sequence of the i-th sample, and P(yi|xi) is the conditional probability of the model predicting the true label yi corresponding to the input xi.

[0061] Table 2 BERT Model Hyperparameter Configuration Table

[0062]

[0063] In the field of NLP, the main evaluation metrics are: accuracy, recall, and F1 score. Accuracy (p) represents the proportion of data that is actually classified as positive out of the data that is identified as positive. In named entity recognition, it means the number of correctly identified entities out of the total number of identified entities.

[0064]

[0065] Recall R refers to the proportion of positive entities identified out of the actual positive data. In named entity recognition, it means the number of correct entities out of the total number of entities in the training set.

[0066]

[0067] The F1 score refers to the proportion of positive entities identified in data that are actually positive. In named entity recognition, it means the number of correct entities out of the total number of entities in the training set. The specific formula is shown below.

[0068]

[0069] In the confusion matrix metrics for binary classification tasks, TP represents the number of samples that are actually positive but are identified as positive; FN represents the number of samples that are actually positive but are identified as negative; FP represents the number of samples that are actually negative but are identified as positive; and TN represents the number of samples that are actually negative but are identified as negative.

[0070] Training was stopped after 50 consecutive epochs, and the current model parameters were saved to prevent overfitting. The recognition results of the BERT+BILSTM+CRF model, the BERT model, and the BERT+BILSTM model on the test set are shown in Tables 3 to 5 below. The comparisons of model accuracy, model recall, and model F1 are as follows: Figures 2-4 As shown.

[0071] Table 3 Experimental Results of the BERT Model Test Set

[0072]

[0073] Table 4. Experimental Results of the BERT+BILSTM Model Test Set

[0074]

[0075] Table 5. Experimental Results of the BERT+BILSTM+CRF Model Test Set

[0076]

[0077] The above analysis shows that in the identification of four types of entities—components, mechanisms, symptoms, and parameters—the model trained using BERT+BILSTM+CRF achieves higher accuracy, recall, and F1 score than the models trained using BERT and BERT+BILSTM. In the accuracy test, the model trained using BERT+BILSTM+CRF achieves an accuracy higher than 78%, indicating that the model has a good ability to identify entity types in the validation set. The F1 score is higher than 0.75, verifying that the model can effectively identify entities with specific labels. Finally, the trained BERT+BILSTM+CRF named entity extraction model was used to extract data from the experimental text "The boiler's reheat steam temperature setpoint is 535 degrees Celsius, the current actual temperature is 530 degrees Celsius, and the deviation is within the normal range. The operator adjusted the desuperheater's water spray volume to fine-tune the temperature." The extraction results are: {"start_pos":0,"end_pos":1,"entity_name":"boiler","entity_type":"equipment"},{"start_pos":3,"end_pos":8,"entity_name":"reheat steam temperature","entity_type":"operating parameters"},{"start_pos":49,"end_pos":51,"entity_name":"desuperheater","entity_type":"equipment"},{"start_pos":53,"end_pos":55,"entity_name":"water spray volume","entity_type":"operating parameters"}. In this experimental text extraction, the BERT+BILSTM+CRF named entity extraction model demonstrated high reliability in identifying and locating equipment entities such as "boiler" and "desuperheater," as well as operational parameter entities such as "reheat steam temperature" and "spray water volume." It not only accurately extracted target entities but also correctly labeled the corresponding entity types, providing a solid foundation for subsequent data analysis and decision-making based on entity information.

[0078] From a practical application perspective, this model has broad application prospects in fields such as industrial equipment operation monitoring and fault diagnosis. By continuously optimizing the model and improving its ability to identify hidden entities, while increasing the training data coverage for entity types such as fault handling and fault phenomena, it is expected that the model can achieve more comprehensive and accurate entity extraction in complex text environments, build a complete industrial text information processing system, and help upgrade the intelligent production and improve management efficiency in industrial production.

Claims

1. A method for entity extraction in boiler operation domain based on Transformer, characterized in that, include: Step 1, Data Collection: Create the dataset using the BIO annotation system; Step 2, Preprocessing: Cleaning and enhancement of technical terms, clarifying the definitions and labeling standards of various entities; Step 3, Domain Adaptive BERT Pre-training: Use BertTokenizer to segment the text data of the training set, validation set, and test set; where the maximum sentence truncation length is set according to the maximum number of characters in the text, the training batch is set, and the number of sentences included in each training batch is set. Step 4, Temporal Feature Extraction: Learn contextual information by extracting temporal features; Step 5: Use the CRF model for global constraint decoding: Input the feature vector output by the temporal feature extraction network into the CRF layer for sequence labeling; Step 6: Train the model to obtain the knowledge graph of the dataset; During model training, the objective function is to minimize the negative likelihood function of the score of the correct label sequence, and the model's loss function Lr is: In the formula: N is the total number of training samples, yi is the true label sequence of the i-th sample, xi is the input sequence of the i-th sample, and P(yi|xi) is the conditional probability of the model predicting the true label yi corresponding to the input xi.

2. The boiler operation domain entity extraction method based on Transformer according to claim 1, characterized in that, In step 1, the data is cleaned using Python's regular expression library. The regular expression library is imported; each collected text file is traversed and its contents are read; for different types of invalid characters, corresponding regular expression patterns are written for matching and replacement.

3. The method for entity extraction in boiler operation domain based on Transformer according to claim 2, characterized in that, The dataset identifies four types of entities: device, fault handling, fault phenomenon, and operating parameters. The dataset contains a total of nine labels: "O", "B-device", "I-device", "B-fault handling", "I-fault handling", "B-fault phenomenon", "I-fault phenomenon", "B-operating parameters", and "I-operating parameters".

4. The boiler operation domain entity extraction method based on Transformer according to claim 1, characterized in that, After performing network processing for temporal feature extraction, the feature vectors output by the forward and backward LSTMs are concatenated to obtain a feature vector sequence with a dimension of 256.

5. The boiler operation domain entity extraction method based on Transformer according to claim 1, characterized in that, The CRF model selects the optimal label sequence by maximizing the conditional probability of the label sequence. The output of the encoding module is used as the input to the CRF layer, and the Viterbi algorithm is used to decode the sequence that outputs the highest score as the predicted label sequence. The calculation formula is as follows: Where S is the total score of the labeled sequence, A is the transition matrix, and n is the length of the input sentence. yi,yi+1 Let P represent the score for transforming label i into label i+1, where P is the performance matrix obtained by passing the output of the BiLSTM-Attention layer through a multilayer perceptron, p(y|x) is the conditional probability that the labeled sequence is y given the input sequence x, and s(x,y) is the matching score between the input sequence x and the labeled sequence y. i,yi This represents the score given by the i-th word in the sentence to the i-th label.

6. The boiler operation domain entity extraction method based on Transformer according to claim 5, characterized in that, The CRF model consists of two parts: feature functions and transition features. The feature functions are functions of the input sequence and the output label, used to capture the local relationships between the input sequence and the output label. The transition features capture the global transition relationships between adjacent labels.

Images