Tire building machine quality defect named entity identification method based on improved BiLSTM-CRF

Abstract

The invention belongs to the field of named entity recognition of natural language processing, and discloses a tire building machine quality defect named entity recognition method for improving BiLSTM-CRF. Comprising the following steps: step 1, preprocessing unstructured data such as on-site production quality of a forming machine, determining a preset BME label, performing manual labeling by using a labeling tool, and providing a data set for a subsequent process; and step 2, constructing a BiLSTM-CRF training learning model. And step 3, adding a learning rate scheduler to carry out timely dynamic adjustment on model parameters. And 4, testing the trained model by using a test set. Test results of the HMM model, the BiLSTM model, the BiLSTM + CRF model and the improved BiLSTM + CRF model are compared, and the results show that compared with other models, the improved BiLSTM-CRF model can consider context information and can dynamically adjust parameters, so that the model can more accurately judge entity boundaries and categories, and the overall effect is improved compared with other models.

Description

Technical Field

[0001] The invention belongs to the field of named entity recognition of natural language processing, and in particular relates to a tire building machine quality defect named entity recognition method based on improved BiLSTM-CRF for the tire industry. Background Art

[0002] Tire manufacturing is an industrial process involving many materials and complex machinery, in which tire building machines play a vital role, and their performance directly determines the quality of the final product. During the installation, commissioning and operation and maintenance process, the building machine will generate a large amount of unstructured quality defect data. By constructing a knowledge graph of building machine quality defects, these data can be effectively utilized to identify potential causes of reduced efficiency and quality problems. The knowledge graph of building machine quality defects can quickly retrieve complete production and inspection history records and quickly and accurately locate the root cause of the problem. In addition, the use of the knowledge graph of building machine quality defects helps companies build a rich quality data resource library. Through in-depth analysis of these data, companies can not only have a more comprehensive understanding of the causes of production defects, but also optimize production processes and implement predictive maintenance, thereby promoting data-based intelligent decision-making and solving the problems of unclear analysis roots and inefficient decision-making.

[0003] Named Entity Recognition (NER) is a critical step in building a knowledge graph. It is mainly used to identify the boundaries and categories of various entities in the text. NER can automatically and quickly extract defect-related information from a large number of production logs, maintenance records and inspection reports, reduce the tediousness and error rate of manual operations, and improve the efficiency and accuracy of the entire quality inspection process. NER has attracted widespread attention because it will affect the performance of other downstream natural language processing tasks, such as relationship extraction, entity linking, etc. Named entity recognition in general fields mainly targets four types of entities: names of people, places, institutions and countries. In the named entity recognition task in the tire field, entities of tire professional terms are added, which has a strong field-specific nature.

[0004] The earliest named entity recognition methods were rule-based or dictionary-based mechanical matching and filtering. With the advent of machine learning, traditional machine learning methods for named entity recognition, including hidden Markov models (HMM), maximum entropy Markov models (MEMM), and conditional random fields (CRF), have emerged. The most popular RNN model is long short-term memory (LSTM). If the information sequence is long, it will lead to the loss of long-term dependencies. The LSTM network improves this problem. Recurrent neural networks such as LSTM have shown their advantages in various NLP tasks, among which BiLSTM-CRF is a mainstream model. Many models have introduced external knowledge to the BiLSTM-CRF model, such as radical features, character features, segmentation features, sentence features, etc. Some researchers have tried to improve the structure of BiLSTM-CRF. For example, the introduction of the attention mechanism enables the model to automatically learn and pay attention to contextual information related to the named entity recognition task. In addition, there are pre-trained models such as BERT and GPT. Through pre-training on large-scale corpora, rich semantic information can be obtained to better serve the task of named entity recognition.

[0005] There are many unstructured data such as quality defects in the installation, debugging and equipment operation and maintenance of tire forming machines. Efficient use of product quality data can solve problems such as unclear root causes of product quality defect analysis and inefficient decision-making. When using the BiLSTM-CRF model for training, the learning rate is adjusted according to the preset scheduling strategy. The adjustment of the learning rate may not always be completely synchronized with the learning needs of the model in a specific period. Especially in the later stages of deep learning model training, the model may be close to the optimal solution. At this time, very fine learning rate adjustments are required for fine-tuning to avoid excessive learning rates causing the model to oscillate or deviate from the optimal path during the optimization process. Summary of the invention

[0006] Aiming at the characteristics of irregular data structure in unstructured data and the problems of untimely and inaccurate adjustment of existing named entity recognition models, the present invention proposes a method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF.

[0007] To achieve the above object, the present invention adopts the following technical solution, including the following steps:

[0008] Step 1: Preprocess the unstructured data such as the on-site production quality of the molding machine, determine the preset BME labels, and use the Label Studio data labeling tool to manually label the actual production quality data to provide a data set for subsequent training models.

[0009] Step 2: Use BiLSTM-CRF to train the learning model. The BiLSTM layer is used to integrate and extract the features of the input sequence. It expands the word vector into a vector representation in combination with the contextual semantic information. The final output of the BiLSTM layer is the probability distribution matrix of the label category of each character in the sequence. The CRF layer is used to determine the most reasonable sequence path in all feasible label sequence spaces based on the probability distribution matrix to obtain the prediction result.

[0010] Step 3: Add a learning rate scheduler to dynamically adjust the model parameters in a timely manner to obtain more accurate prediction results.

[0011] Step 4: Use the test set to test the trained model.

[0012] Furthermore, in step 2, the BiLSTM-CRF model follows the order of input layer, embedding layer, BiLSTM layer, CRF layer and output layer. The specific functions of each layer are as follows:

[0013] Step 2-1: Segment the input sentence with preset BME tags into tokens, that is, into the smallest unit in the text. Take the Chinese text tire production quality: x = "fixed screw is too short" as an example. After inputting the model, it is segmented and the output results are ω1, ω2, ω3, ω4, ω5, ω6, that is, "fixed", "fixed", "screw", "wire", "too", "short". x is used as input, and its actual physical meaning is the actual production quality text of the tire building machine;

[0014] Step 2-2: Input tokens to the embedding layer, first number them as num, then vectorize them, perform fixed vector conversion, and output the vector sequence X = (x1, x2, ..., x T ),For example:

[0015] Table 1 Fixed vector conversion results

[0016]

[0017] Step 2-3: The BiLSTM layer is used to integrate and extract the features of the input sequence, expand the word vector into a vector representation by combining the contextual semantic information, and transform the word vector into a vector representation by the forward hidden state. and the backward hidden state Together, they represent the word in the context h t The BiLSTM layer extracts the context information in the text data, and the final output is the probability distribution matrix y of the label category of each character in the sequence t .

[0018]

[0019] yt =W·h t +b (2)

[0020] Where W is the weight matrix of the fully connected layer and b is the bias term.

[0021] Step 2-4: Each label score output by BiLSTM is passed to the CRF layer to construct a transition probability matrix, including the probability value of each prediction result, which is then decoded and annotated by the CRF layer to output the optimal prediction label. Given an input sequence X = (x1, x2, ..., x T ) and the label sequence Y = (y1, y2, ..., y T ), CRF defines the conditional probability P(Y|X) as follows:

[0022]

[0023] Where φ(y t-1 ,y t ,X,t) is a feature function, which usually consists of two parts: one is the transfer feature, which describes the label y t-1 to y t The other is the emission feature, which describes the t The label y is observed under the condition t The denominator is a normalization factor that ensures that P(Y|X) is a valid probability distribution, which is calculated by summing the scores of all possible label sequences Y'.

[0024] The goal of the CRF layer is to find the label sequence Y that maximizes the conditional probability P(Y|X) * ,

[0025]

[0026] Furthermore, in step 3, the learning rate scheduler helps the model to quickly drop to a better local optimal area with a higher learning rate in the early stage by periodically reducing the learning rate, and then fine-tunes with a lower learning rate to avoid excessive oscillation when approaching the global minimum. This learning rate scheduler is suitable for tasks that show a rapid decrease in loss and then stabilize during training, because it can help the model find a more accurate minimum by reducing the learning rate when the loss decreases slowly. The initial learning rate is set to 0.1, and the step size is 10, that is, the learning rate is updated once every 10 epochs.

[0027] 1. n+1 =lr n *gamma (epoch / / step size) (5)

[0028] gamma is the predetermined update factor, lr n is the initial learning rate.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention effectively improves the unstructured data such as many quality defects in the tire forming machine during installation and debugging and equipment operation and maintenance. The efficient use of product quality data can solve the problems of unclear root causes of product quality defect analysis and inefficient decision-making. In view of the characteristics of unstructured data such as irregular data structure, the present invention adds a learning rate scheduler on the basis of BiLSTM-CRF to dynamically adjust the learning rate according to the specified interval and attenuation factor. Such a strategy can ensure that the adjustment of the learning rate is more timely and in line with the actual needs of model training, thereby optimizing the training process and final performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. The protection scope of the present invention is not limited to the following descriptions.

[0032] Figure 1 It is a work flow chart of the present invention.

[0033] Figure 2 This is the LSTM structure diagram of the present invention.

[0034] Figure 3 This is the BiLSTM structure diagram of the present invention.

[0035] Figure 4-6 It is the result diagram of each evaluation index of the present invention and other models. DETAILED DESCRIPTION

[0036] like Figure 1 As shown, the present invention proposes a method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF. Through the improved BiLSTM-CRF model, a large number of professional terms and specific contexts such as operation and maintenance data of tire building machines are automatically recognized and extracted, including data preprocessing, establishing a BiLSTM-CRF model, adding a learning rate scheduler, and model test results. The design method is as follows:

[0037] In the BiLSTM-CRF model, the input sentence with the preset BME tag is first segmented and divided into tokens. Take the Chinese text tire production quality: "fixed screw is too short" as an example. After entering the model, it is segmented and the output results are "fixed", "fixed", "screw", "wire", "too", and "short" respectively; the tokens are input to the embedding layer for fixed vector conversion, and the vector sequence is output to the BiLSTM layer; then, the BiLSTM layer is used to integrate and extract the features of the input sequence. It expands the word vector into a vector representation in combination with the contextual semantic information. The BiLSTM layer extracts the contextual information in the text data, and the final output is the probability distribution matrix of the label category of each character in the sequence; each label score output by the BiLSTM is passed to the CRF layer to construct a transition probability matrix, including the probability value of each prediction result, which is then decoded and labeled by the CRF layer to output the optimal prediction label; finally, the added learning rate scheduler adjusts the model parameters and restarts the training cycle.

[0038] The specific steps are as follows:

[0039] The first step is data preprocessing.

[0040] The data in this paper uses the production quality problem data of a company's tire forming machine. Its product quality data is mainly recorded in the form of unstructured problem descriptions. The collected data includes creation date, submission date, customer name, product name, project number, problem description, etc. The named entity recognition in this paper takes "problem description" as the main recognition object. The data totals 3517 and is randomly divided into training set, validation set and test set in a ratio of 8:1:1.

[0041] The second step is to establish a BiLSTM-CRF model.

[0042] RNN has the problem of gradient explosion and vanishing. To solve this problem, RNN variants have been added with internal gate control mechanisms to keep information for a long time, resulting in LSTM. Compared with RNN, LSTM has better performance in long time series. LSTM consists of forget gate, input gate and output gate, and its structure is as follows: Figure 2 shown.

[0043] (1) Forget gate: The output of the previous moment and the input of the current moment are input into the forget gate. After calculation, the output of the forget gate is obtained as shown below:

[0044] f t =σ(W f ·[h t-1 ,X t ]+b f ) (6)

[0045] (2) Input gate: The output of the previous moment and the input of the current moment are input to the input gate. The output of the input gate and the state of the candidate unit are obtained by the following calculation:

[0046] i t =σ(W i ·[h t-1 ,X t ]+b i ) (7)

[0047] (3) Update cell state (new cell state):

[0048]

[0049] (4) Output gate:

[0050] o t =σ(W o ·[h t-1 ,X t ]+b o ) (9)

[0051] Among them, f t The value range is (0, 1), W f represents the weight of the forgetting layer, b f Denotes the offset of the forgetting layer. h t-1 represents the hidden state at the previous time point, X t Represents the input information at the current time point. In the range of (0, 1), W i represents the weight of the input gate, b i represents the bias of the input gate, W c represents the weight of the candidate input gate, b c Represents the bias of the candidate input gate.

[0052] Although LSTM can learn which information to remember and forget through training, LSTM cannot encode forward and backward information. BiLSTM not only has a forward-pass LSTM network, but also adds a backward-pass LSTM. One layer of BiLSTM contains two LSTM layers, one for processing the sequence forward and one for processing the sequence backward. The outputs of the two LSTMs are spliced ​​together after processing. In other words, BiLSTM is a variant of LSTM that runs two LSTMs on a time series simultaneously, one for processing from front to back and one for processing from back to front. It has a wide range of applications in natural language processing, time series analysis and other fields. By processing the input sequence in two directions, BiLSTM can better capture the contextual information of each time step in the sequence, and can better capture bidirectional semantic dependencies. The BiLSTM model is a mature method for solving sequence labeling tasks. The BiLSTM structure diagram is shown below. Figure 3 .

[0053] CRF (Conditional Random Field) is a basic model for natural language processing and is widely used in annotation scenarios such as Chinese word segmentation, named entity recognition, and part-of-speech tagging. The CRF layer, as the output layer of the model, generates sequence annotation results for the text, and the output is the label of each unit in the sentence. The CRF layer can add some constraints to the final predicted label and ensure that the predicted label is valid. CRF has the following constraints:

[0054] (1) The sentence should begin with "B" instead of "M" or "E".

[0055] (2) The labeling method for rubber entities is: "B-label1 M-label2E-label3..." In this pattern, label1, label2, label3... should be the same named entity label. For example, "B-component name M-component name" is valid, but "B-component name M-problem description" is invalid.

[0056] The BiLSTM-CRF model processing steps are as follows:

[0057] (1) Segment the input sentence with preset BME tags into tokens. Take the Chinese text tire production quality: "fixed screw is too short" as an example. After inputting into the model, the word segmentation is performed and the output results are "fixed", "fixed", "screw", "thread", "too", and "short" respectively.

[0058] (2) Tokens are input into the embedding layer, first numbered as num, then vectorized, converted to fixed vectors, and output to the BiLSTM layer a vector sequence X = (x1, x2, ..., x T ):

[0059] Table 1 Fixed vector conversion results

[0060]

[0061] Where N represents the dimension and T represents the dictionary size;

[0062] (3) The BiLSTM layer integrates and extracts the features of the input sequence. It expands the word vector into a vector representation by combining the contextual semantic information. The BiLSTM layer extracts the contextual information in the text data and expands the word vector into a vector representation by combining the contextual semantic information. and the backward hidden state Together they represent the word in context The final output is the probability distribution matrix y of the label category for each character in the sequence t =W·h t +b;

[0063] (4) Each label score output by BiLSTM is passed to the CRF layer to construct a transition probability matrix, including the probability value of each prediction result, which is then decoded and annotated by the CRF layer. Given an input sequence X = (x1, x2, ..., x T ) and the label sequence Y = (y1, y2, ..., y T ), solving for the optimal predicted label The CRF loss function is:

[0064]

[0065] P RealPath is the true path score, P i Score all paths.

[0066] The third step is to add a learning rate scheduler.

[0067] Fixed-step learning rate adjustment: This method dynamically adjusts the learning rate according to the specified interval and decay factor. This strategy ensures that the learning rate is adjusted more timely and in line with the actual needs of model training, thereby optimizing the model training process and final performance.

[0068] The initial learning rate is set to 0.1, and the step size is 10, that is, the learning rate is updated once every 10 epochs.

[0069] 1. n+1 =lr n *gamma (epoch / / step size) (11)

[0070] gamma is the predetermined update factor, lr n is the initial learning rate.

[0071] The fourth step is to send the test set to the corresponding model for prediction to obtain the prediction results.

[0072] In order to test the performance of the method proposed in the present invention, the test results of four models, HMM, BiLSTM, BiLSTM+CRF and improved BiLSTM+CRF, are compared respectively, with precision, recall and F1-score as evaluation indicators, and their expressions are:

[0073]

[0074] Where:

[0075] TP (True Positives): True positive examples, that is, positive examples are predicted to be true; FP (False Positives): False positive examples, that is, negative examples are predicted to be true;

[0076] FN (false Negatives): false negative examples, that is, positive examples are predicted to be false; TN (True Negatives): true negative examples, that is, negative examples are predicted to be false.

[0077] Figure 4-6 It is the result diagram of each evaluation index of the present invention and other models. Table 2 Overall evaluation results of each model.

[0078] Table 2 Overall evaluation results of each model

[0079]

[0080] As shown in Table 2, compared with other models, the improved BiLSTM-CRF model can consider contextual information, that is, the context of the previous and following texts, so as to more accurately judge entity boundaries and categories. At the same time, the learning rate can be adjusted in real time. The obtained precision, recall rate, and F1 value are 90.39%, 89.89%, and 90.06%, respectively. The overall effect of the model is improved compared with other models. The F1 value is improved by 12.97%, 3.09%, and 2.37%, respectively, compared with other models.

[0081] The test results of each entity category of the improved BiLSTM-CRF model are as follows: Figure 4-6 As shown, entity categories are divided into component name, abnormal status, and recommended operation, where each entity category can be divided into three types: B (Begin), M (Middle), and E (End).

[0082] Combined with Table 2 and Figure 4-6For the data, HMM has the lowest evaluation index, with an F1 value of only 77.09%. This is because HMM only depends on each state and its corresponding observation object, but the sequence labeling problem is not only related to a single word, but also to the length of the observation sequence, the context of the word, and so on. Compared with the HMM model, the BiLSTM model has significantly improved various evaluation indicators, among which the F1 value has increased by 9.88%. BiLSTM is a bidirectional LSTM, and LSTM has been significantly improved compared to RNN, solving the problems of gradient disappearance and gradient explosion to a certain extent. However, LSTM cannot encode information from back to front, while BiLSTM can better capture bidirectional semantic dependencies, and CRF further provides constraints for BiLSTM. At the same time, the learning rate scheduler is added to automatically adjust the learning rate according to the performance of the validation set, which can better cope with fluctuations during training, making the various evaluation indicators of the BiLSTM-CRF model further improved.

[0083] Aiming at the characteristics of unstructured data such as tire building machine product quality defects, such as irregular or incomplete data structure, no predefined data model, and not conducive to the use of database two-dimensional logic to express, this paper proposes a method for named entity recognition of tire building machine quality defects based on improved BiLSTM-CRF. When using the BiLSTM-CRF model for training, the learning rate needs to be adjusted frequently, but as the iteration proceeds, the learning rate cannot be adjusted in time. Therefore, a learning rate scheduler is added to the BiLSTM-CRF model to dynamically adjust the learning rate to obtain a higher accuracy. This method can automatically learn features through training without manually designing features. It can better adapt to different tasks and languages, and can consider contextual information, that is, the context of the previous and next text, so as to more accurately judge the entity boundaries and categories. Through example analysis, the improved BiLSTM-CRF method has the best results, providing data support for the subsequent generation of the knowledge graph of the quality defects of the building machine.

[0084] It can be understood that the above specific description of the present invention is only used to illustrate the present invention and is not limited to the technical solutions described in the embodiments of the present invention. Those skilled in the art should understand that the present invention can still be modified or replaced by equivalents to achieve the same technical effects; as long as the use requirements are met, they are within the protection scope of the present invention.

Claims

1. A method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF is proposed. It is characterized by The following steps are involved: Step 1. Preprocess the unstructured data such as the on-site production quality of the molding machine, determine the preset BME labels, use the Label Studio data labeling tool to manually label the actual production quality data, and provide a data set for the subsequent training model; Step 2. Use BiLSTM-CRF to train the learning model, where the BiLSTM layer is used to integrate and extract the features of the input sequence. It expands the word vector into a vector representation in combination with the contextual semantic information. The final output of the BiLSTM layer is the probability distribution matrix of the label category of each character in the sequence. The CRF layer is used to determine the most reasonable sequence path in all feasible label sequence spaces according to the probability distribution matrix to obtain the prediction result; Step 3. Add a learning rate scheduler to the BiLSTM-CRF model to dynamically adjust the model parameters in a timely manner to obtain a more accurate prediction result; Step 4. Use the test set to test the trained model.

2. The method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1-1: Use the BME labeling method to create preset labels, including: component name, abnormal status, and operation suggestions. The specific forms are as follows: B-part name, M-part name, E-part name, B-abnormal state, M-abnormal state, E-abnormal state, B-operation suggestions, M-operation suggestions, E-operation suggestions; Step 1-2: Clean the database, remove useless characters, and mark them with "0" in the marking method.

3. The method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2-1: Segment the input sentence into tokens, that is, the smallest unit in the text. Take the Chinese text tire production quality: "fixing screw is too short" as an example. After entering the model, perform word segmentation; Step 2-2: Input tokens to the embedding layer, perform fixed vector conversion, and output vector sequence to the BiLSTM layer; Step 2-3: The BiLSTM layer integrates and extracts the features of the input sequence, expands the word vector into a vector representation by combining the contextual semantic information, and transforms the word vector into a vector representation by the forward hidden state. and the backward hidden state Together, they represent the word in the context h t The BiLSTM layer extracts the context information in the text data, and the final output is the probability distribution matrix y of the label category of each character in the sequence t , the formula is as follows: y t =W·h t +b (2) Where W is the weight matrix of the fully connected layer and b is the bias term; Step 2-4: Each label score output by BiLSTM is passed to the CRF layer to construct a transition probability matrix, including the probability value of each prediction result, which is then decoded and annotated by the CRF layer to output the optimal prediction label. Given an input sequence X = (x1, x2, ..., x T ) and the label sequence Y = (y1, y2, ..., y T ), CRF defines the conditional probability P(Y|X) as follows: Where φ(y t-1 ,y t ,X,t) is a feature function, which consists of two parts: one is the transfer feature, which describes the label y t-1 to y t The other is the emission feature, which describes the t The label y is observed under the condition t The probability of, the denominator is a normalization factor, ensuring that P(Y|X) is a valid probability distribution, which is calculated by summing the scores of all possible label sequences Y'; The goal of the CRF layer is to find the label sequence Y that maximizes the conditional probability P(Y|X) * : CRF has the following constraints: (1) The sentence should start with "B" instead of "M" or "E"; (2) Labeling mode for rubber entities: "B-label1 M-label2 E-label3..." In this mode, label1, label2, label3... should be the same named entity label. For example, "B-component name M-component name" is valid, but "B-component name M-problem description" is invalid.

4. The method for named entity recognition of quality defects of tire building machines based on improved BiLSTM-CRF according to claim 1, characterized in that: In step 3: The learning rate scheduler helps the model to quickly drop to a good local optimal area at the beginning by using a higher learning rate by regularly reducing the learning rate, and then fine-tunes with a lower learning rate to avoid excessive oscillation when approaching the global minimum. When the loss decreases slowly, the learning rate is reduced to help the model find a more accurate minimum. The initial learning rate is set to 0.1, and the step size is 10, that is, the learning rate is updated once every 10 epochs. The formula is as follows: lr n+1 =lr n *gamma (epoch / / stepsize) (5) gamma is the update factor, lr n is the initial learning rate.

Images