A sequence labeling method based on deep learning and its application

Through the deep learning sequence labeling method, combined with text preprocessing, translation and error correction processing, the joint labeling of multiple deep learning models is solved, and efficient and accurate knowledge graph labeling is achieved. .

CN114692568BActive Publication Date: 2024-12-06NAT UNIV OF DEFENSE TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210310976.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-28
Publication Date
2024-12-06
Estimated Expiration
2042-03-28

AI Technical Summary

Technical Problem

The existing technology is inefficient in the sequence labeling process in the knowledge graph, unable to adapt to multilingual labeling, labeling tools cannot automatically label, and the joint labeling of multiple models is missing, which affects the update speed and accuracy of the model.

Method used

Using deep learning-based sequence labeling methods, including text preprocessing, translation processing, error correction processing and joint annotation of multiple deep learning models, pre-annotation by constructing rule dictionaries and regular expressions, combining attention mechanisms and multi-head attention Mechanism, calculate the weight values ​​output by each model to obtain the optimal annotation content.

Benefits of technology

It significantly improves the efficiency and accuracy of sequence labeling, saves manual labeling time, improves the automation capabilities of labeling tools, and enhances the adaptability to multilingual and multi-models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114692568B_ABST
    Figure CN114692568B_ABST
Patent Text Reader

Abstract

The present invention discloses a sequence annotation method and application based on deep learning. The method comprises the steps of: obtaining a text to be annotated and preprocessing the text; performing text translation processing and error correction processing on the preprocessed text; constructing a rule dictionary and a regular expression, pre-annotating the text after error correction processing based on the rule dictionary and the regular expression, and outputting the pre-annotated content; inputting the pre-annotated text into multiple sequence annotation models based on deep learning, and calculating the optimal annotation content according to the weight values ​​of the output results of the multiple sequence annotation models. The present invention can improve the efficiency and accuracy of sequence annotation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of NLP natural language processing and relates to a sequence labeling method and application based on a deep learning model. Background Art

[0002] In the application process of knowledge graph, sequence labeling is an indispensable link. Sequence labeling models are widely used in text processing related fields, such as word segmentation, part-of-speech tagging, named entity recognition, etc. In the past, data labeling required a lot of manpower and material resources for labeling, which was inefficient. Ordinary labeling software could not adapt to multi-language labeling, and the entity pronoun model of the corpus could not be well identified and extracted. Ordinary labeling tools could not be included in the vocabulary in a targeted manner, and the labeled corpus could not be automatically labeled. Although the previous labeling software had multiple models, it did not use multiple models for joint labeling, or simply used multiple models to predict the results separately and took the majority to determine the final labeling result. The labeling speed and accuracy greatly affected the update of the model. Therefore, improving the efficiency and accuracy of the labeling tool is very important for the construction of knowledge graphs. Summary of the invention

[0003] The present invention provides a deep learning-based sequence labeling method and application, which can improve the efficiency and accuracy of sequence labeling.

[0004] To achieve the above object, according to a first aspect of the present invention, a sequence labeling method based on deep learning comprises the steps of:

[0005] Get the text to be annotated and preprocess the text;

[0006] Perform text translation and error correction on the preprocessed text;

[0007] Construct a rule dictionary and regular expressions, pre-annotate the text after error correction based on the rule dictionary and regular expressions, and output the pre-annotated content;

[0008] The pre-annotated text is input into multiple sequence annotation models based on deep learning, and the optimal annotation content is calculated according to the weight values ​​of the output results of multiple sequence annotation models.

[0009] Furthermore, the translation process includes:

[0010] A seq2seq model with an attention mechanism is used for translation. The seq2seq model includes an encoder and a decoder. The attention mechanism causes the decoder to pay different attention to different parts during decoding. The decoder includes a labeling sequence prediction module and a constraint module. The labeling sequence prediction module is used to output multiple predicted labeling sequences. The constraint module is used to define a feature function set, which includes several feature functions. Each feature function is used to determine whether a labeling sequence meets the feature corresponding to the feature function. The scores of all feature functions in the set for the same predicted labeling sequence are obtained respectively, and the scores of all feature functions in the set for the same predicted labeling sequence are summed as the final score of the predicted labeling sequence. In this way, the final score of each predicted labeling sequence output by the labeling sequence prediction module is obtained, and the predicted labeling sequence with the highest final score is selected as the translation processing result.

[0011] Furthermore, the error correction process includes:

[0012] Collect professional term libraries in different fields, preset the correct translation text of each professional term in the professional term library, and if a professional term in the professional term library exists in the translated text, and the translation of the professional term is different from the preset correct translation text, directly replace the translation of the professional term with the preset correct translation text;

[0013] Then, the text sequence processed in the previous step is sent to an error correction model based on deep learning, which includes a word vector mapping module, an error detection module, an encoding module, a multi-head attention mechanism module, a fusion module and a classification module. The word vector mapping module is used to convert the input text sequence into a text vector and output it to the error detection module. It is also used to identify entities in the text sequence according to a pre-built knowledge graph and convert the identified entities into entity vectors and output them to the multi-head attention mechanism module. The error detection module is used to obtain a text semantic representation tensor according to the text vector. The encoding module is used to encode the text semantic representation tensor and output it to the multi-head attention mechanism module. The multi-head attention mechanism is used to output a text embedding representation and an entity embedding representation to the fusion module for feature fusion. The classification module is used to output an error correction result according to the fused features.

[0014] Further, the multiple deep learning-based sequence labeling models include a first model, a second model, and a third model;

[0015] The first model includes an ALBERT model, a BiGRU model and a CRF model, wherein the ALBERT model is used to obtain word vectors and extract important features of text, the BiGRU model is used to perform named entity recognition through deep learning of context feature information, and the CRF model is used to process the output sequence of the BiGRU model, and obtain a global optimal sequence according to adjacent labels in combination with the state transfer matrix in the CRF model;

[0016] The second model includes a BERT model, a BiLSTM model and a CRF model, wherein the BERT model is used to obtain word vectors and extract important features of text, the BiLSTM model is used to perform named entity recognition through deep learning context feature information, and the CRF model is used to process the output sequence of the BiLSTM model, and obtain a global optimal sequence according to adjacent labels in combination with the state transfer matrix in the CRF model;

[0017] The third model includes the ERNIE model. In the pre-training of the ERNIE model, basic text masks are used in the first stage, phrase-level masks are used in the second stage to allow the model to predict phrases, and named entity masks are used in the third stage to allow the model to predict named entities.

[0018] Furthermore, the BiGRU model and the BiLSTM model both add a pooling layer after the basic RNN model.

[0019] Furthermore, obtaining the optimal annotation content according to the output results of the multiple sequence annotation models includes:

[0020] Determine the weight coefficient of each sequence labeling model;

[0021] If the output labels of all sequence labeling models are the same, then the output label is used as the optimal label content;

[0022] If the output labels of different sequence labeling models are different, the output labels of all sequence labeling models with the same output labels are multiplied by their corresponding weight coefficients and then summed to obtain the summed probability value of each type of output label, and the output label with the largest summed probability value is selected as the optimal labeling content.

[0023] Furthermore, the weight coefficient of each sequence labeling model is determined through training.

[0024] According to a second aspect of the present invention, a sequence labeling system based on deep learning includes:

[0025] A preprocessing module is used to obtain the text to be annotated and preprocess the text;

[0026] A translation error correction module is used to perform text translation error correction on the preprocessed text;

[0027] A rule matching module is used to construct a rule dictionary and regular expressions, pre-annotate the translated text after error correction based on the rule dictionary and regular expressions, and output the pre-annotated content;

[0028] The joint model annotation module is used to input the pre-annotated text into multiple sequence annotation models based on deep learning, and obtain the optimal annotation content according to the weight values ​​of the output results of multiple sequence annotation models.

[0029] According to the third aspect of the present invention, the present invention provides multi-model joint labeling, and each model will obtain its own prediction result for the predicted text, and the optimal label of the text type is determined according to the weight value of each model.

[0030] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0031] In general, compared with the prior art, the present invention has beneficial effects: the sequence annotation of the present invention combines text translation, text error correction, rule matching and joint annotation of multiple deep learning models to achieve the purpose of rapid text annotation. The existing entities can be quickly marked using rules, which not only saves a lot of time, but also improves the accuracy of annotation much more than manual annotation. Only manual review of the annotation content is required, which greatly shortens the time of manual annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flowchart of a sequence labeling method based on deep learning according to an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of the principle of a sequence labeling method based on deep learning according to an embodiment of the present invention;

[0034] Figure 3 is a network structure diagram of a sequence labeling model based on deep learning according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of a pooling layer of a sequence labeling model according to an embodiment of the present invention;

[0036] Figure 5 Schematic diagram of a deep learning-based sequence labeling system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] like Figure 1 and Figure 2 As shown, the sequence labeling method based on deep learning in an embodiment of the present invention includes the following steps:

[0039] Step 1: Get the text to be annotated and preprocess the text.

[0040] Preferably, the training data used is Wikipedia data; data source: crawled from Wikipedia.

[0041] The acquired structured data and unstructured data are classified. This optimization method is mainly aimed at unstructured and semi-structured data. It can be used directly for structured data.

[0042] Clean the data and perform data preprocessing. The preprocessing can be implemented by any method in the prior art.

[0043] Step 2: Perform text translation on the preprocessed text.

[0044] In order to adapt to different countries and different languages, the seq2seq model with the attention mechanism is used for translation, and improvements are made on this mechanism to achieve text translation. The basic seq2seq model is a neural network with an encoder-decoder structure. Its input is a sequence and its output is also a sequence. The attention mechanism makes the decoder pay different attention to different parts when decoding. The decoder includes a label sequence prediction module and a constraint module. The label sequence prediction module is used to output multiple predicted label sequences. The constraint module is used to define a set of feature functions, and the scores of all feature functions in the set for the same predicted label sequence are obtained respectively. The scores of all feature functions in the set for the same predicted label sequence are summed as the final score of the predicted label sequence. In this way, the final score of each predicted label sequence output by the label sequence prediction module is obtained, and the predicted label sequence with the highest final score is selected as the translation processing result. Among them, the feature function set includes several feature functions, each feature function is used to determine whether the annotation sequence meets the feature corresponding to the feature function. For example, the output value of the feature function is 0 or 1, 0 means that the annotation sequence to be scored does not meet this feature, and 1 means that the annotation sequence to be scored meets this feature.

[0045] The improvement of the present invention lies in the optimization of the output sequence of the seq2seq model. The output of the existing seq2seq model uses greedy decoding. The so-called greedy decoding is to select the maximum probability value of the predicted output of the model each time, so that the rationality of the text sequence after the entire text is output is not considered, and only the result of the current text output is considered. Therefore, the present invention optimizes this when decoding, selects k probability values ​​in turn each time, and then uses the k outputs obtained this time as the input of the next moment to participate in the prediction of the next moment, and finally obtains multiple groups of sequence outputs, and finally selects a group of the largest overall probability value from the multiple groups of sequence outputs as the final output of the model, but the final output does not take into account the rationality of the output sequence, that is, the grammaticality of the sentence. For example, in Chinese, the text translation result should be "I go to eat today", but the maximum probability value predicted by the model is "I go to eat today", and "go" as a preposition itself should not be placed at the beginning of the sentence as a subject, and the phenomenon of grammatical errors is obvious in this sentence. After the use of conditional restrictions, according to the results of part-of-speech tagging, the final output sequence will slowly learn that such non-standard output is unreasonable, thereby reducing such sequence output, and better completing the process of mutual translation between countries. This model has been trained in the languages ​​of dozens of countries, and can complete the task of mutual translation between more than 30 languages, including Chinese-English, Chinese-German, German-English, Chinese-Japanese, Japanese-English, Chinese-French, Chinese-Russian, Japanese-French, German-French, Japanese-Russian, Chinese-Arabic, English-Spanish, Chinese-Spanish, etc., thereby greatly reducing the limitations of the use of this annotation device, and people from different countries can use it.

[0046] The translation process is implemented as follows:

[0047] Seq2seq is divided into an encoder and a decoder. In the encoder, the text is first segmented into words, and then the words are converted into word vectors, represented by Vi, and input in chronological order with the hidden state h(i-1) of the previous moment. A hidden state hi is output at each moment. The transformation of the RNN hidden layer can be represented by the function f, hi=f(vi,h(i-1)), which can be understood as the result of inputting h(i-1) of the previous moment into the RNN of the current moment to obtain the current hidden state. Assuming there are t words, the hidden state of each moment is finally retained by the encoder. After the encoder finishes encoding the input sequence, the initial state of the decoder is S0, and the last state of the encoder is h. The attention mechanism is to calculate the weight correlation between the current state St of the decoder and all the states h1, h2, and h3 of the encoder, so that the c vector encoded by the encoder is weighted with each output of the decoder during the decoding process, and the weight a ij There are many ways to calculate, given i, a ij A probability distribution at time j = 1, 2, 3…T, a ij =softmax(e(Si-1 ,h i )), S i is the hidden state of the decoder at time i-1, assuming that each hidden state of the encoder is h j , sequence length T, the calculation method of c vector at the i-th moment is:

[0048]

[0049] Vi (represents word vector, which can be simply understood as a sequence vector that converts a word into model input, such as: apple [0.20, 0.30, 0.1, -0.21, -0.23, -0.43, -0.34, 0.234, 0.456, ...]), hi (represents the output representation of the word vector at the i-th moment calculated by the encoder. There will be a word input model at each moment), RNN is the structure in seq2seq, W (parameter matrix, which is initialized by itself and will be continuously updated through backpropagation through the model), and e is the distance calculation function.

[0050] The traditional decoding method uses beam search, which selects the top K words at each moment as the output of this moment, and uses them as the input of the next moment one by one to participate in the prediction of the next moment, and then selects the top K values ​​with the largest probability value from these K*L (L is the size of the word list) results as the output of the next moment, and so on. At the last moment, the top 1 is selected as the final output, which cannot effectively reduce the output of the wrong sequence.

[0051] Therefore, in the embodiment of the present invention, a sequence X=(X 1 ,X 2 ,X 3 ...), and output the target sequence Y = (Y 1 ,Y 2 ,..Y n ), in this process a characteristic function f(X,i,y i ,y i-1 ), where X represents the input sequence, such as “Bob drank coffee at Starbucks”, i represents the current node, and y represents the i The predicted value of the current node, y i-1It represents the predicted value of the previous node. This function is related to the context of the node, that is, it is related to both node i and node i-1. The existing LTP tool is used to annotate the part of speech. After annotating the part of speech of each word, it looks like this: "Bob (noun) drink (verb) coffee (noun) at (preposition) Starbucks (noun). There are many optional annotation sequences. (noun, verb, noun, preposition, noun) is called l as a annotation sequence. For example, l can also be like this: (noun, verb, verb, preposition, noun). In so many optional annotation sequences, the most reliable one should be selected as the label for this sentence. Note. How to judge whether a labeling sequence is reliable or not? If we give a score to each labeling sequence, the higher the score, the more reliable the labeling sequence is. At least, we can say that any labeling sequence with a verb followed by a verb should be given a negative score. The verb followed by a verb mentioned above is a feature function. We can define a feature function set, use this feature function set to score a labeling sequence, and select the most reliable labeling sequence based on this. In other words, each feature function can be used to score a labeling sequence, and the scores of all feature functions in the set for the same labeling sequence are combined to get the final score of the labeling sequence.

[0052] Assume that X = (X1, X2, ..., Xn), Y = (Y1, Y2, ..., Yn) are all random variable sequences represented by linear chains. If, under the condition of a given random variable sequence X, the conditional probability distribution P(Y|X) of the random variable sequence Y constitutes a conditional random field, P(Y i |X,Y1,...,Y i-1 ,Y i+1 ,....,Yn)=P(Y i |X,Y i-1 ,Y i+1 )i=1,2,...,n (only one side is considered when i=1 and n).

[0053]

[0054] in,

[0055] In the labeling problem, X represents the input observation sequence, Y represents the corresponding output tag sequence or state sequence, and t k and l is the characteristic function, λ k and μ l is the corresponding weight. Z(x) is the normalization factor, and the sum is performed over all possible output sequences.

[0056] t kis a characteristic function defined on the edge, called the transfer feature, which depends on the current and previous positions, t(y i-1 ,y i ,x,i) expresses "given observation x, from the previous node y i-1 Transfer to this node y i situation".

[0057] s l It is a characteristic function defined on the node, called the state characteristic, which depends on the current position and expresses "the current node y i Is it the case of marking x”.

[0058] t k and l They all depend on the position and are local characteristic functions. Usually, the characteristic function t k and l The value is 1 or 0; when the characteristic condition is met, the value is 1, otherwise it is 0. The conditional random field is completely determined by the characteristic function t k ,s l and the corresponding weight λ k , μ l OK. Sentence s (the sentence to be marked with part of speech); i, used to represent the i-th word in sentence s; l i , represents the word annotated for the i-th word in the tag sequence to be scored; l i-1 , which represents the part of speech of the i-1th word in the tag sequence to be scored.

[0059] For example: Assume there is a characteristic function f in the characteristic function set 1 and f 2 .f 1 and f 2 The definition of is as follows. i is an "adverb" and the i-th word ends with "ly", let f 1 =1, other cases f 1 is 0. The weight λ of the f1 feature function 1 should be positive. And λ 1 The larger the value, the more likely it is to adopt the tagging sequences that mark words ending with "ly" as "adverbs".

[0060] f 2 (s,i,l i , l i-1 )=1

[0061] If i=1, l i = verb, and the sentence s ends with "?", f 2 =1, other cases f 2 = 0. Similarly, λ2 should be positive, and λ 2 The larger the value, the more likely it is to use the labeling sequence that labels the first word of the question as "verb". After this constraint, some sequences with obvious errors in the model translation results will be deleted and will not be output. Without this condition, according to the size of the probability value predicted by the model, it is very likely to output an incorrect output sequence. In this way, compared with the original model, the effect of sequence output will be significantly improved.

[0062] Step 3: Correct the translated text.

[0063] Because different countries' languages ​​have different characteristics, especially proper nouns, the effect of model translation is much worse. Therefore, to address this phenomenon, this annotation device has added a text correction function. The text correction function can be used to correct errors based on the above translation results.

[0064] Further, the error correction process comprises the steps of:

[0065] First, professional terminology libraries in different fields are collected, and the correct translation text of each professional term in the professional terminology library is preset. If a professional term in the professional terminology library exists in the translated text, if the translation of the professional term is different from the preset correct translation text, the translation of the professional term is directly replaced by the preset correct translation text.

[0066] Then, the text sequence processed in the previous step is sent to the error correction model based on deep learning. The error correction model includes a word vector mapping module, an error detection module, an encoding module, a multi-head attention mechanism module, a fusion module and a classification module. The word vector mapping module is used to convert the input text sequence into a text vector and output it to the error detection module. It is also used to identify entities in the text sequence according to a pre-built knowledge graph and convert the identified entities into entity vectors and output them to the multi-head attention mechanism module. The error detection module is used to obtain a text semantic representation tensor based on the text vector. The encoding module is used to encode the text semantic representation tensor and output it to the multi-head attention mechanism module. The multi-head attention mechanism is used to output text embedding representation and entity embedding representation to the fusion module for feature fusion. The classification module is used to output error correction results based on the fused features.

[0067] The error correction model improves the existing Soft-MaskedBert model and adopts a text error correction method that integrates knowledge graph information. The knowledge graph itself is a network knowledge base formed by entities linked by relationships. It describes concepts, entities and their relationships in the objective world in a structured form, and expresses Internet information in a form closer to the human cognitive world. It is precisely because of the fusion of relevant information in the knowledge graph that the model has a more comprehensive understanding of the collocation of terms between text semantics, reducing the probability of term collocation errors.

[0068] The representation of the knowledge graph is G=(E, R, S), where E={e1, e2, …, e|E|} represents the entity set in the knowledge base, R={r1, r2, …, r|E|} represents the relationship set in the knowledge graph, and S∈E×R×E represents the set of triples in the knowledge graph. Traditional BERT can better mine the semantic information of text data for text error correction. This method adds knowledge graph information to jointly embed word embedding and knowledge embedding, so that the model can learn the complete semantics of semantic knowledge units and improve the error correction performance of the model.

[0069] Assume the input word sequence is {w 1 , w 2 ,…,w n}, where n is the length of the word sequence, and the entity vector sequence recognized by the word vector mapping module is recorded as {e 1 , e 2 ,…,e m}, where m represents the length of the entity sequence. Then {w 1 , w 2 ,…,w n} and {e 1 , e 2 ,…,e m} as the input of the encoder, and the encoder obtains their respective vectors {w 1o , w2 o ,…,w no} and {e 1o , e 2o ,…,e mo}, these are used as features of related tasks. Then {w 1o , w2 o ,…,w no} and {e 1o , e 2o ,…,e mo} is input into the fusion module to obtain fusion features. The fusion features are then input into the decoding module based on the multi-head attention mechanism.

[0070] Because Soft-MaskedBert comes with a mask, it can be simply understood as masking the entities in the knowledge graph. For example, in the music app, the singer "Wu Junyu" sang the song "17-year-old Rainy Season", and these two data resources are a kind of "singing" relationship; in the video app, the TV station "Hunan Satellite TV" produced an entertainment program "Transformation", and these two data resources are a kind of "production" relationship. This kind of relationship exists in the storage process of the knowledge graph. There is a certain relationship between the entities in the graph, so when the model mask is learned, it will focus on the data with such a relationship. In text error correction, this type of error belongs to "term collocation". Through the fusion of the knowledge graph, the entities themselves have a relationship, which can reduce the phenomenon of term collocation errors.

[0071] Step 4: Construct a rule dictionary and regular expressions, pre-annotate the replaced text based on the rule dictionary and regular expressions, and output the pre-annotated content.

[0072] Recommended annotations are displayed in different colors on the annotation tool page. Technicians can also match and modify the text first, and then send the modified text to the annotation tool and have it displayed on the annotation page.

[0073] Furthermore, pre-annotation of replaced text based on rule dictionary and regular expression includes: after sending data to the annotation tool, different dictionaries are constructed according to different tasks. First, hard matching of text is performed based on the dictionary. According to the hard matching principle, long strings are given priority to reduce the probability of manual withdrawal after matching. Text characters hard matched based on the dictionary are equivalent to direct annotation. In the text annotation task, 15% of time can be saved, and the effect is obvious. Second, matching based on regular expression, for entity characters with the same rules, based on the rules, only different colors need to be displayed, and the number of times the same entity appears is set by the threshold. After exceeding this threshold, the software automatically annotates the same entity, and finally manually confirms it, saving 10% of the annotation workload. Further, the threshold is set. During the customer annotation process, a confidence threshold is set for repeated entities. When the confidence exceeds a certain threshold, an entity prompt is given.

[0074] Step 5: Input the pre-annotated text into multiple sequence annotation models based on deep learning, and calculate the optimal annotation content according to the weight values ​​of the output results of multiple sequence annotation models.

[0075] Preferably, the plurality of sequence labeling models based on deep learning include a first model, a second model and a third model. The first model and the second model are as follows: Figure 3 shown.

[0076] The first model includes the ALBERT model, the BiGRU model and the CRF model. It consists of three modules: BERT, BiGRU and CRF. First, the ALBERT model is used to obtain word vectors and extract important features of the text; then the BiGRU deep learning context feature information is used to perform named entity recognition; finally, the CRF layer processes the output sequence of the BiGRU, combines the state transfer matrix in the CRF, and obtains a global optimal sequence based on adjacent labels.

[0077] The second model includes the BERT model, the BiLSTM model and the CRF model. This model is similar to the previous model and also has three modules. The effects produced by each module are also similar. Finally, an optimal sequence is obtained through the output of the model.

[0078] The third model includes the ERNIE model. In the pre-training of the ERNIE model, the first stage uses a basic level of masking, that is, randomly masking a Chinese character. The second stage uses a phrase-level masking to let the model predict the phrase. In the third stage, named entities, such as names of people, places, and institutions, are masked at this stage. The model learns these entities after the training is completed.

[0079] In the first model and the second model, BiLSTM and BiGRU are improved. Both BiLSTM and BiGRU add a pooling layer after the basic RNN model, such as Figure 4 The reason why this pooling layer is effective is that after a feature is found, its precise position is far less important than its relative position to other features. The pooling layer will continuously reduce the spatial size of the data, so the number of parameters and the amount of computation will also decrease, which also controls overfitting to a certain extent.

[0080] Obtaining the optimal annotation content according to the output results of multiple sequence annotation models includes the following steps: determining the weight coefficient of each sequence annotation model; if the output labels of all sequence annotation models are the same, taking the output label as the optimal annotation content; if the output labels of different sequence annotation models are different, multiplying the output labels of all sequence annotation models with the same output labels by their corresponding weight coefficients and then summing them to obtain the sum probability value of each type of output label, and selecting the output label with the largest sum probability value as the optimal annotation content.

[0081] The optimal result obtained by combining multiple models becomes the result of automatic annotation. For different data, the model will compare the results of manual annotation with the results of its own annotation. For different models and different texts, there will be a threshold determination, and the model will send the text to different models based on the probability of previous prediction results. The selection of the model is based on the combination of previous models, which are all taking the mode, finding the average, etc. The present invention is a method of combining the three models for evaluation.

[0082] The weight coefficient of each sequence labeling model is determined through training. Specifically, for each sequence labeling model, the error value between its predicted label and the manual labeling result in each round is calculated, and the error value of each round is averaged, and then the reciprocal is taken and divided by 100 to obtain the quotient as the weight coefficient of the sequence labeling model. The above method is used to obtain the corresponding weight coefficients of the three sequence labeling models. The weight coefficient will change continuously according to the rounds of training.

[0083] There will be an error value between the results of manual annotation and the results of model annotation. For the input text x, the model will have a prediction value, and each model will have a weight value W, for example W = (0.2, 0.34, 0.04). By calculating the prediction of each model for the label, if the predictions of the three models are all O labels, the final result is three models W1*(probability of predicted value O) + 2*(probability of predicted value O) + W3*(probability of predicted value O). If the prediction results of the three models are different, select a result with the smallest weight value *x probability value, add the same ones minus the different differences, W1(probability of predicted value I) - W2(probability of predicted value O) + W3(probability of predicted value O). Because there may be multiple prediction probability values ​​for the output of the model, if the three models predict three different labels, the prediction values ​​of the three models for the three labels will be obtained respectively, so as to find the weight values ​​of each model for different labels, and get the final prediction values ​​of the three labels respectively. Compare the calculation results of the three to see if they are greater than 0, and select the prediction result.

[0084] The deep learning models ALBERT / BERT+BiGRU / BiLSTM+CRF and ERNIE of the present invention have a significant effect on saving annotation time, including the model training stage and the knowledge entity recognition stage.

[0085] In the model training phase, based on the characteristics of the ALBERT / BERT model, ALBERT / BERT takes a single character as input, adds the start identifier CLS and the end identifier SEP at the beginning and end of the sentence, and outputs the embedding of each word. This model framework mainly uses the word embedding function of the pre-trained model ALBERT to fine-tune ALBERT / BERT. ERNIR is similar to the above two models and is also adjusted based on the BERT model.

[0086] The input of BiGRU / BiLSTM is the embedding output of ALBERT / BERT, with a project_layer added in the middle to ensure that its output is [batch_size, num_length, num_tags]. batch_size is the batch size in the model, num_length is the length of the input sentence, and num_tags is the number of sequence annotations, that is, the score of each word on each tag will be output.

[0087] The CRF layer is used to constrain the output of BiGRU. If the maximum score of each word in each tag is directly used as the output of BiGRU, a sequence like [B-NAME, O, I-NAME, O, I-Location] may appear, which is obviously not in line with the actual situation. The CRF layer can add some constraints to ensure that the final prediction result is valid. The CRF layer training obtains a transition probability score, which is the probability value of transferring from one label to another.

[0088] The json file obtained by processing the annotation software is processed and cleaned, and the data is converted into BIO format. B represents the beginning part of the knowledge entity, I represents the non-first character of the knowledge entity, [START] represents the beginning tag of the text, and represents the end tag of the text.

[0089] Depending on the number of rounds, the model's performance will improve accordingly until it approaches a stable state. The following is a description of the indicators:

[0090] Precision: P = TP / (TP+FP), Recall: R = TP / (TP+FN), F1 score: F1-score = 2*P*R / (P+R), TP: positive sample, predicted as a positive sample, FP: negative sample, predicted as a positive sample, FN: positive sample, predicted as a negative sample, TN: negative sample, predicted as a negative sample.

[0091] The purpose of multi-model combination is not to simply take the majority predicted by two models as the value, but to obtain a weight value for each model through the results of manual and model training. This weight value is used to judge the results predicted by the model, and multiple models are combined. Each model has a prediction value for the input text. The results of multiple prediction values ​​are compared to calculate the optimal result.

[0092] A sequence labeling system based on deep learning in an embodiment of the present invention, such as Figure 5 As shown, including:

[0093] A preprocessing module is used to obtain the text to be annotated and preprocess the text;

[0094] A translation error correction module is used to perform text translation error correction on the preprocessed text;

[0095] A rule matching module is used to construct a rule dictionary and regular expressions, pre-annotate the translated text after error correction based on the rule dictionary and regular expressions, and output the pre-annotated content;

[0096] The joint model annotation module is used to input the pre-annotated text into multiple sequence annotation models based on deep learning, and obtain the optimal annotation content according to the weight values ​​of the output results of multiple sequence annotation models.

[0097] The implementation principle and technical effects of the system are similar to the above method and will not be repeated here.

[0098] This embodiment also provides an electronic device, which includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes the steps of the sequence labeling method based on deep learning. The specific steps are shown in the method embodiment and will not be repeated here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc. The memory can be a volatile memory, a non-volatile memory, or a combination thereof.

[0099] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to implement the technical solution of any of the above-mentioned sequence labeling method embodiments based on deep learning. Its implementation principle and technical effect are similar to the above-mentioned method, and will not be repeated here.

[0100] It must be noted that in any of the above embodiments, the method is not necessarily executed in sequence according to the sequence number. As long as it cannot be inferred from the execution logic that it must be executed in a certain order, it means that it can be executed in any other possible order.

[0101] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sequence labeling method based on deep learning, characterized in that: Includes steps: Get the text to be annotated and preprocess the text; Perform text translation and error correction on the preprocessed text; Construct a rule dictionary and regular expressions, pre-annotate the text after translation error correction based on the rule dictionary and regular expressions, and output the pre-annotated content; Input the pre-annotated text into multiple sequence annotation models based on deep learning, and obtain the optimal annotation content according to the output results of multiple sequence annotation models; The translation process includes: A seq2seq model with an attention mechanism is used for translation. The seq2seq model includes an encoder and a decoder. The attention mechanism causes the decoder to pay different attention to different parts during decoding. The decoder includes a label sequence prediction module and a constraint module. The label sequence prediction module is used to output multiple predicted label sequences. The constraint module is used to define a feature function set. The set includes several feature functions. Each feature function is used to determine whether a label sequence meets the feature corresponding to the feature function. The scores of all feature functions in the set for the same predicted label sequence are obtained respectively. The scores of all feature functions in the set for the same predicted label sequence are summed as the final score of the predicted label sequence. In this way, the final score of each predicted label sequence output by the label sequence prediction module is obtained, and the predicted label sequence with the highest final score is selected as the translation processing result. The error correction process includes: Collect professional term libraries in different fields, preset the correct translation text of each professional term in the professional term library, and if a professional term in the professional term library exists in the translated text, and the translation of the professional term is different from the preset correct translation text, directly replace the translation of the professional term with the preset correct translation text; The text sequence processed in the previous step is sent to an error correction model based on deep learning. The error correction model includes a word vector mapping module, an error detection module, an encoding module, a multi-head attention mechanism module, a fusion module and a classification module. The word vector mapping module is used to convert the input text sequence into a text vector and output it to the error detection module. It is also used to identify entities in the text sequence according to a pre-built knowledge graph and convert the identified entities into entity vectors and output them to the multi-head attention mechanism module. The error detection module is used to obtain a text semantic representation tensor according to the text vector. The encoding module is used to encode the text semantic representation tensor and output it to the multi-head attention mechanism module. The multi-head attention mechanism module is used to output a text embedding representation and an entity embedding representation to the fusion module for feature fusion. The classification module is used to output an error correction result according to the fused features.

2. The deep learning-based sequence labeling method according to claim 1, wherein: The multiple deep learning-based sequence labeling models include a first model, a second model, and a third model; The first model includes an ALBERT model, a BiGRU model and a CRF model, wherein the ALBERT model is used to obtain word vectors and extract important features of text, the BiGRU model is used to perform named entity recognition through deep learning of context feature information, and the CRF model is used to process the output sequence of the BiGRU model, and obtain a global optimal sequence according to adjacent labels in combination with the state transfer matrix in the CRF model; The second model includes a BERT model, a BiLSTM model and a CRF model, wherein the BERT model is used to obtain word vectors and extract important features of text, the BiLSTM model is used to perform named entity recognition through deep learning context feature information, and the CRF model is used to process the output sequence of the BiLSTM model, and obtain a global optimal sequence according to adjacent labels in combination with the state transfer matrix in the CRF model; The third model includes the ERNIE model. In the pre-training of the ERNIE model, basic text masks are used in the first stage, phrase-level masks are used in the second stage to allow the model to predict phrases, and named entity masks are used in the third stage to allow the model to predict named entities.

3. A sequence labeling method based on deep learning as claimed in claim 2, characterized in that: The BiGRU model and the BiLSTM model both add a pooling layer after the basic RNN model.

4. The deep learning-based sequence labeling method according to claim 1, wherein: The obtaining of the optimal annotation content according to the output results of the multiple sequence annotation models includes: Determine the weight coefficient of each sequence labeling model; If the output labels of all sequence labeling models are the same, then the output label is used as the optimal label content; If the output labels of different sequence labeling models are different, the output labels of all sequence labeling models with the same output labels are multiplied by their corresponding weight coefficients and then summed to obtain the summed probability value of each type of output label, and the output label with the largest summed probability value is selected as the optimal labeling content.

5. A sequence labeling method based on deep learning as claimed in claim 4, characterized in that: The weight coefficient of each sequence labeling model is determined through training.

6. A sequence labeling system based on deep learning, characterized in that: include: A preprocessing module is used to obtain the text to be annotated and preprocess the text; A translation error correction module, used for performing text translation error correction on the preprocessed text; A rule matching module is used to construct a rule dictionary and regular expressions, pre-annotate the translated text after error correction based on the rule dictionary and regular expressions, and output the pre-annotated content; The joint model annotation module is used to input the pre-annotated text into multiple sequence annotation models based on deep learning, and obtain the optimal annotation content according to the output results of multiple sequence annotation models; The translation process includes: A seq2seq model with an attention mechanism is used for translation. The seq2seq model includes an encoder and a decoder. The attention mechanism causes the decoder to pay different attention to different parts during decoding. The decoder includes a label sequence prediction module and a constraint module. The label sequence prediction module is used to output multiple predicted label sequences. The constraint module is used to define a feature function set. The set includes several feature functions. Each feature function is used to determine whether a label sequence meets the feature corresponding to the feature function. The scores of all feature functions in the set for the same predicted label sequence are obtained respectively. The scores of all feature functions in the set for the same predicted label sequence are summed as the final score of the predicted label sequence. In this way, the final score of each predicted label sequence output by the label sequence prediction module is obtained, and the predicted label sequence with the highest final score is selected as the translation processing result. The error correction process includes: Collect professional term libraries in different fields, preset the correct translation text of each professional term in the professional term library, and if a professional term in the professional term library exists in the translated text, and the translation of the professional term is different from the preset correct translation text, directly replace the translation of the professional term with the preset correct translation text; The text sequence processed in the previous step is sent to an error correction model based on deep learning. The error correction model includes a word vector mapping module, an error detection module, an encoding module, a multi-head attention mechanism module, a fusion module and a classification module. The word vector mapping module is used to convert the input text sequence into a text vector and output it to the error detection module. It is also used to identify entities in the text sequence according to a pre-built knowledge graph and convert the identified entities into entity vectors and output them to the multi-head attention mechanism module. The error detection module is used to obtain a text semantic representation tensor according to the text vector. The encoding module is used to encode the text semantic representation tensor and output it to the multi-head attention mechanism module. The multi-head attention mechanism module is used to output a text embedding representation and an entity embedding representation to the fusion module for feature fusion. The classification module is used to output an error correction result according to the fused features.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Seq2seq model-based Chinese word segmentation method

    CN108491372A

  • A method for jointly extracting entity relations from sequential annotations based on attention mechanism

    CN109408812A