Entity labeling and recognition method and device based on deep learning

Abstract

The invention belongs to the technical field of named entity recognition, and particularly relates to an entity labeling and recognition method and device based on deep learning, and the method comprises the steps: obtaining a training text, and labeling the training text through a BIO labeling method; the marked training text is processed; the processed text is input into a pre-trained BERT-CRF model, and fine tuning of the model on named entity recognition is completed; inputting a to-be-recognized text A into the fine-tuned BERT-CRF model to obtain an entity set of each category; performing topic analysis on a to-be-recognized text A by utilizing the trained LDA model to obtain a topic of the text A and a keyword of which the probability is greater than a set value under the topic; and verifying the entity set of each category, and correcting the entities which are not matched with the main body and the keyword in the set. The entity can be manually corrected, and the accuracy of named entity recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of named entity recognition, and specifically relates to a method and device for entity labeling and recognition based on deep learning. Background Art

[0002] The task of named entity recognition in text data is one of the core tasks of natural language processing. Its purpose is to identify specific categories of entities from massive texts, such as names of people, places, and organizational names. When facing Chinese text, the difficulties faced by the named entity recognition task are: (1) There are many types and numbers of named entities, and the generalization ability is insufficient when dealing with newly emerged named entities or new named entities in specific fields. (2) There are many words in the text that have "multiple pronunciations" and "multiple meanings", which need to be analyzed in combination with the context and language of the word. (3) The structure of named entities is relatively complex. For example, there are a large number of nested, aliased, and abbreviation problems in organizational structures, and there is no strict naming rule.

[0003] Faced with this series of problems, existing methods such as rule-based methods, traditional machine learning-based methods, deep learning-based methods, etc., have low accuracy in named entity recognition when "polysemy" occurs in the text. Summary of the invention

[0004] In view of the problem that the traditional method of named entity recognition has low accuracy, the present invention provides an entity labeling and recognition method and device based on deep learning.

[0005] In a first aspect, the technical solution of the present invention provides an entity labeling and recognition method based on deep learning, comprising the following steps: Obtain training text and annotate the training text using BIO annotation method; Process the annotated training text; Input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; Input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; Use the trained LDA model to perform topic analysis on the text A to be recognized, and obtain the topic of the text A and the keywords under the topic with a probability greater than a set value; The entity sets of each category are verified according to the subject and the keywords under the subject with probabilities greater than a set value, and the entities in the set that do not match the subject and the keywords are corrected.

[0006] As a further limitation of the technical solution of the present invention, the step of processing the annotated training text includes: Convert all marked types into dictionaries of key-value pairs; The training text is segmented, and each sentence in the training text is processed and packaged into an entity in combination with the dictionary.

[0007] As a further limitation of the technical solution of the present invention, the steps of performing word segmentation processing on the training text and processing each sentence in the training text and encapsulating it into an entity by combining it with the dictionary include: Convert each sentence in the training text into a list, recorded as the first list, and convert the corresponding categories into lists according to the marked types, recorded as the second list; Fill the first list and the second list with 0 to a set maximum length, to obtain the filled first list and the second list; Add the first marker to the front of the first list after filling, add the second marker to the end of the first list after filling, and convert it into a coded list according to the index in the corpus, which is recorded as the third list; Add the first mark to the front of the filled second list, add the second mark to the end of the filled second list, and convert it into an encoded list according to the value in the dictionary, which is recorded as the fourth list; wherein each sentence in the text is a sequence of length E; A mask list for indicating the position information of the characters is created based on the information in the fourth list, which is recorded as the fifth list; at the same time, a position list for indicating the sentence component information of the characters is created based on the information in the fourth list, which is recorded as the sixth list; Encapsulate the third list, the fifth list, the sixth list, and the fourth list into one entity.

[0008] As a further limitation of the technical solution of the present invention, the steps of inputting the processed data into a pre-trained BERT-CRF model and completing fine-tuning the BERT-CRF model for named entity recognition include: Input all packaged entities into the BERT model unit in the pre-trained BERT-CRF model; The BERT model unit operates on a set number of sequences per batch and outputs the optimal result by optimizing the cross entropy loss function; The CRF unit in the pre-trained BERT-CRF model constructs a state transfer matrix based on the preset constraints and number of categories; Use the optimal results output by the BERT model unit, the constructed state transfer matrix and the likelihood function to calculate and optimize the loss value, obtain the optimal path in the state transfer matrix, and complete the fine-tuning of the BERT-CRF model for named entity recognition.

[0009] As a further limitation of the technical solution of the present invention, the BERT model unit operates on a set number of sequences per batch, and in the step of outputting the optimal result by optimizing the cross entropy loss function, the set number is N, each word in the sequence is encoded as a B-dimensional vector, and the BERT model unit operates on a (N, E, B)-dimensional matrix per batch, and the matrix is ​​recorded as the first matrix; the specific steps are as follows: The first matrix is ​​reshaped into the second matrix and input into the BERT model unit; the second matrix is ​​linearly transformed through the Q, K, and V matrices of the BERT model unit to obtain the third matrix; Construct weight matrix W, bias matrix , reshape the third matrix into the fourth matrix Znew; Calculate and optimize the cross entropy loss function to obtain the optimal result matrix; Among them, the cross entropy loss function .

[0010] As a further limitation of the technical solution of the present invention, the step of calculating and optimizing the loss value using the optimal result output by the BERT model unit, the constructed state transfer matrix and the likelihood function to obtain the optimal path in the state transfer matrix includes: Given a sequence x of length n, and a label sequence y of length n, , , ,in Represents the probability value of the label corresponding to each word in the matrix output by the BERT model, It is a characteristic function, usually taking 0 or 1, indicating whether a certain state characteristic is satisfied; Represents the probability value corresponding to each state transition in the state transition matrix, It is a characteristic function, usually taking 0 or 1, indicating whether a certain transfer feature is satisfied. ,in , use Z(x) to achieve the normalization of the final result, y represents a path formed by the labels of x, and the set formed by the permutations and combinations of all labels is y'.

[0011] right Take the negative log-likelihood function, that is The optimal solution is obtained by using the gradient descent method on the loss, which is the optimal path.

[0012] As a further limitation of the technical solution of the present invention, the step of using the trained LDA model to perform topic analysis on the text A to be recognized to obtain the topic of the text A and the keywords under the topic with a probability greater than a set value includes: Set model parameters and train the LDA model; model parameters include the number of topics, number of iterations, learning rate, and ,in, represents the text-topic density, Indicates topic-word density; The specific training process includes: For each topic, randomly initialize a word distribution; For each word in each text, calculate the probability that the word belongs to each topic; Update the text-topic distribution and topic-word distribution according to the calculated probabilities; For each text, update the topic distribution of the text according to the current topic-word distribution; For each topic, update the word distribution of the topic based on the current text-topic distribution; Until the model converges or reaches the preset number of iterations, a trained LDA model is obtained.

[0013] As a further limitation of the technical solution of the present invention, the step of using the trained LDA model to perform topic analysis on the text A to be recognized to obtain the topic of the text A and the keywords under the topic with a probability greater than a set value includes: Perform topic analysis on the text A using the trained LDA model, and output the topic distribution of the text and the word distribution of each topic; According to the topic with the highest probability ranking in text A, the topic of text A is confirmed; and the keywords under the topic whose probability is greater than the set threshold are confirmed.

[0014] In a second aspect, the technical solution of the present invention also provides an entity annotation and recognition device based on deep learning, including a training text annotation module, a training text processing module, a model fine-tuning module, an entity recognition execution module, a topic analysis module and a recognition result verification and correction module; A training text annotation module is used to obtain training text and annotate the training text using the BIO annotation method; A training text processing module is used to process the annotated training text; The model fine-tuning module is used to input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; The entity recognition execution module is used to input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; A topic analysis module is used to perform topic analysis on the text A to be recognized using the trained LDA model to obtain the topic of the text A and keywords with a probability greater than a set value under the topic; The recognition result verification and correction module is used to verify the entity sets of each category according to the subject and the keywords with probability greater than a set value under the subject, and correct the entities in the set that do not match the subject and the keywords.

[0015] As a further limitation of the technical solution of the present invention, the training text processing module includes a tag type processing unit and an analysis processing unit; A tag type processing unit, used to convert all tagged types into a dictionary of key-value pairs; The analysis and processing unit is used to perform word segmentation on the training text, and process each sentence in the training text and encapsulate it into an entity in combination with the dictionary.

[0016] As a further limitation of the technical solution of the present invention, the analysis and processing unit is specifically used to convert each sentence in the training text into a list, recorded as the first list, and convert the corresponding classification into a list according to the type of annotation, recorded as the second list; fill the first list and the second list with 0 to a set maximum length to obtain the filled first list and the second list; add a first mark to the front of the filled first list, add a second mark to the end of the filled first list, and convert it into an encoded list according to the index in the corpus, recorded as the third list; add a first mark to the front of the filled second list, add a second mark to the end of the filled second list, and convert it into an encoded list according to the value in the dictionary, recorded as the fourth list; wherein each sentence in the text is a sequence of length E; create a mask list for indicating the position information of the word according to the information of the fourth list, recorded as the fifth list; at the same time, create a position list for indicating the sentence component information of the word according to the information of the fourth list, recorded as the sixth list; encapsulate the third list, the fifth list, the sixth list, and the fourth list into one entity.

[0017] As a further limitation of the technical solution of the present invention, the model fine-tuning module includes an input execution unit, a model unit and an optimization fine-tuning unit; The input execution unit is used to input all packaged entities into the BERT model unit in the pre-trained BERT-CRF model; The model units include BERT model units and CRF units; The BERT model unit operates on a set number of sequences per batch and outputs the best result by optimizing the cross entropy loss function; the CRF unit in the pre-trained BERT-CRF model constructs a state transfer matrix based on the preset constraints and number of categories; The optimization fine-tuning unit is used to use the optimal results output by the BERT model unit, the constructed state transfer matrix and the likelihood function to calculate and optimize the loss value, obtain the optimal path in the state transfer matrix, and complete the fine-tuning of the BERT-CRF model for named entity recognition.

[0018] As a further limitation of the technical solution of the present invention, the BERT model unit operates on a set number of sequences per batch, and in the step of outputting the optimal result by optimizing the cross entropy loss function, the set number is N, each word in the sequence is encoded as a B-dimensional vector, and the BERT model unit operates on a (N, E, B)-dimensional matrix per batch, and the matrix is ​​recorded as the first matrix; the specific steps are as follows: Reshape the first matrix into the second matrix and input it into the BERT model unit; linearly transform the second matrix through the Q, K, V matrices of the BERT model unit to obtain the third matrix; construct the weight matrix W and the bias matrix , reshape the third matrix into the fourth matrix Znew; calculate and optimize the cross entropy loss function to obtain the optimal result matrix; where the cross entropy loss function .

[0019] As a further limitation of the technical solution of the present invention, the step of calculating and optimizing the loss value using the optimal result output by the BERT model unit, the constructed state transfer matrix and the likelihood function to obtain the optimal path in the state transfer matrix includes: Given a sequence x of length n, and a label sequence y of length n, , , ,in Represents the probability value of the label corresponding to each word in the matrix output by the BERT model, It is a characteristic function, usually taking 0 or 1, indicating whether a certain state characteristic is satisfied; Represents the probability value corresponding to each state transition in the state transition matrix, It is a characteristic function, usually taking 0 or 1, indicating whether a certain transfer feature is satisfied. ,in , use Z(x) to achieve the normalization of the final result, y represents a path formed by the labels of x, and the set formed by the permutations and combinations of all labels is y'.

[0020] right Take the negative log-likelihood function, that is The optimal solution is obtained by using the gradient descent method on the loss, which is the optimal path.

[0021] As a further limitation of the technical solution of the present invention, the device also includes a topic model training module for setting model parameters and training the LDA model; the model parameters include the number of topics, the number of iterations, the learning rate, and ,in, represents the text-topic density, Indicates topic-word density; The specific training process of the topic model training module includes: for each topic, randomly initialize a word distribution; for each word in each text, calculate the probability that the word belongs to each topic; based on the calculated probability, update the text-topic distribution and topic-word distribution; for each text, update the topic distribution of the text based on the current topic-word distribution; for each topic, update the word distribution of the topic based on the current text-topic distribution; until the model converges or reaches the preset number of iterations, a trained LDA model is obtained.

[0022] As a further limitation of the technical solution of the present invention, the topic analysis module is specifically used to use the trained LDA model to perform topic analysis on the text A, output the topic distribution of the text, and the word distribution of each topic; confirm the topic of text A based on the topic with the highest probability ranking in text A; and the keywords under the topic whose probability is greater than a set threshold.

[0023] It can be seen from the above technical solutions that the present invention has the following advantages: Compared with the traditional named entity recognition method, the BERT-CRF model can associate the context in which the word is located, thereby improving the accuracy of named entity recognition and effectively solving some "polysemy" problems. Secondly, with the help of the text topic analysis ability of the LDA model, the topic and core keywords of the text are provided, and the entity can be manually corrected, further improving the accuracy of named entity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0026] Figure 2 This is an example diagram of text annotation in an embodiment of the present invention.

[0027] Figure 3 is a schematic block diagram of an apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present application proposes a method for entity labeling and recognition based on deep learning, which can effectively improve the accuracy of named entity recognition in the case of "polysemy". The method utilizes the bidirectional encoding architecture of the BERT model, which can provide powerful context understanding capabilities, effectively extract feature vectors, and then use the CRF model to add restrictions, remove obvious entity recognition errors, decode the feature vectors and select the optimal sequence combination to complete entity recognition, and finally use the characteristics of the LDA model in the direction of topic classification to analyze the text topic, and use the text topic to manually correct the obviously erroneous named entities. In order to enable personnel in this technical field to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described in combination with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative labor should belong to the scope of protection of the present invention.

[0029] like Figure 1 As shown, an embodiment of the present invention provides an entity labeling and recognition method based on deep learning, comprising the following steps: Step 1: Obtain training text and annotate it using the BIO annotation method; Step 2: Process the annotated training text; This step specifically includes: converting all the marked types into a dictionary of key-value pairs; performing word segmentation on the training text, and processing each sentence in the training text and encapsulating it into an entity by combining the dictionary.

[0030] Step 3: Input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; This step specifically includes: inputting all packaged entities into the BERT model unit in the pre-trained BERT-CRF model; the BERT model unit operates on a set number of sequences per batch and outputs the optimal result by optimizing the cross entropy loss function; the CRF unit in the pre-trained BERT-CRF model constructs a state transfer matrix according to the preset constraints and the number of classifications; Use the optimal results output by the BERT model unit, the constructed state transfer matrix and the likelihood function to calculate and optimize the loss value, obtain the optimal path in the state transfer matrix, and complete the fine-tuning of the BERT-CRF model for named entity recognition.

[0031] Step 4: Input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; Step 5: Use the trained LDA model to perform topic analysis on the text A to be recognized, and obtain the topic of the text A and the keywords whose probability under the topic is greater than the set value; Step 6: Verify the entity sets of each category according to the subject and the keywords with probability greater than a set value under the subject, and modify the entities in the set that do not match the subject and the keywords.

[0032] It should be noted that the BERT-CRF model is a deep learning model that combines the BERT pre-trained model and the conditional random field (CRF) layer. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model. Through the bidirectional Transformer architecture, BERT can capture rich contextual information in the text and provide high-quality word embedding for sequence labeling tasks. CRF (Conditional Random Fields) is a probabilistic graph model that can take into account the dependencies between labels and select the optimal label sequence through global optimization. In the BERT-CRF model, the CRF layer is responsible for decoding the feature vector output by the BERT model and generating the final label sequence.

[0033] The LDA (Latent Dirichlet Allocation) model is a topic model. In the LDA model, each document is regarded as a probability distribution consisting of multiple potential topics, and each topic is represented by a word distribution in the vocabulary.

[0034] The embodiment of the present invention also provides an entity labeling and recognition method based on deep learning, comprising the following steps: Step S1: Prepare training data and manually annotate the text data for training using the BIO annotation method (i.e., B represents the beginning of the entity, I represents the word inside the entity, and O represents the non-entity part), such as Figure 2 shown.

[0035] Step S2: Use the trained BERT-BASE, chinese_L-12_H-768_A-12 model (i.e. 12 layers, 768-dimensional vector, 12 self-attention heads). During the fine-tuning of the BERT model unit, take the training text data annotated in step S1 as input and perform the following processing: (1) Convert all categories into dictionaries of key-value pairs; For example, if there are 7 categories, the corresponding dictionary is: label_map={O:1,B-PER:2,I-PER:3,B-LOC:4,I-LOC:5,CLS:6,SEP:7}; (2) Perform word segmentation on the training text. Specifically, perform steps (3) to (6) on each sentence in the training text. (3) Convert each sentence into a list, recorded as tokens, for example, tokens = [I, and, Zhang, San, went, to, Shijiazhuang, for, a, trip]. At the same time, convert the corresponding categories into a list according to the annotation type, recorded as labels, for example, labels = [O, O, B-PER, I-PER, O, O, B-LOC, I-LOC, I-LOC, O, O]; (4) Fill tokens and labels with 0 to the agreed maximum length; For example, if the maximum length is 128, then after padding , , the current length is 126; (5) Add the [CLS] tag before tokens and the [SEP] tag at the end, and then convert it into a coded list based on the index in the corpus, recorded as input_ids; For example .

[0036] Add the [CLS] tag before labels and the [SEP] tag at the end, and then convert it into a coded list according to the value in the label_map dictionary, recorded as label_ids; For example .

[0037] Create a mask list to indicate the position information of the word, that is, the value of the part with content is 1, and the value of the remaining filled part is 0, recorded as input_mask, .

[0038] Create a position list to indicate the sentence component information of the word. Since they are all in one sentence, the values ​​are all 0, recorded as segment_ids. .

[0039] (6) Encapsulate the input_ids, input_mask, segment_ids, and label_ids obtained in step (5) into one entity; (7) All encapsulated entities in the text are input into the pre-trained BERT model. Each sentence in the text is equivalent to a sequence of length 128. The BERT model operates on 8 sequences per batch. Each word in the sequence is encoded as a 768-dimensional vector, that is, an (8, 128, 768)-dimensional matrix is ​​operated. The matrix is ​​recorded as .

[0040] (8) Reshape the matrix first for To facilitate calculation, the matrix Execute the self-attention mechanism in the BERT model, that is, use the Q, K, V matrices to perform self-attention processing on each word in the associated context, and finally obtain the matrix .

[0041] Step S3: Use logistic regression to quantify the matrix For multiple classifications, for example, there are 7 categories and a weight matrix is ​​constructed. , the bias matrix , then reshape for , the purpose of reshaping the matrix is ​​to transform the dimension to meet the input needs; calculate , by optimizing Cross entropy loss, get the best result .

[0042] Step S4: Add a CRF layer based on the BERT model, because in CRF, the state transfer matrix can be constructed according to the restrictions (for example, each entity category must start with B, not I or O, B of the same entity category can only be followed by I of the same entity category, not I of other categories or directly followed by O, etc.) and the number of categories, thereby optimizing the final classification results. For example, if there are 5 categories, the state transfer matrix , as shown in Table 1.

[0043] Table 1: State transition matrix

[0044] In this state transition matrix, each element represents the probability of state transition from i to j, for example It means the probability of state transition from O to B-PER is b.

[0045] Therefore, some probabilities can be optimized according to the constraints, for example, the probability of the state changing from B-PER to I-LOC is 0, the probability of the state changing from O to I-LOC is 0, and so on.

[0046] Using the output of step S3 , state transfer matrix The loss value is calculated and optimized with the likelihood function to find the optimal path in the state transfer matrix. That is, given a sequence x with a length of 1024, its label sequence (7 categories correspond to 7 labels) is y, and the length of y is also 1024. , , ,in Represents the probability value of the label corresponding to each word in the matrix output by the BERT model, It is a characteristic function, usually taking 0 or 1, indicating whether a certain state characteristic is satisfied; Represents the probability value corresponding to each state transition in the state transition matrix, is a characteristic function, usually 0 or 1, indicating whether a certain transfer characteristic is satisfied. In this embodiment, n is 1024. Conditional probability ,in , use Z(x) to achieve the normalization of the final result, y represents a path formed by the labels of x, and the set formed by the permutations and combinations of all labels is y'.

[0047] Emit The probability value of the corresponding label of each word in the matrix under the label sequence y, Trans corresponds to the state transfer matrix The probability value corresponding to the transition from the previous state to the current state.

[0048] Then Take the negative log-likelihood function, that is The optimal solution is obtained by gradient descent method on the loss, which is the optimal path. For each row in , the number with the largest probability among all columns is the classification of the corresponding word.

[0049] Step S5: Based on steps S1 to S4, the BERT-CRF model is fine-tuned for named entity recognition.

[0050] Step S6: Use the BERT-CRF model output from step S5 to perform a named entity recognition task on a text, denoted as text A, and obtain a set of entities of each category, denoted as S.

[0051] For example, the entity set obtained by performing named entity recognition task on the text "Zhang San likes to eat fruits, so he bought 3 apples and 1 watermelon in a shopping mall in Beijing" is recorded as: .

[0052] Step S7: Data preprocessing is performed on N texts related to text A. This step includes word segmentation, stop word removal, punctuation and number processing, part-of-speech analysis (only nouns, proper nouns and gerunds are retained through part-of-speech analysis), etc. After preprocessing, the result is formed into a dictionary, and each word in the dictionary is assigned a unique integer ID. The dictionary object stores the mapping relationship between words and their corresponding IDs, as well as the frequency of words in the entire text collection. For each text, traverse each word in the text, find the ID of each word in the dictionary, calculate the number of times each word appears in the text (i.e., word frequency), combine each word ID and the corresponding word frequency into a tuple (word_id, word_freq), put all such tuples into a list to form a sparse vector. The length of the vector should be equal to the size of the dictionary. This vector is the bag-of-words model representation of the text, also known as a corpus.

[0053] Step S8: Train the LDA model, specify the number of topics (the number of topics can be determined by the topic perplexity method. In the case of overfitting, the value with the minimum topic perplexity is selected as the number of topics. The curve of topic perplexity and the number of topics can be expressed in a graphical way. Usually, the topic perplexity will reach a global minimum as the number of topics increases. The number n corresponding to the global minimum of the topic perplexity can be selected as the number of topics. At the same time, too many topics should be avoided to cause overfitting), number of iterations, learning rate, and And other parameters, and The parameters are the parameters of the Dirichlet prior distribution, represents the text-topic density, The higher it is, the more topics the text contains, and vice versa. represents the topic-word density, The higher the value, the more words the topic contains, and vice versa. You can set it by default. and Both are equal to 1 / number of topics. The model training process is as follows: (1) For each topic, randomly initialize a word distribution, usually a uniform distribution; (2) For each word in each text, calculate the probability that the word belongs to each topic; Based on the probabilities calculated in step (2), update the text-topic distribution and topic-word distribution; (4) For each text, update the topic distribution of the text based on the current topic-word distribution; (5) For each topic, update the word distribution of the topic based on the current text-topic distribution; (6) Repeat steps (2) to (5) until the model converges or the preset number of iterations is reached. Convergence can be determined by monitoring the change in ELBO or the change in topic-word distribution.

[0054] After the training is completed, the trained LDA model is recorded as , and the topic distribution of each text and the word distribution of each topic are output.

[0055] Step S9: Filter the top 10 words with the highest probability in each topic in step S8, and then manually customize the topic based on the top 10 words with the highest probability in each topic. Use the LDA model trained in step S8 to perform topic analysis on text A in step S6, and then confirm the topic words of text A based on the topic with the highest probability in text A. The topic of text A and the top 5 keywords with the highest probability in the topic are the output of this step.

[0056] Step S10: According to the topic and keywords output by step S9, the entity set S of each category output by step S6 is manually verified. If an entity that obviously does not match the topic and keywords appears in the entity set of each category, the named entity is manually corrected and moved to other more matching categories. For example, the topic of text A after LDA model analysis is "fruit purchase". After the text is recognized by the BERT-CRF model, the categories are "fruit", "electronic brand", etc. If the BERT-CRF model recognizes "apple" as "electronic brand", then according to the results of the topic analysis, it is more reasonable for "apple" to be classified into the category of "fruit".

[0057] like Figure 3 As shown, an embodiment of the present invention further provides an entity annotation and recognition device based on deep learning, including a training text annotation module, a training text processing module, a model fine-tuning module, an entity recognition execution module, a topic analysis module and a recognition result verification and correction module; A training text annotation module is used to obtain training text and annotate the training text using the BIO annotation method; A training text processing module is used to process the annotated training text; The model fine-tuning module is used to input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; The entity recognition execution module is used to input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; A topic analysis module is used to perform topic analysis on the text A to be recognized using the trained LDA model to obtain the topic of the text A and keywords with a probability greater than a set value under the topic; The recognition result verification and correction module is used to verify the entity sets of each category according to the subject and the keywords with probability greater than a set value under the subject, and correct the entities in the set that do not match the subject and the keywords.

[0058] In some embodiments, the training text processing module includes a tag type processing unit and an analysis processing unit; A tag type processing unit, used to convert all tagged types into a dictionary of key-value pairs; The analysis and processing unit is used to perform word segmentation on the training text, and process each sentence in the training text and encapsulate it into an entity in combination with the dictionary.

[0059] In some embodiments, the analysis and processing unit is specifically used to convert each sentence in the training text into a list, recorded as the first list, and convert the corresponding classification into a list according to the type of annotation, recorded as the second list; fill the first list and the second list with 0 to a set maximum length to obtain the filled first list and the second list; add a first tag to the front of the filled first list and a second tag to the end of the filled first list, and convert it into an encoded list according to the index in the corpus, recorded as the third list; add a first tag to the front of the filled second list and a second tag to the end of the filled second list, and convert it into an encoded list according to the value in the dictionary, recorded as the fourth list; wherein each sentence in the text is a sequence of length E; create a mask list for indicating the position information of the word according to the information in the fourth list, recorded as the fifth list; at the same time, create a position list for indicating the sentence component information of the word according to the information in the fourth list, recorded as the sixth list; encapsulate the third list, the fifth list, the sixth list, and the fourth list into one entity.

[0060] In some embodiments, the model fine-tuning module includes an input execution unit, a model unit, and an optimization fine-tuning unit; The input execution unit is used to input all packaged entities into the BERT model unit in the pre-trained BERT-CRF model; The model units include BERT model units and CRF units; The BERT model unit operates on a set number of sequences per batch and outputs the best result by optimizing the cross entropy loss function; the CRF unit in the pre-trained BERT-CRF model constructs a state transfer matrix based on the preset constraints and number of categories; The optimization fine-tuning unit is used to use the optimal results output by the BERT model unit, the constructed state transfer matrix and the likelihood function to calculate and optimize the loss value, obtain the optimal path in the state transfer matrix, and complete the fine-tuning of the BERT-CRF model for named entity recognition.

[0061] In some embodiments, the BERT model unit operates on a set number of sequences per batch, and in the step of outputting the optimal result by optimizing the cross entropy loss function, the set number is N, each word in the sequence is encoded as a B-dimensional vector, and the BERT model unit operates on a (N, E, B)-dimensional matrix per batch, and the matrix is ​​recorded as the first matrix; the specific steps are as follows: Reshape the first matrix into the second matrix and input it into the BERT model unit; linearly transform the second matrix through the Q, K, V matrices of the BERT model unit to obtain the third matrix; construct the weight matrix W and the bias matrix , reshape the third matrix into the fourth matrix Znew; calculate and optimize the cross entropy loss function to obtain the optimal result matrix; where the cross entropy loss function .

[0062] In some embodiments, the step of calculating and optimizing the loss value using the optimal result output by the BERT model unit, the constructed state transfer matrix, and the likelihood function to obtain the optimal path in the state transfer matrix includes: Given a sequence x of length n, and a label sequence y of length n, , , ,in Represents the probability value of the label corresponding to each word in the matrix output by the BERT model, It is a characteristic function, usually taking 0 or 1, indicating whether a certain state characteristic is satisfied; Represents the probability value corresponding to each state transition in the state transition matrix, It is a characteristic function, usually taking 0 or 1, indicating whether a certain transfer feature is satisfied. ,in , use Z(x) to achieve the normalization of the final result, y represents a path formed by the labels of x, and the set formed by the permutations and combinations of all labels is y'.

[0063] P Take the negative log-likelihood function, that is The optimal solution is obtained by using the gradient descent method on the loss, which is the optimal path.

[0064] In some embodiments, the device further includes a topic model training module for setting model parameters to train the LDA model; the model parameters include the number of topics, the number of iterations, the learning rate, and ,in, represents the text-topic density, Indicates topic-word density; The specific training process of the topic model training module includes: for each topic, randomly initialize a word distribution; for each word in each text, calculate the probability that the word belongs to each topic; based on the calculated probability, update the text-topic distribution and topic-word distribution; for each text, update the topic distribution of the text based on the current topic-word distribution; for each topic, update the word distribution of the topic based on the current text-topic distribution; until the model converges or reaches the preset number of iterations, a trained LDA model is obtained.

[0065] In some embodiments, the topic analysis module is specifically used to perform topic analysis on the text A using a trained LDA model, output the topic distribution of the text, and the word distribution of each topic; confirm the topic of text A based on the topic with the highest probability ranking in text A; and the keywords under the topic whose probability is greater than a set threshold.

[0066] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0067] An embodiment of an entity labeling and recognition device based on deep learning is provided in an embodiment of the present invention. The device and the entity labeling and recognition method based on deep learning in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the entity labeling and recognition device based on deep learning, please refer to the embodiment of the above-mentioned entity labeling and recognition method based on deep learning.

[0068] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A method for entity labeling and recognition based on deep learning, characterized in that: The steps include: Obtain training text and annotate the training text using BIO annotation method; Process the annotated training text; Input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; Input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; Use the trained LDA model to perform topic analysis on the text A to be recognized, and obtain the topic of the text A and the keywords under the topic with a probability greater than a set value; The entity sets of each category are verified according to the subject and the keywords under the subject with probabilities greater than a set value, and the entities in the set that do not match the subject and the keywords are corrected.

2. The entity labeling and recognition method based on deep learning according to claim 1, characterized in that: The steps for processing the labeled training text include: Convert all marked types into dictionaries of key-value pairs; The training text is segmented, and each sentence in the training text is processed and packaged into an entity in combination with the dictionary.

3. The entity labeling and recognition method based on deep learning according to claim 2, characterized in that: The steps of performing word segmentation processing on the training text and processing each sentence in the training text and encapsulating it into an entity by combining it with the dictionary include: Convert each sentence in the training text into a list, recorded as the first list, and convert the corresponding categories into lists according to the marked types, recorded as the second list; Fill the first list and the second list with 0 to a set maximum length, to obtain the filled first list and the second list; Add the first marker to the front of the first list after filling, add the second marker to the end of the first list after filling, and convert it into a coded list according to the index in the corpus, which is recorded as the third list; Add the first mark to the front of the filled second list, add the second mark to the end of the filled second list, and convert it into an encoded list according to the value in the dictionary, which is recorded as the fourth list; wherein each sentence in the text is a sequence of length E; A mask list for indicating the position information of the characters is created based on the information in the fourth list, which is recorded as the fifth list; at the same time, a position list for indicating the sentence component information of the characters is created based on the information in the fourth list, which is recorded as the sixth list; Encapsulate the third list, the fifth list, the sixth list, and the fourth list into one entity.

4. The entity labeling and recognition method based on deep learning according to claim 3 is characterized in that: The processed data is input into the pre-trained BERT-CRF model. The steps to complete the fine-tuning of the BERT-CRF model for named entity recognition include: Input all packaged entities into the BERT model unit in the pre-trained BERT-CRF model; The BERT model unit operates on a set number of sequences per batch and outputs the optimal result by optimizing the cross entropy loss function; The CRF unit in the pre-trained BERT-CRF model constructs a state transfer matrix based on the preset constraints and number of categories; Use the optimal results output by the BERT model unit, the constructed state transfer matrix and likelihood function to calculate and optimize the loss value, obtain the optimal path in the state transfer matrix, and complete the fine-tuning of the BERT-CRF model for named entity recognition.

5. The entity labeling and recognition method based on deep learning according to claim 4, characterized in that: The BERT model unit operates on a set number of sequences per batch and outputs the optimal result by optimizing the cross entropy loss function. The set number is N, each word in the sequence is encoded as a B-dimensional vector, and the BERT model unit operates on a (N, E, B)-dimensional matrix per batch. The matrix is ​​recorded as the first matrix. The specific steps are as follows: The first matrix is ​​reshaped into the second matrix and input into the BERT model unit; the second matrix is ​​linearly transformed through the Q, K, and V matrices of the BERT model unit to obtain the third matrix; Construct weight matrix W, bias matrix , reshape the third matrix into the fourth matrix Znew; Calculate and optimize the cross entropy loss function to obtain the optimal result matrix; Among them, the cross entropy loss function .

6. The entity labeling and recognition method based on deep learning according to claim 5, characterized in that: The step of using the trained LDA model to perform topic analysis on the text A to be recognized to obtain the topic of the text A and the keywords under the topic with a probability greater than a set value includes: Set model parameters and train the LDA model; model parameters include the number of topics, number of iterations, learning rate, and ,in, represents the text-topic density, Indicates topic-word density.

7. The entity labeling and recognition method based on deep learning according to claim 6, characterized in that: The steps to train the LDA model include: For each topic, randomly initialize a word distribution; For each word in each text, calculate the probability that the word belongs to each topic; Update the text-topic distribution and topic-word distribution according to the calculated probabilities; For each text, update the topic distribution of the text according to the current topic-word distribution; For each topic, update the word distribution of the topic based on the current text-topic distribution; Until the model converges or reaches the preset number of iterations, a trained LDA model is obtained.

8. The entity labeling and recognition method based on deep learning according to claim 7, characterized in that: The steps of using the trained LDA model to perform topic analysis on the text A to be recognized to obtain the topic of the text A and keywords with a probability greater than a set value under the topic include: Perform topic analysis on the text A using the trained LDA model, and output the topic distribution of the text and the word distribution of each topic; According to the topic with the highest probability ranking in text A, the topic of text A is confirmed; and the keywords under the topic whose probability is greater than the set threshold are confirmed.

9. A deep learning-based entity labeling and recognition device, characterized in that: It includes a training text annotation module, a training text processing module, a model fine-tuning module, an entity recognition execution module, a topic analysis module, and a recognition result verification and correction module; A training text annotation module is used to obtain training text and annotate the training text using the BIO annotation method; A training text processing module is used to process the annotated training text; The model fine-tuning module is used to input the processed text into the pre-trained BERT-CRF model to complete the fine-tuning of the BERT-CRF model for named entity recognition; The entity recognition execution module is used to input the text A to be recognized into the fine-tuned BERT-CRF model to perform the named entity recognition task and obtain the entity sets of each category; A topic analysis module is used to perform topic analysis on the text A to be recognized using the trained LDA model to obtain the topic of the text A and keywords with a probability greater than a set value under the topic; The recognition result verification and correction module is used to verify the entity sets of each category according to the subject and the keywords with probability greater than a set value under the subject, and correct the entities in the set that do not match the subject and the keywords.

10. The deep learning-based entity labeling and recognition device according to claim 9, characterized in that: The device also includes a topic model training module for setting model parameters and training the LDA model; the model parameters include the number of topics, the number of iterations, the learning rate, and ,in, represents the text-topic density, Indicates topic-word density; The specific training process of the topic model training module includes: for each topic, randomly initialize a word distribution; for each word in each text, calculate the probability that the word belongs to each topic; based on the calculated probability, update the text-topic distribution and topic-word distribution; for each text, update the topic distribution of the text based on the current topic-word distribution; for each topic, update the word distribution of the topic based on the current text-topic distribution; until the model converges or reaches the preset number of iterations, a trained LDA model is obtained.

Images