Text content-oriented new label entity recognition method, device, equipment and medium

Abstract

This application relates to a method, apparatus, device, and medium for new tag entity recognition of text content. The method includes: retraining a pre-trained model using a training dataset to construct a full-word masking language model task and an NTP task; constructing a candidate entity recognition model based on the retrained model and GlobalPointer; using the candidate entity recognition model to recognize new tags on an information dataset; ranking the recognized new tags to obtain the entity tags with the highest relevance to the article; filtering a manually annotated tag lexicon based on the entity tags to obtain a new tag lexicon; cleaning the new tag lexicon; modifying and expanding the training dataset using the cleaned tag lexicon; training the candidate entity recognition model using the expanded training set; and performing new tag entity recognition on text content based on the trained entity recognition model. This method can improve the accuracy of new tag entity recognition.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a new tag entity recognition method, device, computer device, and storage medium for text content. Background Technique

[0002] For a tag system, the perfection of the tag library is crucial. Here, the tag system refers to outputting a certain number of summarizable words (keywords) or categorized classes for the input information carrier (here, an article). These classes are called tags, and the keywords that can summarize tags are called feature words. The system that assigns tags is called a tag system. However, the discovery of tag words and feature words cannot rely solely on manual accumulation and discovery, so automatic tag discovery is required technically.

[0003] Tag discovery is essentially close to the problem of new word discovery in Chinese natural language processing, but it cannot be completely equated with new word discovery because some tag words may not be new words but existing words. In traditional methods, new word discovery based on mutual information is commonly performed using the statistical method mentioned by Matrix67 for unsupervised new word discovery. Ngram (n-ary word segmentation) is used for vocabulary recall, that is, full-scale enumeration is performed based on n-ary word segmentation, and the current vocabulary is scored by calculating the internal cohesion (PMI) and the freedom degree (left and right entropy) of the vocabulary. The scores are sorted for recall. Among them, the internal cohesion refers to the probability of the characters in the current vocabulary appearing together, representing that these characters often appear as a whole and are more likely to be a word, while the freedom degree of the vocabulary refers to whether the characters on the left and right sides of the group formed by these characters are rich enough. For example, the group formed by the three characters "bottle of soy sauce" appears frequently enough, that is, its internal cohesion is high enough, but the richness of the characters appearing on the left side of this group is very low, and only characters such as "a few", "one", "two", etc. often appear, indicating that the richness of the characters appearing on the left side of the group formed by these three characters is not high enough, that is, the freedom degree of this vocabulary is not high enough. Therefore, after weighing the internal cohesion and external freedom degree of the current vocabulary, the scoring score of this vocabulary is obtained. The scores can be sorted from high to low to screen out more likely new words, and then the existing words are filtered out to obtain the final result.

[0004] However, current novel label discovery methods based on unsupervised corpora have the following drawbacks: they rely heavily on large amounts of text input because they are based on statistical methods, requiring enumeration and frequency calculation of the entire input data to determine lexical cohesion and degrees of freedom. Therefore, they fail to yield effective results when the input is a single article, and struggle to discover words (labels) that are too long or contain English characters. This is because word recall is based on enumeration using n-gram segmentation; if n is too large, efficiency suffers. Since this method involves full recall followed by sorting and filtering, it essentially removes erroneous candidate words. This makes it highly dependent on the completeness of the existing vocabulary, and the proportion of usable words in the results is very low; typically, even the top 100 results have a usability rate of less than 50%. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, computer device, and storage medium for text-based new tag entity recognition that can improve the accuracy of new tag entity recognition, addressing the aforementioned technical problems.

[0006] A novel tag entity recognition method for text content, the method comprising:

[0007] Obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags;

[0008] The training dataset is obtained by filtering the information text based on the tag vocabulary.

[0009] The pre-built BERT model is trained using the training dataset and a self-supervised approach to obtain a pre-trained model;

[0010] The pre-trained model was retrained by constructing a full-word masking language model task and an NTP task using the training dataset to obtain a retrained model.

[0011] A candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer;

[0012] A candidate entity recognition model is used to identify new labels in an information dataset. The identified new labels are then ranked to obtain the entity labels with the highest relevance to the article.

[0013] A new tag vocabulary is obtained by filtering the manually labeled tag vocabulary based on entity tags.

[0014] The new tag library is cleaned based on the number of feature words corresponding to the tags in the new tag library, the number of times the feature words are filtered, the number of days the feature words are added to the new tag library, and the number of times the tags are filtered within a preset time period, to obtain the cleaned tag library.

[0015] The training dataset is modified and expanded using the cleaned label library to obtain an expanded training set. The candidate entity recognition model is then trained using the expanded training set to obtain a trained entity recognition model.

[0016] The text content is used to perform new label entity recognition based on the trained entity recognition model.

[0017] In one embodiment, the sentence containing all tags in the new tag library is encoded according to the BERT model, the word vectors at the corresponding positions of the tags are concatenated in these four layers, and then average pooling is performed to obtain the word vectors of all tags.

[0018] The Faiss index is used to store the word vectors of all tags. After the entity recognition result of the new tag is vectorized in the Faiss index, the cosine similarity is calculated with all tags in the new tag library, and the two tags with the highest scores are returned as the first candidate synonym tags.

[0019] All tags in the new tag library are constructed into a bkTree based on the edit distance. After normalizing the entity recognition results of the new tags, multiple tags with an edit distance of less than 2 are searched in the bkTree as second candidate synonym tags.

[0020] The new tag word library is expanded by locating the new tag entity recognition results to their positions in the tag library based on the first and second candidate synonym tags.

[0021] In one embodiment, the process of constructing the masking language model task and the NTP task includes:

[0022] Random full-word masking is performed on the training dataset, and the masked text is segmented using the ansj word segmenter to obtain a word segmentation lexicon; full words refer to complete words in Chinese; a masking language model task is constructed based on the word segmentation lexicon and the tag lexicon.

[0023] Vector extraction is performed on sentences containing tags to obtain vector representations of the tags. The hierarchical relationship between tags and feature words in the tag lexicon is used as the NTP task for the tag lexicon.

[0024] In one embodiment, a candidate entity recognition model is used to identify new labels on the information dataset, and the identified new labels are ranked to obtain the entity labels with the highest article relevance, including:

[0025] A candidate entity recognition model is used to identify new labels in the information dataset, and the text in the information dataset is represented by full-text vectors to obtain the first vector representation.

[0026] The text is masked before being represented by vectors to obtain the second vector representation;

[0027] Cosine similarity is calculated on the first and second vector representations to obtain the degree of association between articles with and without the current tag; the degree of association between articles is represented by the importance of the tag.

[0028] Based on the importance of the tags, all tags in the information dataset are sorted from smallest to largest to obtain the entity tags with the highest relevance to the article.

[0029] In one embodiment, the new tag lexicon is cleaned based on the number of feature words corresponding to the tags in the new tag lexicon, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tags are filtered within a preset time period, to obtain a cleaned tag lexicon, including:

[0030] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0031] If the timeliness value of a feature word is less than 0 within a preset three-times period, then the feature word and its corresponding tag are deleted, resulting in a cleaned tag library.

[0032] In one embodiment, the timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time period. The timeliness value of the feature words is then obtained, including:

[0033] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0034]

[0035] Where i represents a feature word, T i M represents the label corresponding to the feature word. i D represents the number of times a feature word is filtered within a pre-set time period. i This indicates the number of days since the feature words were added to the new tag vocabulary. T represents i The corresponding number of feature words T represents i The number of times it is filtered within a preset time period.

[0036] In one embodiment, the information text is filtered according to a tag lexicon to obtain a training dataset, including:

[0037] Sentences containing words from a tag lexicon in the news text are stripped of their contents, and the resulting sentences and the corresponding words from the tag lexicon are combined to form a training dataset.

[0038] In one embodiment, a pre-built BERT model is trained using a training dataset and a self-supervised approach to obtain a pre-trained model, including:

[0039] The words in the sentences of the training dataset are masked, and the BERT model is used to perform cloze test training on the masked training dataset to obtain the trained BERT model.

[0040] The order of sentences in the training dataset is shuffled, and the pre-trained BERT model is used to sort the sentences in the shuffled training dataset to obtain a pre-trained model.

[0041] A novel tag entity recognition device for text content, the device comprising:

[0042] The training dataset construction module is used to obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags; the news text is filtered based on the tag lexicon to obtain the training dataset;

[0043] The model training module is used to train a pre-built BERT model using the training dataset and a self-supervised approach to obtain a pre-trained model; and to retrain the pre-trained model using the training dataset to construct a full-word masking language model task and an NTP task to obtain a retrained model.

[0044] The candidate entity recognition model building module is used to build a candidate entity recognition model based on the retrained model and the GlobalPointer global pointer.

[0045] The tag lexicon filtering module is used to identify new tags in the information dataset using a candidate entity recognition model, sort the identified new tags, and obtain the entity tags with the highest relevance to the article; it also filters the manually annotated tag lexicon based on the entity tags to obtain a new tag lexicon.

[0046] The new tag terminology cleaning module is used to clean the new tag terminology based on the number of feature words corresponding to the tags in the new tag terminology, the number of times the feature words are filtered, the number of days the feature words are added to the new tag terminology, and the number of times the tags are filtered within a preset time period, so as to obtain a cleaned tag terminology.

[0047] The new label entity recognition module is used to modify and expand the training dataset using the cleaned label library to obtain an expanded training set. The expanded training set is then used to train the candidate entity recognition model to obtain a trained entity recognition model. Based on the trained entity recognition model, new label entity recognition is performed on the text content.

[0048] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0049] Obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags;

[0050] The training dataset is obtained by filtering the information text based on the tag vocabulary.

[0051] The pre-built BERT model is trained using the training dataset and a self-supervised approach to obtain a pre-trained model;

[0052] The pre-trained model was retrained by constructing a full-word masking language model task and an NTP task using the training dataset to obtain a retrained model.

[0053] A candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer;

[0054] A candidate entity recognition model is used to identify new labels in an information dataset. The identified new labels are then ranked to obtain the entity labels with the highest relevance to the article.

[0055] A new tag vocabulary is obtained by filtering the manually labeled tag vocabulary based on entity tags.

[0056] The new tag library is cleaned based on the number of feature words corresponding to the tags in the new tag library, the number of times the feature words are filtered, the number of days the feature words are added to the new tag library, and the number of times the tags are filtered within a preset time period, to obtain the cleaned tag library.

[0057] The training dataset is modified and expanded using the cleaned label library to obtain an expanded training set. The candidate entity recognition model is then trained using the expanded training set to obtain a trained entity recognition model.

[0058] The text content is used to perform new label entity recognition based on the trained entity recognition model.

[0059] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0060] Obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags;

[0061] The training dataset is obtained by filtering the information text based on the tag vocabulary.

[0062] The pre-built BERT model is trained using the training dataset and a self-supervised approach to obtain a pre-trained model;

[0063] The pre-trained model was retrained by constructing a full-word masking language model task and an NTP task using the training dataset to obtain a retrained model.

[0064] A candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer;

[0065] A candidate entity recognition model is used to identify new labels in an information dataset. The identified new labels are then ranked to obtain the entity labels with the highest relevance to the article.

[0066] A new tag vocabulary is obtained by filtering the manually labeled tag vocabulary based on entity tags.

[0067] The new tag library is cleaned based on the number of feature words corresponding to the tags in the new tag library, the number of times the feature words are filtered, the number of days the feature words are added to the new tag library, and the number of times the tags are filtered within a preset time period, to obtain the cleaned tag library.

[0068] The training dataset is modified and expanded using the cleaned label library to obtain an expanded training set. The candidate entity recognition model is then trained using the expanded training set to obtain a trained entity recognition model.

[0069] The text content is used to perform new label entity recognition based on the trained entity recognition model.

[0070] The aforementioned method, apparatus, computer equipment, and storage medium for new label entity recognition based on text content first filters the information text according to a label lexicon to obtain a training dataset. A pre-built BERT model is then trained using this training dataset and a self-supervised approach. This allows the pre-trained model to learn the basic text features of the training dataset, facilitating subsequent label entity recognition. The pre-trained model is then retrained using a full-word masking language model task and an NTP task, resulting in a retrained model. This retraining process further enhances the model's ability to learn existing label knowledge and the correlation between labels and text, improving recognition accuracy during subsequent text entity recognition. Finally, a candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer. This candidate entity recognition model is then used to perform new label recognition on the information dataset, improving the accuracy of the recognition process. Other new tags are sorted to obtain the entity tags with the highest relevance to the article. The manually annotated tag lexicon is then filtered based on these entity tags to obtain a new tag lexicon. The discovered results are deduplicated from the manually annotated tag lexicon to prevent duplicate results. The new tag lexicon is cleaned based on the number of feature words corresponding to each tag, the number of times the feature words have been filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tag has been filtered within a pre-set timeframe. The tag lexicon is cleaned periodically to remove unnecessary tags and feature words, thereby improving its accuracy and timeliness. This allows it to be updated in subsequent model training and entity recognition, improving tag recognition accuracy. The expanded training set is used to train the candidate entity recognition model. At this point, the old entity recognition model can be upgraded through repeated training data preparation. This iterative tag discovery method can further improve the accuracy of new tag entity recognition. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a novel tag entity recognition method for text content in one embodiment;

[0072] Figure 2 This is a schematic diagram of the synonym recall process in one embodiment;

[0073] Figure 3 This is a schematic diagram of the initial entity recognition model training process in one embodiment;

[0074] Figure 4 This is a schematic diagram illustrating the process of using a candidate entity recognition model to identify new labels for an information dataset in one embodiment.

[0075] Figure 5This is a structural block diagram of a novel tag entity recognition device for text content in one embodiment;

[0076] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0078] In one embodiment, such as Figure 1 As shown, a novel tag entity recognition method for text content is provided, including the following steps:

[0079] Step 102: Obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags; filter the news text based on the tag lexicon to obtain the training dataset.

[0080] Since the input to the entity recognition model is text, and the input consists of entities in the text that may be new labels, the training data also needs to prepare such input and output. A label lexicon and a portion of information text data are collected manually in advance. The words in the label lexicon are used to search in these text data to extract sentences containing these words from the text. For example, "Shanghai Stock Exchange A-shares" is a word in the label lexicon. The sentence containing "Shanghai Stock Exchange A-shares" is found to be "XXX Shanghai Stock Exchange A-shares XXXXX". Then, this sentence and the range of the subscripts of "Shanghai Stock Exchange A-shares" in this sentence are used to construct a training data, such as ["XXX Shanghai Stock Exchange A-shares XXXXX", (4,8)], which is a training data and is used to construct the training dataset.

[0081] Step 104: Train the pre-built BERT model using the training dataset and a self-supervised approach to obtain the pre-trained model; then, use the training dataset to construct a full-word masking language model task and an NTP task to retrain the pre-trained model to obtain the retrained model.

[0082] Using the training dataset, a pre-built BERT model is trained through self-supervised training. There are generally two methods of self-supervised training: one is to mask words in sentences within the text, making the model perform a cloze test-like task; the other is to shuffle the sentence order, making the model perform a sentence sorting-like task. Training the model through these two self-supervised methods allows the pre-trained model to learn the basic text features of the training dataset, facilitating subsequent label entity recognition. The pre-trained model is then retrained using a full-word masking language model task and an NTP task, resulting in a retrained model. This retraining process allows the model to better learn existing label knowledge and the correlation between labels and the text, improving the accuracy of subsequent text entity recognition.

[0083] Step 106: Construct a candidate entity recognition model based on the retrained model and the GlobalPointer global pointer.

[0084] Using the GlobalPointer pointer as a decoder can solve the problem of nested labels, recognize multiple labels, constrain the output of the pre-trained model, and improve the rationality and accuracy of the results. Because the results of neural networks are random, using the GlobalPointer pointer can reduce the probability of very unreasonable results, thereby improving the recognition accuracy of the candidate entity recognition model.

[0085] Step 108: Use the candidate entity recognition model to identify new tags in the information dataset, sort the identified new tags to obtain the entity tags with the highest relevance to the article, and filter the manually annotated tag lexicon based on the entity tags to obtain a new tag lexicon.

[0086] A candidate entity recognition model is used to identify new labels on an information dataset. The model is treated as a black box; a new piece of information is input, and after passing through the black box, results such as "Shanghai Stock Exchange A-shares," "medical stocks," etc., are obtained. These results will inevitably contain some dirty data, i.e., invalid labels. However, when there is a large amount of input data, the results can be statistically analyzed, and labels with high frequency can be prioritized to improve the accuracy of label discovery. Then, the discovered results are compared with a manually labeled label lexicon to remove duplicates and prevent duplicate results.

[0087] Step 110: Clean the new tag library based on the number of feature words corresponding to the tags in the new tag library, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag library, and the number of times the tags are filtered within a preset time period, to obtain the cleaned tag library.

[0088] As new labels are constantly discovered, the label library expands continuously, which reduces the efficiency of entity recognition models when performing label recognition. Considering the time-sensitivity of some label words and the efficiency of discovering new labels, it is necessary to clean the label library regularly, removing some unnecessary label words and feature words, thereby improving the accuracy and timeliness of the label library. This allows the library to keep pace with the times in subsequent model training and entity recognition, thus improving the accuracy of label recognition.

[0089] Step 112: Modify and expand the training dataset using the cleaned tag library to obtain an expanded training set. Use the expanded training set to train the candidate entity recognition model to obtain a trained entity recognition model. Perform new tag entity recognition on the text content based on the trained entity recognition model.

[0090] By training the candidate entity recognition model using the expanded training set, the old entity recognition model can be upgraded through repeated training data preparation. This iterative approach to label discovery can further improve the accuracy of new label entity recognition.

[0091] The aforementioned text-based novel tag entity recognition method first filters the news text data according to a tag lexicon to obtain a training dataset. A pre-built BERT model is then trained using this training dataset and a self-supervised approach. This allows the pre-trained model to learn the basic text features of the training dataset, facilitating subsequent tag entity recognition. The pre-trained model is then retrained using a full-word masking language model task and an NTP task, resulting in a retrained model. This retraining process further enhances the model's ability to learn existing tag knowledge and the correlation between tags and the text, improving recognition accuracy during subsequent text entity recognition. Finally, a candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer. This candidate entity recognition model is then used to identify new tags on the news dataset. The identified new tags are then processed... The results are sorted to obtain the entity tags with the highest relevance to the articles. A new tag lexicon is then obtained by filtering the manually annotated tag lexicon based on these entity tags. The discovered results are deduplicated from the manually annotated tag lexicon to prevent duplicate results. The new tag lexicon is then cleaned based on the number of feature words corresponding to each tag, the number of times the feature words have been filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tags have been filtered within a pre-set timeframe. Regular cleaning of the tag lexicon removes unnecessary tags and feature words, thereby improving the accuracy and timeliness of the tag lexicon. This allows it to be updated in subsequent model training and entity recognition, improving the accuracy of tag recognition. The expanded training set is used to train the candidate entity recognition model. At this point, the old entity recognition model can be upgraded by retraining through repeated training data preparation. This iterative approach to tag discovery further improves the accuracy of new tag entity recognition.

[0092] In one embodiment, the sentence containing all tags in the new tag library is encoded according to the BERT model, the word vectors at the corresponding positions of the tags are concatenated in these four layers, and then average pooling is performed to obtain the word vectors of all tags.

[0093] The Faiss index is used to store the word vectors of all tags. After the entity recognition result of the new tag is vectorized in the Faiss index, the cosine similarity is calculated with all tags in the new tag library, and the two tags with the highest scores are returned as the first candidate synonym tags.

[0094] All tags in the new tag library are constructed into a bkTree based on the edit distance. After normalizing the entity recognition results of the new tags, multiple tags with an edit distance of less than 2 are searched in the bkTree as second candidate synonym tags.

[0095] The new tag word library is expanded by locating the new tag entity recognition results to their positions in the tag library based on the first and second candidate synonym tags.

[0096] In a specific embodiment, such as Figure 2 As shown, firstly, word vectors are calculated for all tags in the tag library. The word vectors are calculated by encoding each tag within a given sentence using BERT. The encoding result is the last four layers of character vectors. The character vectors corresponding to the tag's position are then concatenated across these four layers, followed by average pooling. All resulting word vectors are stored in Faiss. Next, a Bktree is constructed. All tags in the tag library are added to the Bktree based on their edit distance. Finally, after inputting a word (including context), the context is input into BERT, and the word vector is added to Faiss for similarity retrieval, yielding two results. The word is then input into the Bktree for edit distance retrieval, yielding two more results. These four results are merged and returned as the final result for synonym recall. Faiss (Facebook AI SimilaritySearch) is an open-source clustering and similarity search library from the Facebook AI team. It provides efficient similarity search and clustering for dense vectors, supports searches of billions of vectors, and is currently a mature approximate nearest neighbor search library. A BK Tree, or Burkhard Keller Tree, is a data structure used to perform spell checking based on the Levenshtein distance concept. Like all other trees, a BK Tree consists of nodes and edges. Each node in a BK Tree represents a single word from our dictionary, and the number of nodes is exactly the same as the number of words in the dictionary. Edges contain integer weights that tell us the edit distance from one node to another. Using this method, the time complexity of finding similar words can be reduced from the n-squared level to the log n level.

[0097] The process of discovering new tags involves not only mining usable tag words and feature words from the news text, but also adding these words to the tag library. However, adding new tag words to the tag library requires a lot of time for word meaning queries. To improve efficiency, a synonym retrieval model can be used to match the discovered new tag words with words in the tag library based on their similarity, and return words with high matching degree (for example, if the new tag word "Beijing Stock Exchange" is discovered, and the synonym "Shanghai Stock Exchange" is retrieved, the new tag word can be quickly located in the tag library, thus realizing the addition of the new tag).

[0098] In one embodiment, the process of constructing the masking language model task and the NTP task includes:

[0099] Random full-word masking is performed on the training dataset, and the masked text is segmented using the ansj word segmenter to obtain a word segmentation lexicon; full words refer to complete words in Chinese; a masking language model task is constructed based on the word segmentation lexicon and the tag lexicon.

[0100] Vector extraction is performed on sentences containing tags to obtain vector representations of the tags. The hierarchical relationship between tags and feature words in the tag lexicon is used as the NTP task for the tag lexicon.

[0101] In a specific embodiment, such as Figure 3 As shown, BERT uses the retraining results of bert-wwm-chinese and the financial domain data and label data structure in the training dataset. Retraining employs multi-task training, with the total training loss being the sum of the losses from the two tasks. The first task uses a masked language model, employing random whole word masking. A whole word refers to a complete word in Chinese. Due to the differences between Chinese and English, Chinese words refer to multiple characters, while English words refer to single words. Therefore, using whole word masking allows the model to learn more lexical information. The ansj word segmenter is then used, incorporating a label lexicon as the segmentation vocabulary. Using a label lexicon accumulated over many years allows the model to better learn existing label knowledge. The second task innovatively utilizes information labeling. It extracts the vector representation of labels from sentences containing labels and then uses the hierarchical relationship between labels and feature words in the label lexicon to perform a label-to-vocabulary prediction task (NTP next tag prediction). By constructing a masked language model task and an NTP task to retrain the initial entity recognition model, the model can better learn existing label knowledge and the degree of association between labels and articles, which is beneficial to improving the recognition accuracy when performing entity recognition on text in the future.

[0102] In one embodiment, a candidate entity recognition model is used to identify new labels on the information dataset, and the identified new labels are ranked to obtain the entity labels with the highest article relevance, including:

[0103] A candidate entity recognition model is used to identify new labels in the information dataset, and the text in the information dataset is represented by full-text vectors to obtain the first vector representation.

[0104] The text is masked before being represented by vectors to obtain the second vector representation;

[0105] Cosine similarity is calculated on the first and second vector representations to obtain the degree of association between articles with and without the current tag; the degree of association between articles is represented by the importance of the tag.

[0106] Based on the importance of the tags, all tags in the information dataset are sorted from smallest to largest to obtain the entity tags with the highest relevance to the article.

[0107] In a specific embodiment, such as Figure 4 As shown, a candidate entity recognition model is used to identify new labels on an information dataset. First, the current information is represented by a full-text embedding (vector). Then, all words with the current label in the full text are masked before being represented by embedding (vector) again. The cosine similarity between these two embeddings is then calculated. The result represents the degree of association between the article with and without the current label, thus indicating the importance of the label. After performing the above processing on all new labels of the current information, the results for each new label are obtained. These results are then sorted in ascending order to obtain the entity label with the highest relevance to the current information article.

[0108] In one embodiment, the new tag lexicon is cleaned based on the number of feature words corresponding to the tags in the new tag lexicon, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tags are filtered within a preset time period, to obtain a cleaned tag lexicon, including:

[0109] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0110] If the timeliness value of a feature word is less than 0 within a preset three-times period, then the feature word and its corresponding tag are deleted, resulting in a cleaned tag library.

[0111] In one embodiment, the timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time period. The timeliness value of the feature words is then obtained, including:

[0112] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0113]

[0114] Where i represents a feature word, T i M represents the label corresponding to the feature word. i D represents the number of times a feature word is filtered within a pre-set time period. i This indicates the number of days since the feature words were added to the new tag vocabulary. T represents i The corresponding number of feature words T represents i The number of times it is filtered within a preset time period.

[0115] In a specific implementation, the time period can be set according to actual needs. For example, if the financial market has been very popular recently and new words appear frequently, the time period can be set to one month for tag word filtering.

[0116] In one embodiment, the information text is filtered according to a tag lexicon to obtain a training dataset, including:

[0117] Sentences containing words from a tag lexicon in the news text are stripped of their contents, and the resulting sentences and the corresponding words from the tag lexicon are combined to form a training dataset.

[0118] In one embodiment, a pre-built BERT model is trained using a training dataset and a self-supervised approach to obtain a pre-trained model, including:

[0119] The words in the sentences of the training dataset are masked, and the BERT model is used to perform cloze test training on the masked training dataset to obtain the trained BERT model.

[0120] The order of sentences in the training dataset is shuffled, and the pre-trained BERT model is used to sort the sentences in the shuffled training dataset to obtain a pre-trained model.

[0121] In specific implementations, a pre-built BERT model is trained by masking words in sentences within the text, allowing the model to perform a cloze test-like task. Another approach involves scrambling the sentence order in the text, making the model perform a sentence sorting task, similar to teaching a 10-year-old a problem versus teaching a 0-year-old. The trained model significantly improves the BERT model's recognition efficiency and accuracy.

[0122] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order requirement for the execution of these steps; they can be executed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0123] In one embodiment, such as Figure 5 As shown, a novel tag entity recognition device for text content is provided, comprising: a training dataset construction module 502, a model training module 504, a candidate entity recognition model construction module 506, a tag lexicon filtering module 508, a novel tag lexicon cleaning module 510, and a novel tag entity recognition module 512, wherein:

[0124] The training dataset construction module 502 is used to obtain manually annotated label lexicon, news text, and news dataset; the label lexicon contains feature words and their corresponding labels; the news text is filtered according to the label lexicon to obtain the training dataset;

[0125] The model training module 504 is used to train a pre-built BERT model using the training dataset and a self-supervised approach to obtain a pre-trained model; and to retrain the pre-trained model using the training dataset to construct a full-word masking language model task and an NTP task to obtain a retrained model.

[0126] Candidate entity recognition model building module 506 is used to build a candidate entity recognition model based on the retrained model and the GlobalPointer global pointer;

[0127] The tag word filtering module 508 is used to identify new tags in the information dataset using a candidate entity recognition model, sort the identified new tags, and obtain the entity tags with the highest relevance to the article; and to filter the manually annotated tag word library based on the entity tags to obtain a new tag word library.

[0128] The new tag terminology cleaning module 510 is used to clean the new tag terminology based on the number of feature words corresponding to the tags in the new tag terminology, the number of times the feature words are filtered, the number of days the feature words are added to the new tag terminology, and the number of times the tags are filtered within a preset time, so as to obtain a cleaned tag terminology.

[0129] The new label entity recognition module 512 is used to modify and expand the training dataset using the cleaned label library to obtain an expanded training set, and to train the candidate entity recognition model using the expanded training set to obtain a trained entity recognition model; and to perform new label entity recognition on the text content based on the trained entity recognition model.

[0130] In one embodiment, the sentence containing all tags in the new tag library is encoded according to the BERT model, the word vectors at the corresponding positions of the tags are concatenated in these four layers, and then average pooling is performed to obtain the word vectors of all tags.

[0131] The Faiss index is used to store the word vectors of all tags. After the entity recognition result of the new tag is vectorized in the Faiss index, the cosine similarity is calculated with all tags in the new tag library, and the two tags with the highest scores are returned as the first candidate synonym tags.

[0132] All tags in the new tag library are constructed into a bkTree based on the edit distance. After normalizing the entity recognition results of the new tags, multiple tags with an edit distance of less than 2 are searched in the bkTree as second candidate synonym tags.

[0133] The new tag word library is expanded by locating the new tag entity recognition results to their positions in the tag library based on the first and second candidate synonym tags.

[0134] In one embodiment, the process of constructing the masking language model task and the NTP task includes:

[0135] Random full-word masking is performed on the training dataset, and the masked text is segmented using the ansj word segmenter to obtain a word segmentation lexicon; full words refer to complete words in Chinese; a masking language model task is constructed based on the word segmentation lexicon and the tag lexicon.

[0136] Vector extraction is performed on sentences containing tags to obtain vector representations of the tags. The hierarchical relationship between tags and feature words in the tag lexicon is used as the NTP task for the tag lexicon.

[0137] In one embodiment, the tag lexicon filtering module 508 is further configured to use a candidate entity recognition model to identify new tags in the information dataset, rank the identified new tags, and obtain the entity tags with the highest article relevance, including:

[0138] A candidate entity recognition model is used to identify new labels in the information dataset, and the text in the information dataset is represented by full-text vectors to obtain the first vector representation.

[0139] The text is masked before being represented by vectors to obtain the second vector representation;

[0140] Cosine similarity is calculated on the first and second vector representations to obtain the degree of association between articles with and without the current tag; the degree of association between articles is represented by the importance of the tag.

[0141] Based on the importance of the tags, all tags in the information dataset are sorted from smallest to largest to obtain the entity tags with the highest relevance to the article.

[0142] In one embodiment, the new tag lexicon cleaning module 510 is further configured to clean the new tag lexicon based on the number of feature words corresponding to tags in the new tag lexicon, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tags are filtered within a preset time period, to obtain a cleaned tag lexicon, including:

[0143] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0144] If the timeliness value of a feature word is less than 0 within a preset three-times period, then the feature word and its corresponding tag are deleted, resulting in a cleaned tag library.

[0145] In one embodiment, the new tag terminology cleaning module 510 is further configured to calculate the timeliness value of the feature words in the new tag terminology based on the number of feature words corresponding to the tags in the new tag terminology, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag terminology, and the number of times the tags are filtered within a preset time period, to obtain the timeliness value of the feature words, including:

[0146] The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time.

[0147]

[0148] Where i represents a feature word, T i M represents the label corresponding to the feature word. i D represents the number of times a feature word is filtered within a pre-set time period. i This indicates the number of days since the feature words were added to the new tag vocabulary. T represents i The corresponding number of feature words T represents i The number of times it is filtered within a preset time period.

[0149] In one embodiment, the training dataset construction module 502 is further configured to perform data filtering on the information text based on a tag lexicon to obtain a training dataset, including:

[0150] Sentences containing words from a tag lexicon in the news text are stripped of their contents, and the resulting sentences and the corresponding words from the tag lexicon are combined to form a training dataset.

[0151] In one embodiment, the pre-trained model training module 504 is further configured to train a pre-built BERT model using a training dataset and a self-supervised approach to obtain a pre-trained model, including:

[0152] The words in the sentences of the training dataset are masked, and the BERT model is used to perform cloze test training on the masked training dataset to obtain the trained BERT model.

[0153] The order of sentences in the training dataset is shuffled, and the pre-trained BERT model is used to sort the sentences in the shuffled training dataset to obtain a pre-trained model.

[0154] Specific limitations regarding the text-content-oriented new tag entity recognition device can be found in the limitations of the text-content-oriented new tag entity recognition method described above, and will not be repeated here. Each module in the aforementioned text-content-oriented new tag entity recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0155] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a novel tag entity recognition method for text content. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0156] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0158] In one embodiment, a computer storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the method described above.

[0159] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0160] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0161] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A novel tag entity recognition method for text content, characterized in that, The method includes: Obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags; The information text is filtered according to the tag vocabulary to obtain a training dataset; The pre-built BERT model is trained using the training dataset and a self-supervised approach to obtain a pre-trained model; The pre-trained model was retrained using the training dataset to construct a full-word masking language model task and an NTP task, resulting in a retrained model. A candidate entity recognition model is constructed based on the retrained model and the GlobalPointer global pointer. The candidate entity recognition model is used to identify new tags in the information dataset. The identified new tags are then sorted to obtain the entity tags with the highest article relevance. The manually labeled tag lexicon is filtered based on the entity tags to obtain a new tag lexicon. The new tag library is cleaned based on the number of feature words corresponding to the tags in the new tag library, the number of times the feature words are filtered, the number of days the feature words are added to the new tag library, and the number of times the tags are filtered within a preset time period, to obtain a cleaned tag library. The training dataset is modified and expanded using the cleaned tag library to obtain an expanded training set. The candidate entity recognition model is then trained using the expanded training set to obtain a trained entity recognition model. The trained entity recognition model is used to perform new label entity recognition on the text content.

2. The method according to claim 1, characterized in that, The method further includes: The sentence containing all tags in the new tag library is encoded according to the BERT model. The word vectors at the corresponding positions of the tags are concatenated in these four layers and then average pooled to obtain the word vectors of all tags. The word vectors of all the tags are stored using the Faiss index. After the new tag entity recognition result is vectorized in the Faiss index, the cosine similarity is calculated with all the tags in the new tag library, and the two tags with the highest scores are returned as the first candidate synonym tags. All tags in the new tag library are constructed into a bkTree based on the edit distance. After the new tag entity recognition results are normalized, multiple tags with an edit distance of less than 2 are searched in the bkTree as second candidate synonym tags. The new tag word library is expanded by locating the new tag entity recognition result in the tag library according to the first candidate synonym tag and the second candidate synonym tag.

3. The method according to claim 1, characterized in that, The process of constructing the masking language model task and the NTP task includes: Random full-word masking is performed on the training dataset, and the masked text is segmented using the ansj word segmenter to obtain a word segmentation lexicon; the full words refer to complete words in Chinese; a masking language model task is constructed based on the word segmentation lexicon and the tag lexicon. Vector extraction is performed on sentences containing tags to obtain vector representations of the tags. The hierarchical relationship between tags and feature words in the tag lexicon is used as the NTP task for the tag lexicon.

4. The method according to claim 1, characterized in that, The candidate entity recognition model is used to identify new tags in the information dataset. The identified new tags are then ranked to obtain the entity tags with the highest article relevance, including: The candidate entity recognition model is used to identify new labels in the information dataset, and the text in the information dataset is represented by full-text vectors to obtain the first vector representation. The text is masked before vector representation is performed to obtain the second vector representation. Cosine similarity is calculated on the first vector representation and the second vector representation to obtain the degree of association between articles with and without the current tag; the degree of association between the articles is represented by the importance of the tag. Based on the importance of the tags, all tags in the information dataset are sorted from smallest to largest to obtain the entity tags with the highest relevance to the article.

5. The method according to claim 1, characterized in that, The new tag lexicon is cleaned based on the number of feature words corresponding to the tags in the new tag lexicon, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag lexicon, and the number of times the tags are filtered within a preset time period, to obtain a cleaned tag lexicon, including: The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words are added to the new tag thesaurus, and the number of times the tags are filtered within a preset time. If the timeliness value of the feature word is less than 0 within a preset three-times period, then the feature word and its corresponding tag are deleted to obtain a cleaned tag library.

6. The method according to claim 5, characterized in that, The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time. This timeliness value includes: The timeliness value of the feature words in the new tag thesaurus is calculated based on the number of feature words corresponding to the tags in the new tag thesaurus, the number of times the feature words are filtered, the number of days the feature words have been added to the new tag thesaurus, and the number of times the tags are filtered within a preset time. Where i represents a feature word, T i M represents the label corresponding to the feature word. i D represents the number of times a feature word is filtered within a pre-set time period. i This indicates the number of days since the feature words were added to the new tag vocabulary. T represents i The corresponding number of feature words T represents i The number of times it is filtered within a preset time period.

7. The method according to claim 1, characterized in that, The information text is filtered according to the aforementioned tag lexicon to obtain a training dataset, including: Sentences containing words from the tag lexicon in the information text are stripped, and the resulting sentences and the corresponding words from the tag lexicon are combined to form a training dataset.

8. The method according to claim 1, characterized in that, The pre-built BERT model is trained using the aforementioned training dataset and a self-supervised approach to obtain a pre-trained model, including: The words in the sentences of the training dataset are masked, and the BERT model is used to perform cloze test training on the masked training dataset to obtain the trained BERT model. The order of sentences in the training dataset is shuffled, and the trained BERT model is used to sort the sentences in the shuffled training dataset to obtain a pre-trained model.

9. A novel tag entity recognition device for text content, characterized in that, The device includes: The training dataset construction module is used to obtain manually annotated tag lexicon, news text, and news dataset; the tag lexicon contains feature words and their corresponding tags; the news text is filtered according to the tag lexicon to obtain the training dataset; The model training module is used to train the pre-built BERT model using the training dataset and a self-supervised approach to obtain a pre-trained model; and to retrain the pre-trained model using the training dataset to construct a full-word masking language model task and an NTP task to obtain a retrained model. The candidate entity recognition model construction module is used to construct a candidate entity recognition model based on the retrained model and the GlobalPointer global pointer. The tag word filtering module is used to identify new tags in the information dataset using the candidate entity recognition model, sort the identified new tags, and obtain the entity tags with the highest relevance to the article; and to filter the manually annotated tag word library based on the entity tags to obtain a new tag word library. The new tag lexicon cleaning module is used to clean the new tag lexicon based on the number of feature words corresponding to the tags in the new tag lexicon, the number of times the feature words are filtered, the number of days the feature words are added to the new tag lexicon, and the number of times the tags are filtered within a preset time, so as to obtain a cleaned tag lexicon. The new label entity recognition module is used to modify and expand the training dataset using the cleaned label library to obtain an expanded training set, and to train the candidate entity recognition model using the expanded training set to obtain a trained entity recognition model; and to perform new label entity recognition on the text content based on the trained entity recognition model.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images