Entity identification method based on semantic analysis and label visual feature extension

Abstract

The invention belongs to the technical field of information extraction, and provides an entity recognition method based on semantic analysis and label visual feature extension. The problem that an entity boundary is difficult to determine is solved by calculating the similarity between candidate entities and text global semantics and extracting interaction features of the candidate entities and the text global semantics to implicitly construct embedded representation containing sentence structure information, visual features are systematically introduced into an entity recognition task, and a text-visual cross-modal entity recognition normal form is constructed. Context semantic information represented by embedding is enriched. The entity recognition method combines character-level, sentence-level and visual feature triple semantic representation, not only considers local entity semantics, but also integrates global context information, and introduces visual features to extend limitation of embedded representation based on plain texts.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information extraction, and relates to an entity recognition method based on semantic analysis and label visual feature expansion. BACKGROUND

[0002] With the popularity of computers and the rapid development of the Internet, a large amount of information appears in the form of electronic documents in front of people. In order to cope with the serious challenge brought by information explosion, it is urgently needed to have some automatic tools to help people quickly find the information really needed in a large amount of information sources, and the information extraction technology research is generated in this background.

[0003] The main function of the information extraction system is to extract specific factual information from the text. For example, detailed information of a terrorist event is extracted from a news report: time, place, perpetrators, victims, targets, weapons used, etc.; the company releases new product information from economic news: company name, product name, release time, product performance, etc.; symptoms, diagnosis records, test results, prescriptions, etc. are extracted from patient medical records. Usually, the extracted information is described in a structured form and can be directly stored in a database for user query and further analysis and utilization.

[0004] In the late 1980s, the Message Understanding Conference (MUC) was held, which made information extraction develop into an important branch of natural language processing. MUC proposed named entity recognition (NER) as a core sub-task in the field of natural language processing (NLP) information extraction. The task of NER is to determine whether a certain text string in unstructured text represents a named entity and to determine the category of the named entity. Therefore, the development of named entity recognition is crucial for building knowledge graph triplets (entity-relation-entity) extraction and natural language question semantic analysis in intelligent question answering and semantic search downstream applications.

[0005] In recent years, with the rapid development of deep learning technology, especially the wide application of pre-trained language models (such as BERT, RoBERTa) based on Transformer architecture, the performance of NER task has been significantly improved. These pre-trained language models (PLM) use attention mechanism and are pre-trained on large-scale corpus in a self-supervised manner, and the context-related character / word embeddings generated by them provide a strong feature basis for NLP tasks. The deep learning-based named entity recognition methods mainly follow the following two paradigms: 1) sequence labeling-based methods (such as BiLSTM-CRF, BERT-CRF), which convert the entity recognition task into a classification task for each character; 2) span-based methods (such as Span-based methods), which directly enumerate all possible text spans and judge whether these text spans are entities and further judge their categories.

[0006] However, most of the existing entity recognition schemes have the following two shortcomings: On the one hand, there is a lack of full use of sentence structure information. Xiao et al. in "DuST: Chinese NER using dual-grained syntax-aware transformer network" pointed out that to deal with the challenge of Chinese entity boundary detection, previous studies have explored the introduction of lexical information to provide word-level knowledge for the Chinese NER task, which has a positive effect on boundary detection, but these methods ignore the importance of the sentence-level linguistic knowledge provided by syntactic information. And Jie and Lu in "Dependency-Guided LSTM-CRF for Named Entity Recognition" showed that by encoding the complete dependency tree structure to capture the long-distance dependency relationships and syntactic associations between words in a sentence, the performance of the named entity recognition task was improved. On the other hand, there is a lack of utilization of multi-modal information. Moon et al. in "Multimodel Named Entity Recognition for Short Social Media Posts" pointed out that although there have been many NER researches that have achieved success in entity recognition in formal texts by fusing word context analysis and character-level features, there are still special challenges when facing extremely short texts and rough content in social media. Short texts lack sufficient context to resolve the ambiguity of polysemous entities. Liu et al. in "Hierarchical Aligned Multimodal Learning for NER on Tweet Posts" pointed out that the supplement of visual information can alleviate the problem of semantic ambiguity and information deficiency caused by pure text.

[0007] Based on the above technical analysis, the research and development idea of entity recognition based on semantic analysis and label visual feature expansion is proposed. The problem of difficult determination of entity boundary is solved by calculating the similarity of candidate entity and global semantic of text and extracting interactive features of the two to implicitly construct embedding representation containing sentence structure information. Visual features are systematically introduced into the entity recognition task to construct a text-visual cross-modal entity recognition paradigm to enrich the context semantic information of embedding representation. SUMMARY

[0008] The purpose of the present application is to provide rich cross-modal semantic information beyond text for the model by innovatively introducing and fusing label visual embedding containing entity category information, to cope with the challenges of semantic representation singularity and low resource scene, and to realize more accurate and robust entity recognition by semantic analysis technology based on local embedding and global embedding, and finally to provide an entity recognition method based on semantic analysis and label visual feature expansion.

[0009] Technical scheme of the present application:

[0010] An entity recognition method based on semantic analysis and label visual feature expansion, the steps are as follows:

[0011] Step 1: Obtain the text data set to be processed, specify the label set of the text data set to be processed, and the label set is a set of predefined entity categories, the elements of the set are entity categories such as organizations and places;

[0012] Step 2: encode each text sample in the text data set to be processed by using a pre-trained language model to obtain , and obtain the character-level embedding vector sequence of the text sample, where , represents the embedding vector of the th character in the text sample, represents the number of characters in the text sample, represents the dimension of the character-level embedding vector;

[0013] Step 3: based on the obtained character-level embedding vector sequence of the text sample, construct the span-based candidate entity embedding representation of the text sample, where each candidate entity embedding vector in represents the embedding vector of the candidate entity with the corresponding character as the start and the corresponding character as the end, and the embedding vector of the candidate entity is represented by the semantic average of the embedding vector and the embedding vector , that is, ​​ Entity recognition is based on the assumption that the global semantic information of a text sample is jointly determined by the local semantic information of each entity in the text sample; the embedding vectors of different spans of candidate entities are calculated to assist entity recognition in terms of the contribution to the global semantics of the text sample;

[0014] Step 4: Sentence-level embedding vector containing the global semantics of the text sample from the sequence of character-level embedding vectors of the text sample the hidden state of the first character output by the Bi-LSTM module as input and the hidden state of the last character the semantic average representation: ; the forward embedding vector of the first character output by the Bi-LSTM module, the reverse embedding vector of the first character output by the Bi-LSTM module, the forward embedding vector of the last character output by the Bi-LSTM module, the reverse embedding vector of the last character output by the Bi-LSTM module;

[0015] Step 5: Design a semantic similarity module to calculate the embedding vector of each candidate entity the similarity between the local semantics contained and the global semantics of the sentence-level embedding vector of the text sample , the inner product of the embedding vector of each candidate entity and the sentence-level embedding vector containing the global semantics of the text sample , and the inner product result is taken as the prediction entity score , i.e. , represents the embedding vector of the candidate entity to the extent of the real entity;

[0016] Step 6: Design a candidate entity embedding and text sample sentence-level embedding interaction feature extraction module to extract the interaction features of the embedding vector of the candidate entity and the sentence-level embedding vector containing the global semantics of the text sample , the embedding vector of the candidate entity and the sentence-level embedding vector containing the global semantics of the text sample are respectively reconstructed into two-dimensional embedding matrices, which are , and stacked into a two-channel embedding tensor, which is ; is the height of the reconstructed two-dimensional embedding matrix, is the width of the reconstructed two-dimensional embedding matrix, and ; the convolution operation is performed on the stacked two-channel embedding tensor activating, extracting embedding vectors of candidate entities sentence-level embedding vectors of the entailment text sample local interaction features of the candidate entity to obtain a feature map; performing average pooling on the obtained feature map, flattening and performing linear transformation, and then inputting to a fully connected layer for classification to obtain a predicted entity score , wherein ; wherein is a convolution operation, is a flattening operation, is a linear transformation matrix, is a fully connected layer calculation;

[0017] Step 7: Design a candidate entity class prediction module to enrich the feature diversity of the candidate entity embedding vector, and collect representative pictures for each entity class in the label set to provide as much corresponding entity class feature as possible, obtain image embedding representations of entity class labels by encoding the pictures through a visual pre-training model fuse image embedding vectors under each entity class label using a self-attention mechanism, and then obtain entity class label image embeddings by averaging the image embedding vectors of each entity class in the label set position by position , wherein then obtain embedding representations of the label information contained in the candidate entity by using cross-attention mechanism , wherein the reconstructed candidate entity embedding representation serves as a query, entity class label image embedding representation serves as a key and a value, denotes the number of labels; concatenate the embedding representation of the label information contained in the candidate entity and the reconstructed candidate entity embedding representation to obtain the candidate entity embedding representation containing label information introduce more other-dimensional feature information to the candidate entity to enhance the class information, and finally input the candidate entity embedding representation containing label information to a fully connected layer for classification to obtain a candidate entity class score , denotes the degree to which the candidate entity belongs to a certain class; wherein denotes the label, is a position-by-position averaging operation, is a self-attention calculation, is a cross-attention calculation;

[0018] Step 8: Design the loss function for each module. During training, the semantic similarity module uses a margin-based loss function. ,in, , Indicates the positive sample interval threshold. This represents the negative sample interval threshold. Indicates a positive candidate entity. Indicates a negative candidate entity. Using continuous functions Approximately obtained, The approximation level is adjusted by parameters; the loss function for the interaction feature extraction module between candidate entity embedding and sentence-level embedding of text samples is a binary classification cross-entropy loss function. ,in When the candidate entity is a real entity When the candidate entity is not a real entity , , , Indicates the first There are 10 candidate entities; the candidate entity category prediction module uses a multi-class cross-entropy loss function. , , Total loss ,in , , These are loss weights used to adjust the learning effect;

[0019] Step 9: During prediction, the semantic similarity module maintains the same loss function as during training, and the set of predicted entities in the semantic similarity module is... Achieve a score greater than The elements, candidate entity embeddings, and the predicted entity set of the interaction feature extraction module with sentence-level embeddings of text samples. The candidate entity category prediction module consists of candidate entities whose positive class scores are greater than their negative class scores. The predicted entity set consists of candidate entities whose predicted category labels are valid, and the final predicted entity set is the intersection of these three sets. After that Decodes the entity into a predefined category in the tag set.

[0020] The beneficial effects of the present application: the entity recognition scheme combines character-level, sentence-level and visual feature triple semantic representation, which not only considers local entity semantics, but also integrates global context information, and introduces visual features to expand the limitations of pure text-based embedding representation. By introducing visual prior knowledge through representative images of entity categories, effectively dealing with the recognition errors of entity boundaries and types caused by semantic ambiguity of pure text model, and enhancing the depth of model understanding of entity semantics. Three independent modules evaluate the candidate entities from different angles, the semantic similarity module evaluates the fitness of the candidate entity and the global semantics, the candidate entity embedding and the text sample sentence-level embedding interaction feature extraction module captures the interaction mode of local information and global information (each entity corresponds to the sentence structure information), and the candidate entity category prediction module combines visual features for fine-grained classification. Three modules evaluate the candidate entities from three orthogonal dimensions of macro correlation, micro structure and external knowledge respectively, and through the shared underlying embedding representation, that is, the BERT character embedding, a robust general semantic embedding representation is optimized. In addition, the intersection strategy ensures that the final predicted entity is supported by multiple evidences, while maintaining high recall rate, the precision rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a flowchart of the entity recognition method based on semantic analysis and label visual feature extension of the present application. DETAILED DESCRIPTION

[0022] The specific embodiments of the present application are further illustrated below in combination with the drawings and technical solutions.

[0023] This embodiment runs on a server equipped with GPU, and the software program of the entity recognition method based on semantic analysis and label visual feature extension of the present application is loaded and executed by the processor on the server to complete the entity recognition task. The software environment includes Python, PyTorch deep learning framework and Transformers library.

[0024] To evaluate the performance of the entity recognition method based on semantic analysis and label visual feature expansion (SA-VFE-ER model) proposed in the present application, experiments were conducted on two widely used Chinese named entity recognition benchmark datasets: Weibo NER and Resume NER. In this embodiment, the standard training set, validation set and test set division was used for both datasets. The label set was defined according to the annotation specification of each dataset, for example, in the Weibo dataset, the label set may include PER, LOC, ORG, GPE and other categories. This embodiment selected five representative baseline models for comparison: a tagger model based on BERT (BERT-Tagger) that predicts the entity label of each character by adding a simple classification layer on top of the pre-trained BERT model; a BERT model based on conditional random field (BERT-CRF) that introduces a conditional random field (CRF) layer on the basis of BERT-Tagger to capture the transition dependency between labels; a BERT model based on bidirectional long short-term memory network and conditional random field (BERT-BiLSTM-CRF) that adds a bidirectional long short-term memory network (BiLSTM) layer on the basis of BERT-CRF to further capture the long distance dependency of character sequences; a span-based entity recognition model (SpanNER) that focuses on locating entities by identifying spans in the text rather than relying on traditional token sequence labeling, which can more flexibly handle nested and overlapping entity structures, and usually scores different length text fragments in combination with context information to determine which fragments constitute named entities; a dynamic graph information extraction model (DyGIE++) that is a model designed for nested NER and joint entity relation extraction, its "dynamic span graph" and "cross-span interaction" mechanism makes it perform well in capturing the complex relationship between entities and handling entity nesting.

[0025] The SA-VFE-ER model of the present application was constructed and trained, and the relevant parameter settings are as follows: the input text is sent into the BERT pre-trained language model for encoding to obtain a character-level embedding vector sequence C with d=768; the sentence-level embedding vector S is obtained by processing the character embedding sequence with a Bi-LSTM network with a hidden unit size of 384, and then averaging the forward and reverse hidden states of the first and last characters; the candidate entity embedding and text sample sentence-level embedding interaction feature extraction module, and S is reshaped into a matrix of (h=32, w=24) and stacked into a tensor of (2, 32, 24), followed by a Conv layer with a kernel size of (3, 3) for feature extraction; for each entity category in the label set such as organization, location, person, K=10 pictures with high representativeness and diversity are manually screened and collected, which are encoded by a ResNet-50 visual pre-training model to obtain a d=768 image embedding representation. The total loss function is the weighted sum of the loss functions of each module, where the positive and negative sample interval threshold of the loss of the semantic similarity module is set to 0.8 and 0.2 respectively, and the weight of the loss is set to 1.0, 1.0 and 1.0 respectively. and respectively.

[0026] To evaluate the effect of the SA-VFE-ER model of the application, comparative experiments and ablation experiments were carried out under the same experimental environment and training parameters (AdamW optimizer, learning rate of 2e-5, weight decay of 0.01, batch size of 16, gradient accumulation step of 2, maximum sequence length n=256, and maximum training round set to 100). Table 1 shows the experimental results of the entity recognition method based on semantic analysis and label visual feature expansion of the application and five baseline models on Weibo and Resume data sets. The results show that the SA-VFE-ER model of the application is superior to the five baseline models.

[0027] Table 1 comparative experiment results

[0028]

[0029] The SA-VFE-ER model of the application has a recall rate of 0.51% higher than the optimal baseline model DyGIE++, and an accuracy of 0.39% higher than DyGIE++ due to the proposed candidate entity class prediction module based on label visual expansion features, and an overall F1 of 0.45% higher than the DyGIE++ model.

[0030] To verify the effectiveness of the SA-VFE-ER model of the application, the application performs an ablation experiment on the Weibo data set, and the experimental results are shown in Table 2.

[0031] Table 2 ablation experiment results

[0032]

[0033] ​The entity recognition method based on semantic analysis and label visual feature extension of this invention extends the candidate entity embedding with label visual features on the basis of the model (SA-ER) based on semantic similarity module and candidate entity embedding and text sample sentence-level embedding interaction feature extraction module. This injects abstract prior knowledge into the model of "what an ideal entity of a certain category should look like". This prior knowledge acts as a "concept anchor" or "classification decision boundary adjuster" when the model makes decisions. When the original character-level embedding vector When semantic ambiguity is present (e.g., "apple"), the SA-VFE-ER model enhances features by fusing the attention mechanism in the candidate entity category prediction module based on label visual extension features with the category visual extension features of "fruit" and "company," resulting in a more robust classification. This directly leads to improved accuracy. The entity recognition method based on semantic analysis and label visual feature extension of this invention adds a semantic similarity module and a candidate entity embedding and text sample sentence-level embedding interaction feature extraction module to the model based on the candidate entity category prediction module (VFE-ER). Through convolution operations, it captures the local interaction features between candidate segments and the full text, enabling it to more sensitively identify entity boundaries and thus find a large number of entities missed by the baseline model. However, it also includes more non-entity or misclassified segments. This indicates that relying solely on semantic interactions within the text is insufficient for the most accurate classification decision. The entity recognition method based on semantic analysis and label visual feature extension of this invention combines the two, achieving synergistic gains and proving the effectiveness of the proposed solution.

Claims

1. An entity recognition method based on semantic analysis and label visual feature expansion, characterized in that, The steps are as follows: Step 1: Obtain a text data set to be processed, specify a label set of the text data set to be processed, and the label set is a set of predefined entity categories, and the elements of the set are entity categories; Step 2: Utilizing a pre-trained language model to process each text sample in the text data set is encoded to obtain , and a character-level embedding vector sequence of the text sample is obtained , where represents the embedding vector of the th character in the text sample, represents the number of characters in the text sample, represents the dimension of the character-level embedding vector. Step 3: Based on the character-level embedding vector sequence of the text sample Construct a span-based candidate entity embedding representation for this text sample. Span-based candidate entity embedding representation Embedding vector of each candidate entity Indicated by The corresponding characters are the start, The embedding vector of the candidate entity whose corresponding character ends in 'embed'. Using embedding vectors and embedding vector The semantic average is used to represent it. Entity recognition is based on the assumption that the global semantic information of a text sample is jointly determined by the local semantic information of each entity in the text sample; this is achieved by calculating the embedding vectors of candidate entities across different spans. The contribution of the text sample to the global semantics is used to assist entity recognition; Step 4: Sentence-level embedding vector that encapsulates the global semantics of the text sample the sequence of character-level embedding vectors of the text sample the hidden state of the first character output by the Bi-LSTM module as input and the hidden state of the last character a semantic average representation of: ; is the forward embedding vector of the first character output by the Bi-LSTM module, is the backward embedding vector of the first character output by the Bi-LSTM module, is the forward embedding vector of the last character output by the Bi-LSTM module, is the backward embedding vector of the last character output by the Bi-LSTM module; Step 5: Design a semantic similarity module; Step 6: Design a candidate entity embedding and text sample sentence level embedding interaction feature extraction module; Step 7: Design a candidate entity category prediction module; Step 8: design the loss function of each module. During training, the loss of the semantic similarity module adopts the interval-based loss wherein , represents the positive sample interval threshold, represents the negative sample interval threshold, represents the positive candidate entity, represents the negative candidate entity, is approximated by a continuous function , is an approximation degree adjustment parameter; the loss of the candidate entity embedding and text sample sentence-level embedding interaction feature extraction module adopts a binary classification cross-entropy loss function, wherein , when the candidate entity is a real entity , , , represents the i-th candidate entity; the candidate entity category prediction module adopts a multi-classification cross-entropy loss function , , , ; the total loss wherein , , is a loss weight for adjusting the learning effect; Step 9: At prediction time, the semantic similarity module is consistent with the loss function at training time, the predicted entity set of the semantic similarity module is obtained by selecting elements with scores greater than , the predicted entity set of the candidate entity embedding and text sample sentence-level embedding interaction feature extraction module is obtained by selecting candidate entities with positive class scores greater than negative class scores, the predicted entity set of the candidate entity class prediction module is obtained by selecting candidate entities with predicted class labels as valid labels, and the final predicted entity set is the intersection of the three, i.e. , and then is decoded into an entity category predefined in the label set. 2.The entity recognition method based on semantic analysis and label visual feature extension according to claim 1, characterized in that, The specific implementation process of step 5 is as follows: Calculate the embedding vector of each candidate entity. Implied local semantics and sentence-level embedding vectors of text samples The global semantic similarity is used to compute the embedding vector of each candidate entity. With sentence-level embedding vectors containing the global semantics of text samples The inner product is used as the predicted entity score. ,Right now , representing the embedding vector of the candidate entity. The degree to which it represents a real entity. 3.The entity recognition method based on semantic analysis and label visual feature extension according to claim 2, characterized in that, The specific implementation process of step 6 is as follows: extracting the embedding vector of the candidate entity and the sentence-level embedding vector of the global semantics of the text sample that contains the candidate entity and the sentence-level embedding vector of the global semantics of the text sample that contains the candidate entity respectively reconstructing into two-dimensional embedding matrices, and stacking into a two-channel embedding tensor ; wherein is the height of the reconstructed two-dimensional embedding matrix is the width of the reconstructed two-dimensional embedding matrix ; performing convolution operation and activation on the stacked two-channel embedding tensor, extracting the embedding vector of the candidate entity and the sentence-level embedding vector of the global semantics of the text sample that contains the candidate entity ; wherein is the convolution operation is the flattening operation is the linear transformation matrix is the full connection layer calculation. 4.The entity recognition method based on semantic analysis and label visual feature extension according to claim 3, characterized in that, The specific implementation process of step 7 is as follows: collect the image embedding vectors of each entity category label in the label set representative pictures, Encode the picture through the visual pre-training model to obtain the image embedding vector of the entity category label Fuse the image embedding vectors under each entity category label using the self-attention mechanism, then average the image embedding vectors of each entity category in the label set position by position to obtain the image embedding of the entity category label , Then use the cross-attention mechanism to obtain the embedding representation of the candidate entity label information , The reconstructed candidate entity embedding representation is used as a query, The image embedding representation of the entity category label is used as a key and value, represents the number of labels; The embedding representation of the candidate entity label information is concatenated with the reconstructed candidate entity embedding representation to obtain the candidate entity embedding representation containing label information Finally, the candidate entity embedding representation containing label information is input into the fully connected layer for classification to obtain the candidate entity category score , , which represents the degree to which the candidate entity belongs to a certain category; wherein, represents the th label, is a position-by-position averaging operation, is a self-attention calculation, is a cross-attention calculation.

Images