Entity recognition method based on semantic analysis and label visual feature extension

Abstract

The application belongs to the technical field of information extraction, and provides an entity recognition method based on semantic analysis and label visual feature expansion. The method implicitly constructs embedding representation containing sentence structure information by calculating the similarity of candidate entities and global semantics of the text and extracting interactive features of both, to solve the problem that the entity boundary is difficult to determine. The method systematically introduces visual features into the entity recognition task, constructs a text-visual cross-modal entity recognition paradigm, and enriches the context semantic information of the embedding representation. The entity recognition method combines character-level, sentence-level and visual feature triple semantic representation, considers local entity semantics, integrates global context information, and introduces visual feature to expand the limitations of embedding representation based on pure text.

Description

Technical Field

[0001] This invention belongs to the field of information extraction technology and relates to an entity recognition method based on semantic analysis and label visual feature extension. Background Technology

[0002] With the widespread use of computers and the rapid development of the internet, a vast amount of information appears before people in the form of electronic documents. To cope with the serious challenges brought about by the information explosion, there is an urgent need for automated tools to help people quickly find the information they truly need from massive information sources. Research on information extraction technology has emerged in this context.

[0003] Information extraction systems primarily extract specific factual information from text. For example, they might extract detailed information about a terrorist incident from news reports: time, location, perpetrators, victims, targets, and weapons used; extract information about new product launches from economic news: company name, product name, launch date, and product specifications; or extract symptoms, diagnoses, test results, and prescriptions from patients' medical records. Typically, the extracted information is described in a structured format and can be directly stored in a database for user querying and further analysis.

[0004] The Message Understanding Conference (MUC) held in the late 1980s propelled information extraction into a significant branch of Natural Language Processing (NLP). MUC proposed Named Entity Recognition (NER) as a core subtask within NLP information extraction. NER's task is to determine whether a string of unstructured text represents a named entity and to classify that entity. Therefore, the development of Named Entity Recognition is crucial for downstream applications such as triple (entity-relation-entity) extraction in knowledge graph construction and semantic analysis of natural language questions in intelligent question answering and semantic search.

[0005] In recent years, with the rapid development of deep learning technology, especially the widespread application of pre-trained language models based on the Transformer architecture (such as BERT and RoBERTa), the performance of NER tasks has been significantly improved. These pre-trained language models (PLMs) employ attention mechanisms and undergo self-supervised pre-training on large-scale corpora. The context-sensitive character / word embeddings they generate provide a powerful feature foundation for NLP tasks. Deep learning-based named entity recognition methods mainly follow two paradigms: 1) Sequence labeling-based methods (such as BiLSTM-CRF and BERT-CRF), which transform the entity recognition task into a classification task for each character; 2) Segment sorting-based methods (such as span-based methods), which directly enumerate all possible text segments and determine whether each segment is an entity and further determine its category.

[0006] However, most existing entity recognition schemes suffer from two main shortcomings: First, they lack full utilization of sentence structure information. Xiao et al., in "DuST: Chinese NER using dual-grained syntax-aware transformer network," point out that to address the challenge of Chinese entity boundary detection, previous research has explored introducing lexical information to provide word-level knowledge for Chinese NER tasks, which has a positive effect on boundary detection. However, these methods neglect the importance of sentence-level linguistic knowledge provided by syntactic information. Jie and Lu, in "Dependency-GuidedLSTM-CRF for Named Entity Recognition," demonstrate that encoding a complete dependency tree structure to capture long-distance dependencies and syntactic associations between words in a sentence improves the performance of named entity recognition. Second, they lack utilization of multimodal information. In their paper "Multimodel Named Entity Recognition for Short Social Media Posts," Moon et al. point out that while numerous NER studies have achieved success in entity recognition of canonical text through the fusion of word context analysis and character-level features, unique challenges remain when dealing with extremely short and coarse content in social media, as short texts lack sufficient context to eliminate ambiguity in polysemous entities. Liu et al., in their paper "Hierarchical Aligned Multimodal Learning for NER on Tweet Posts," indicate that supplementing with visual information can alleviate semantic ambiguity and information insufficiency caused by plain text.

[0007] Based on the above technical analysis, a research and development concept for entity recognition based on semantic analysis and label visual feature extension is proposed. By calculating the similarity between candidate entities and the global semantics of the text and extracting the interaction features between the two, an embedded representation containing sentence structure information is implicitly constructed to solve the problem of difficult determination of entity boundaries. Furthermore, visual features are systematically introduced into the entity recognition task to construct a text-visual cross-modal entity recognition paradigm to enrich the contextual semantic information of the embedded representation. Summary of the Invention

[0008] The purpose of this invention is to provide the model with rich cross-modal semantic information that goes beyond text by innovatively introducing and integrating label visual embeddings that contain entity category information, thereby addressing the challenges of semantic representation simplification and low-resource scenarios. Furthermore, through semantic analysis techniques based on local and global embedding, more accurate and robust entity recognition is achieved, ultimately providing an entity recognition method based on semantic analysis and label visual feature extension.

[0009] The technical solution of this invention:

[0010] An entity recognition method based on semantic analysis and label visual feature extension, comprising the following steps:

[0011] Step 1: Obtain the text dataset to be processed. Specify the label set of the text dataset to be processed. The label set is a collection of predefined entity categories. The elements of this collection are entity categories such as organization and location.

[0012] Step 2: Use a pre-trained language model for each text sample in the text dataset to be processed. Encode to obtain Obtain the character-level embedding vector sequence of the text sample. ,in Indicates the first in the text sample Embedding vectors of characters, This indicates the number of characters in the text sample. Indicates the dimension of the character-level embedding vector;

[0013] Step 3: Based on the character-level embedding vector sequence of the obtained text samples Construct a span-based candidate entity embedding representation for this text sample. ,in Embedding vector of each candidate entity Indicated by The corresponding characters are the start, The embedding vector of the candidate entity whose corresponding character ends in 'embed'. Using embedding vectors and embedding vector The semantic average is used to represent it, that is Entity recognition is based on the assumption that the global semantic information of a text sample is jointly determined by the local semantic information of each entity in the text sample; this is achieved by calculating the embedding vectors of candidate entities across different spans. The contribution of the text sample to the global semantics is used to assist entity recognition;

[0014] Step 4: Sentence-level embedding vectors containing the global semantics of the text samples The character-level embedding vector sequence of text samples The hidden state of the first character output by the Bi-LSTM module as input. and the hidden state of the last character Semantic average representation: ; The first character forward embedding vector output by the Bi-LSTM module. The first character inverse embedding vector output by the Bi-LSTM module. This is the last character forward embedding vector output by the Bi-LSTM module. The last character inverse embedding vector output by the Bi-LSTM module;

[0015] Step 5: Design a semantic similarity module to calculate the embedding vector for each candidate entity. Implied local semantics and sentence-level embedding vectors of text samples The global semantic similarity is used to compute the embedding vector of each candidate entity. With sentence-level embedding vectors containing the global semantics of text samples The inner product is used as the predicted entity score. ,Right now , representing the embedding vector of the candidate entity. The degree to which it is a real entity;

[0016] Step 6: Design an interactive feature extraction module for candidate entity embedding and sentence-level embedding of text samples to extract the embedding vectors of candidate entities. With sentence-level embedding vectors containing the global semantics of text samples The interaction features of the candidate entities are used to embed their vectors. and sentence-level embedding vectors containing the global semantics of text samples Reconstruct them into two-dimensional embedding matrices respectively, for And stacked into a two-channel embedding tensor, for ; The height of the reconstructed two-dimensional embedding matrix, The width of the reconstructed two-dimensional embedding matrix, and Perform convolution operations on the stacked two-channel embedding tensors and Activation, extracting the embedding vectors of candidate entities With sentence-level embedding vectors containing the global semantics of text samples The local interaction features are used to obtain feature maps; the obtained feature maps are then flattened by average pooling and subjected to linear transformation, and then input into a fully connected layer for classification to obtain predicted entity scores. ,Right now ;in, For convolution operations, For flattening operation, It is a linear transformation matrix. Calculations are performed for fully connected layers;

[0017] Step 7: Design a candidate entity category prediction module. To enrich the feature diversity of candidate entity embedding vectors, collect data for each entity category in the label set. Provide as many representative images as possible to represent the corresponding entity category features. Image embedding representations of entity category labels are obtained by encoding images using a visual pre-trained model. The image embedding vectors under each entity category label are fused using a self-attention mechanism. Then, the image embedding vectors of each entity category in the label set are averaged position by position to obtain the entity category label image embedding. ,Right now Then, the cross-attention mechanism is used to obtain the embedding representation containing candidate entity label information. , The reconstructed candidate entity embedding representation As a query Entity category label image embedding representation As keys and values, Indicates the number of tags; embeds the information of candidate entity tags into the representation. With the reconstructed candidate entity embedding representation Concatenation yields candidate entity embedding representations containing label information. This involves introducing more feature information from other dimensions to candidate entities to enhance category information, and finally embedding candidate entities containing label information into the representation. The input is fed into a fully connected layer for classification, and candidate entity category scores are obtained. , , indicating the degree to which a candidate entity belongs to a certain category; where, Indicates the first A tag, This is an operation that takes the average value position by position. For self-attention computation, For cross-attention calculation;

[0018] Step 8: Design the loss function for each module. During training, the semantic similarity module uses a margin-based loss function. ,in, , Indicates the positive sample interval threshold. This represents the negative sample interval threshold. Indicates a positive candidate entity. Indicates a negative candidate entity. Using continuous functions Approximately obtained, The approximation level is adjusted by parameters; the loss function for the interaction feature extraction module between candidate entity embedding and sentence-level embedding of text samples is a binary classification cross-entropy loss function. ,in When the candidate entity is a real entity When the candidate entity is not a real entity , , , Indicates the first There are 10 candidate entities; the candidate entity category prediction module uses a multi-class cross-entropy loss function. , , Total loss ,in , , These are loss weights used to adjust the learning effect;

[0019] Step 9: During prediction, the semantic similarity module maintains the same loss function as during training, and the set of predicted entities in the semantic similarity module is... Achieve a score greater than The elements, candidate entity embeddings, and the predicted entity set of the interaction feature extraction module with sentence-level embeddings of text samples. The candidate entity category prediction module consists of candidate entities whose positive class scores are greater than their negative class scores. The predicted entity set consists of candidate entities whose predicted category labels are valid, and the final predicted entity set is the intersection of these three sets. After that Decodes the entity into a predefined category in the tag set.

[0020] The beneficial effects of this invention are as follows: This entity recognition scheme combines character-level, sentence-level, and visual feature-based semantic representations. It considers both local entity semantics and global contextual information, and introduces visual features to expand the limitations of plain text-based embedding representations. By introducing visual prior knowledge through representative images of entity categories, it effectively addresses the problem of entity boundary and type recognition errors caused by semantic ambiguity in plain text models, enhancing the model's understanding of entity semantics. Three independent modules evaluate candidate entities from different perspectives: the semantic similarity module evaluates the fit between candidate entities and global semantics; the candidate entity embedding and text sample sentence-level embedding interaction feature extraction module captures the interaction patterns between local and global information (sentence structure information corresponding to each entity); and the candidate entity category prediction module performs fine-grained classification by combining visual features. The three modules evaluate candidate entities from three orthogonal dimensions: macro-relevance, micro-structure, and external knowledge. By sharing the underlying embedding representation, namely BERT character embedding, a robust general semantic embedding representation is collaboratively optimized. Furthermore, the intersection strategy ensures that the final predicted entity is supported by multiple pieces of evidence, improving precision while maintaining high recall. Attached Figure Description

[0021] Figure 1 This is a flowchart of the entity recognition method based on semantic analysis and label visual feature extension of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0023] This embodiment runs on a server equipped with a GPU. The software program for the entity recognition method based on semantic analysis and label visual feature extension, as described in this invention, is loaded and executed by the processor on the server to complete the entity recognition task. The software environment includes Python, the PyTorch deep learning framework, and the Transformers library.

[0024] To evaluate the performance of the entity recognition method (SA-VFE-ER model) based on semantic analysis and label visual feature extension proposed in this invention, experiments were conducted on two widely used Chinese named entity recognition benchmark datasets: Weibo NER and Resume NER. In this embodiment, both datasets were divided into standard training, validation, and test sets. The label set was defined according to the annotation specifications of each dataset. For example, in the Weibo dataset, the label set may include categories such as PER, LOC, ORG, and GPE. This embodiment selects five representative baseline models for comparison: the BERT-Tagger-based labeler model predicts the entity label for each character by adding a simple classification layer on top of the pre-trained BERT model; the BERT-CRF-based Conditional Random Field model introduces a Conditional Random Field (CRF) layer on top of BERT-Tagger to capture transition dependencies between labels; the BERT-BiLSTM-CRF-based model adds a BiLSTM layer on top of BERT-CRF to further capture long-distance dependencies in character sequences; the SpanNER-based entity recognition model focuses on locating entities by recognizing spans in text, rather than relying on traditional labeled sequences, thus handling nested and overlapping entity structures more flexibly. It typically combines contextual information to score text segments of different lengths to determine which segments constitute named entities; and the DyGIE++-based dynamic graph information extraction model is a model for nested, NER (Nested Graph Recognition)... The model, designed with NER and joint entity relationship extraction, excels in capturing complex relationships between entities and handling entity nesting due to its "dynamic span graph" and "span interaction" mechanisms.

[0025] The SA-VFE-ER model of this invention is constructed and trained with the following parameter settings: Input text is fed into a BERT pre-trained language model for encoding, resulting in a character-level embedding vector sequence C with d=768; sentence-level embedding vector S is obtained by averaging the forward and backward hidden states of the first and last characters after processing the character embedding sequence by a Bi-LSTM network with a hidden unit size of 384; in the candidate entity embedding and text sample sentence-level embedding interaction feature extraction module... S is reshaped into a matrix (h=32, w=24) and stacked into a tensor of (2, 32, 24), then subjected to feature extraction through a Conv layer with a kernel size of (3,3). For each entity category in the label set, such as organization or location, K=10 highly representative and diverse images are manually selected and collected. These images are encoded using a ResNet-50 visual pre-trained model to obtain an image embedding representation of d=768. The total loss function is a weighted sum of the loss functions of each module, where the positive and negative sample interval threshold of the loss of the semantic similarity module is used. and The weights for the loss are set to 0.8 and 0.2 respectively. Set them to 1.0, 1.0, and 1.0 respectively.

[0026] To evaluate the performance of the SA-VFE-ER model of this invention, comparative and ablation experiments were conducted under the same experimental environment and training parameters (AdamW optimizer, learning rate 2e-5, weight decay 0.01, batch size 16, gradient accumulation steps 2, maximum sequence length n=256, maximum training epochs set to 100). Table 1 shows the experimental results of the entity recognition method based on semantic analysis and label visual feature extension of this invention and five baseline models on the Weibo and Resume datasets. The results indicate that the SA-VFE-ER model of this invention outperforms the five baseline models.

[0027] Table 1 Comparison of experimental results

[0028]

[0029] The SA-VFE-ER model of this invention improves the recall rate by 0.51% compared to the best baseline model DyGIE++ by proposing a semantic similarity module and a candidate entity embedding and text sample sentence-level embedding interaction feature extraction module. Furthermore, due to the proposed candidate entity category prediction module based on label visual extended features, the recognition accuracy is improved by 0.39% compared to DyGIE++, and the overall F1 score is improved by 0.45% compared to the DyGIE++ model.

[0030] To verify the effectiveness of the SA-VFE-ER model of this invention, an ablation experiment was conducted on the Weibo dataset, and the experimental results are shown in Table 2.

[0031] Table 2 shows the results of the ablation experiment.

[0032]

[0033] The entity recognition method based on semantic analysis and label visual feature extension of this invention extends the candidate entity embedding with label visual features on the basis of the model (SA-ER) based on semantic similarity module and candidate entity embedding and text sample sentence-level embedding interaction feature extraction module. This injects abstract prior knowledge into the model of "what an ideal entity of a certain category should look like". This prior knowledge acts as a "concept anchor" or "classification decision boundary adjuster" when the model makes decisions. When the original character-level embedding vector When semantic ambiguity is present (e.g., "apple"), the SA-VFE-ER model enhances features by fusing the attention mechanism in the candidate entity category prediction module based on label visual extension features with the category visual extension features of "fruit" and "company," resulting in a more robust classification. This directly leads to improved accuracy. The entity recognition method based on semantic analysis and label visual feature extension of this invention adds a semantic similarity module and a candidate entity embedding and text sample sentence-level embedding interaction feature extraction module to the model based on the candidate entity category prediction module (VFE-ER). Through convolution operations, it captures the local interaction features between candidate segments and the full text, enabling it to more sensitively identify entity boundaries and thus find a large number of entities missed by the baseline model. However, it also includes more non-entity or misclassified segments. This indicates that relying solely on semantic interactions within the text is insufficient for the most accurate classification decision. The entity recognition method based on semantic analysis and label visual feature extension of this invention combines the two, achieving synergistic gains and proving the effectiveness of the proposed solution.

Claims

1. An entity recognition method based on semantic analysis and label visual feature extension, characterized in that, The steps are as follows: Step 1: Obtain the text dataset to be processed, specify the label set of the text dataset to be processed. The label set is a collection of predefined entity categories, and the elements of this collection are entity categories; Step 2: Use a pre-trained language model for each text sample in the dataset to be processed. Encode to obtain Then obtain the character-level embedding vector sequence of the text sample. ,in Indicates the first in the text sample Embedding vectors of characters, This indicates the number of characters in the text sample. Indicates the dimension of the character-level embedding vector; Step 3: Based on the character-level embedding vector sequence of the text sample Construct a span-based candidate entity embedding representation for this text sample. Span-based candidate entity embedding representation Embedding vector of each candidate entity Indicates The corresponding characters are the start, The embedding vector of the candidate entity whose corresponding character ends in 'embed'. Using embedding vectors and embedding vector The semantic average is used to represent it. Entity recognition is based on the assumption that the global semantic information of a text sample is jointly determined by the local semantic information of each entity in the text sample; this is achieved by calculating the embedding vectors of candidate entities across different spans. The contribution of the text sample to the global semantics is used to assist entity recognition; Step 4: Sentence-level embedding vectors containing the global semantics of the text samples The character-level embedding vector sequence of text samples The hidden state of the first character output by the Bi-LSTM module as input. and the hidden state of the last character Semantic average representation: ; The first character forward embedding vector output by the Bi-LSTM module. The first character inverse embedding vector output by the Bi-LSTM module. This is the last character forward embedding vector output by the Bi-LSTM module. The last character inverse embedding vector output by the Bi-LSTM module; Step 5: Design the semantic similarity module; Step 6: Design an interactive feature extraction module for candidate entity embedding and sentence-level embedding of text samples; Step 7: Design the candidate entity category prediction module; Step 8: Design the loss function for each module. During training, the semantic similarity module uses a margin-based loss function. ,in, , , This represents the predicted entity score tensor output by the semantic similarity module. Indicates the positive sample interval threshold. This represents the negative sample interval threshold. Indicates a positive candidate entity. Indicates a negative candidate entity. Using continuous functions Approximately obtained, The approximation level is adjusted by parameters; the loss function for the interaction feature extraction module between candidate entity embedding and sentence-level embedding of text samples is a binary classification cross-entropy loss function. ,in When the candidate entity is a real entity When the candidate entity is not a real entity , , , , This represents the predicted entity score output by the interaction feature extraction module between candidate entity embeddings and sentence-level embeddings of text samples. For convolution operations, For flattening operation, It is a linear transformation matrix. Calculations are performed for fully connected layers; Indicates the first One candidate entity, This indicates an element-wise normalization operation on the tensor; This represents tensor stacking operations; the candidate entity category prediction module uses a multi-class cross-entropy loss function. , , , , , Indicates the first A tag, Indicates the number of tags; Candidate entity embedding representation; total loss ,in , , These are loss weights used to adjust the learning effect; Step 9: During prediction, the semantic similarity module maintains the same loss function as during training, and the set of predicted entities in the semantic similarity module is... Achieve a score greater than The elements, candidate entity embeddings, and the predicted entity set of the interaction feature extraction module with sentence-level embeddings of text samples. The candidate entity category prediction module consists of candidate entities whose positive class scores are greater than their negative class scores. The predicted entity set consists of candidate entities whose predicted category labels are valid, and the final predicted entity set is the intersection of these three sets. After that Decodes the entity into a predefined category in the tag set.

2. The entity recognition method based on semantic analysis and label visual feature extension according to claim 1, characterized in that, The specific implementation process of step 5 is as follows: Calculate the embedding vector of each candidate entity. Implied local semantics and sentence-level embedding vectors of text samples The global semantic similarity is used to compute the embedding vector of each candidate entity. With sentence-level embedding vectors containing the global semantics of text samples The inner product is used as the predicted entity score. .

3. The entity recognition method based on semantic analysis and label visual feature extension according to claim 2, characterized in that, The specific implementation process of step 6 is as follows: Extract the embedding vector of the candidate entity. With sentence-level embedding vectors containing the global semantics of text samples The interaction features of the candidate entities are used to embed their vectors. and sentence-level embedding vectors containing the global semantics of text samples Reconstruct them into two-dimensional embedding matrices respectively, for And stacked into a two-channel embedding tensor, for ;in, The height of the reconstructed two-dimensional embedding matrix, The width of the reconstructed two-dimensional embedding matrix, and , This represents a tensor stacking operation; it involves performing a convolution operation on the embedded tensors of the stacked two channels. Activation, extracting the embedding vectors of candidate entities With sentence-level embedding vectors containing the global semantics of text samples The local interaction features are used to obtain feature maps; these feature maps are then flattened by average pooling and subjected to linear transformation before being input into a fully connected layer for classification to obtain predicted entity scores. .

4. The entity recognition method based on semantic analysis and label visual feature extension according to claim 3, characterized in that, The specific implementation process of step 7 is as follows: Collect data for each entity category in the tag set. Zhang is a representative picture. Image embedding vectors for entity category labels are obtained by encoding images using a visual pre-trained model. The image embedding vectors under each entity category label are fused using a self-attention mechanism. Then, the image embedding vectors of each entity category in the label set are averaged position by position to obtain the entity category label image embedding. , Then, the cross-attention mechanism is used to obtain the embedding representation containing candidate entity label information. , The reconstructed candidate entity embedding representation As a query Entity category label image embedding representation As keys and values, Indicates the number of tags; embeds the information of candidate entity tags into the representation. With the reconstructed candidate entity embedding representation Concatenation yields candidate entity embedding representations containing label information. Finally, the candidate entities containing the label information are embedded in the representation. The input is fed into a fully connected layer for classification, and candidate entity category scores are obtained. , , indicating the degree to which a candidate entity belongs to a certain category; where, This is an operation that takes the average value position by position. For self-attention computation, This is for cross-attention calculation.

Images