Enhanced perception named entity recognition method based on large model and mixed experts
By adopting enhanced perception methods of large models and hybrid experts in named entity recognition technology, using multimodal information and graphic search, the problems of single modal limitations and language dependence in unified data management systems are solved, and named entities are improved The accuracy and robustness of recognition enable better multilingual processing capabilities.
Patent Information
- Application Number
- CN202411180411.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-17
AI Technical Summary
The existing named entity recognition technology has problems such as single modal limitations, information loss, noise and unstructured data, and language dependence in the unified data management system, resulting in word meaning ambiguity, information loss and multilingual processing difficulty, which is difficult to effectively Identify entities.
Adopting the enhanced perceptual named entity recognition method based on big models and hybrid experts, through four task unit architectures: data augmentation based on literary and biographical graphics model, graphic and text retrieval, multimodal information fusion based on hybrid expert models, and large-modal model-based generation The design of named entity recognition task uses multimodal information and graphic retrieval to improve the training effect and generalization ability of the model.
It improves the accuracy and robustness of named entity recognition, reduces information loss and noise interference, enhances the generalization ability of the model, and can better handle multilingual and cross-language entity recognition tasks.
Smart Images
Figure CN119150868A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of natural language processing and computer vision technology, and in particular to an enhanced perception named entity recognition method based on a large model and mixed experts. Background Art
[0002] Named entity recognition technology has received extensive attention in recent years and is one of the most basic areas in unified data management systems. Although existing named entity recognition-related work has achieved remarkable performance, it often has the following defects in unified data management systems:
[0003] First, the limitations of a single modality make it impossible for us to obtain enough context from text information to accurately identify entities.
[0004] For example, without specific context, the word "Apple" can refer to both a fruit and a technology company, which creates a problem of word meaning ambiguity.
[0005] Second, the text information itself may be incomplete and lack necessary visual cues, which are crucial for understanding certain text content.
[0006] For example, due to language and cultural differences, the same text description may correspond to different scenarios; for another example, when describing a game, the text may not go into detail about the specific actions or scenes of each athlete, resulting in missing information.
[0007] 3. Noise and unstructured data are also a big problem.
[0008] Texts from different sources and dissemination channels, such as social media content posted by individuals, official documents, paper books or web pages, are often unstructured and may contain varying degrees of typos, grammatical errors or semantic transmission errors, which poses a challenge to the understanding ability of traditional named entity recognition models.
[0009] At the same time, diverse expressions also increase the difficulty of entity recognition, because the same entity may have multiple names or writing methods, and different entities may show certain similarities in the text.
[0010] Fourth, language dependence is also an issue that cannot be ignored.
[0011] Traditional named entity recognition models are usually trained and optimized for a single language, which makes them difficult to handle multilingual or cross-lingual entity recognition tasks.
[0012] The above problems together constitute complex challenges in named entity recognition tasks, which require the development of more flexible and intelligent methods to solve. Summary of the invention
[0013] In order to solve the above technical problems, the present invention provides an enhanced perceptual named entity recognition method based on a large model and hybrid experts. Through the original design of the four task unit architecture, multimodal information is fully utilized to better perform named entity recognition; in the data enhancement stage, a graphic retrieval method is used to further extract similar content from the generated data, and a large model and a hybrid expert model are introduced in the final named entity recognition part to better process multimodal information, improve the training effect and generalization ability of the model, thereby improving the performance of the model in the named entity recognition task.
[0014] An enhanced perceptual named entity recognition method based on a large model and hybrid experts is designed with four task unit architectures, including: data enhancement task based on a text-graph model, image-text retrieval task, multimodal information fusion task based on a hybrid expert model, and generative named entity recognition task based on a large model, where:
[0015] ① The data enhancement task based on the Wensheng graph model:
[0016] It is used to enhance the effect of named entity recognition tasks. By introducing image information, the text information and the image information can be mutually verified, reducing the risk of information loss and misrecognition under a single modality. The text-image model can construct an image that is more consistent with the text semantics and has less noise information. The image-text pairs generated by this method are used as a database.
[0017] As an example, extracting high-level features from the mixture of text information and image information can effectively strengthen the alignment relationship between entity details and image information and text information, thereby improving the performance of named entity recognition tasks.
[0018] ②The image and text retrieval task:
[0019] The image information and text information of the currently input target image-text pair sample are calculated to obtain the corresponding image features and text features. Then, each image-text pair in the database is traversed to calculate the features of its image information and text information respectively. Then, the cosine similarity between the image features and text features of the target image-text pair sample is calculated respectively, and the weighted average is taken as the final image-text pair similarity.
[0020] As an example, by searching the database, K image-text pairs with the highest similarity are extracted to further improve the information completeness of the relevant image-text features.
[0021] As an example, the K image-text pairs with the highest similarity retrieved are used as knowledge enhancement samples.
[0022] ③The multimodal information fusion task based on the hybrid expert model:
[0023] It is used to ensure that the features of each input text information and image information can be optimally processed to enhance the processing of image and text features; the input image and text features are processed through the cross-attention mechanism to obtain mixed features, which are then passed through one or more linear layers of the router to obtain the expert scores, and then routed to each mixed expert model for further processing;
[0024] As an example, the multimodal information fusion task based on the hybrid expert model introduces a hybrid expert mechanism in the multimodal feature fusion stage. The hybrid features are distributed to each hybrid expert model according to the calculated scores through the router to expand the parameter scale and improve the robustness of the multimodal information fusion model, so that it can better cope with noise interference.
[0025] As an example, the image-text retrieval task and the multimodal information fusion task based on the hybrid expert model are both improved based on the multimodal model open-flamingo. The CLIP model is used to extract features of text information and image information pairs, and then similarity retrieval is performed with other image-text pairs in the database.
[0026] ④ The generative named entity recognition task based on the large model:
[0027] First, for the knowledge-enhanced samples, the Vision Encoder and Language Encoder are used to encode them respectively, and the image information and text information of the samples are combined with the target image and text. The text information needs to be concatenated with the instructions before encoding. The image and text features obtained after encoding and combining are input into the multimodal information fusion model for processing.
[0028] Then, the traditional named entity recognition classification task is improved into a generation task, and the multimodal information fusion task based on the hybrid expert model is combined with the decoding of the large model; for each decoding layer, the input image and text features are first processed by the third unit architecture, and then further processed by the decoding layer;
[0029] Finally, entities are generated from the image and text features, and the model is trained with the goal of minimizing the language model loss L.
[0030] As an example, the introduction of the large model improves the output method of the traditional named entity recognition task. Through the generative output method, the generalization ability of the multimodal named entity recognition model on the visible classes is optimized, and the knowledge base of the multimodal named entity recognition model is deepened.
[0031] Beneficial effects of the present invention:
[0032] The design architecture of the present invention is scientific and reliable, and enhances the effect of improving named entity recognition. The rich knowledge of the large model itself is used to provide additional knowledge for named entity recognition, enhance external information, and thus assist the named entity recognition effect.
[0033] The present invention makes full use of image knowledge and utilizes a hybrid expert model to screen important information, remove interference information, and enhance the robustness of named entity recognition.
[0034] The present invention uses a graphic-text retrieval method to retrieve similar named entity recognition examples, provides enhanced knowledge for target sample recognition, and enhances entity recognition performance. Compared with the prior art, the method proposed in the present invention has good performance and robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 The figure is a schematic diagram of the overall process of an enhanced perceptual named entity recognition method based on a large model and hybrid experts of the present invention.
[0036] Figure 2 This is a recognition overall model diagram of an enhanced perception named entity recognition method based on a large model and mixed experts in the present invention (the model is a description of each module of the named entity recognition method proposed in the present invention, indicating the structure and interaction of each functional module included in the method) DETAILED DESCRIPTION
[0037] Below, reference Figure 1 to Figure 2 As shown, an enhanced perceptual named entity recognition method based on a large model and mixed experts includes:
[0038] S1, data augmentation task based on Wensheng graph model;
[0039] In order to enhance the effect of named entity recognition task, by introducing image information, the text information and the image information can be mutually verified, reducing the risk of information loss and misrecognition under a single modality; specifically, firstly, representative text and corresponding entities are artificially constructed to form text information; then, the text-to-graph model is used to generate corresponding images from the text information. At this time, the image information is highly correlated with the text information, thereby constructing high-quality image-text pairs as database D, which provides accurate reference for subsequent use;
[0040] S2, image and text retrieval task;
[0041] Image-text retrieval tasks can generally be performed using text-text retrieval, image-image retrieval, and image-text retrieval. Relatively speaking, retrieval that combines images and text will be more accurate.
[0042] Here, we use the similarity between images and texts to perform retrieval. Specifically, for the target image-text pair sample Target(T image ,T text ), respectively encode the image and text with the help of Vison Encoder and Language Encoder to obtain the image features and text features of the target sample. Then traverse each pair of image and text in the database j (d j_image ,d j_text ), encode its image information and text information with the same Vison Encoder and Language Encoder to obtain the corresponding image features and text features, and then calculate the cosine similarity between the sample image features and text features of the target image-text pair, and take the weighted average as the final image-text pair similarity. The cosine similarity calculation formula is designed as:
[0043] Simalarity j =λcosine(E i (T image ),E i (d j_image ))+cosine(E t (T text ),E t (d j_text ))
[0044] Among them: λ is a hyperparameter, generally λ∈R, a real number between 0 and 1, used to weigh the importance of the two modes, cosine represents the calculation of the cosine similarity between the two, E i Represents Vision Encoder, used to encode images, E t Represents a LanguageEncoder, used to encode text.
[0045] By searching the database, K image-text pairs with the highest similarity are selected and used as knowledge enhancement samples.
[0046] S3, multimodal information fusion task based on hybrid expert model;
[0047] In order to ensure that the features of each input text information and image information can be optimally processed to enhance the processing of text and image features, a hybrid expert model is used here. This multimodal information fusion model is called MoE crossattention. The input text and image features are processed through a cross-attention mechanism to obtain hybrid features, which are then passed through one or more linear layers of the router to obtain the expert scores, and then routed to each hybrid expert model for further processing.
[0048] Specifically, the latent vector h1 of the previous layer is used as the query (if it is in the first layer, the latent vector h1 is the text feature), and the image feature is used as the key and value, and input into the cross-attention mechanism to obtain a latent vector h2, which is concatenated with the latent vector h1 to obtain the latent vector h3, which is input into the router to obtain the expert scores, and then routed to each hybrid expert model for further processing to obtain the latent vector h for subsequent processing.
[0049] S4, Generative Named Entity Recognition Task Based on Large Model;
[0050] The K image-text pairs with the highest similarity retrieved from S2, i.e., the knowledge-enhanced samples, are processed separately according to the image information and the text information.
[0051] For image information, the K images in the knowledge enhancement sample and the target sample image are encoded by the Vision encoder respectively, and processed into a fixed dimension with the help of the perceiver network to obtain their respective image features, so that they can be input into the module of the large model, and the K image features and the target sample image features are spliced together according to the similarity sorting to obtain the final image features. Among them, the parameters of the Vision encoder are not updated during the training process, and the parameters of the perceiver network are updated.
[0052] For text information, you need to define instructions in advance, and then sort and concatenate the instructions and K texts in the knowledge enhancement sample and the target sample text according to the similarity, and pass the concatenated string through the Language encoder to obtain text features. Among them, the parameters of the Language encoder are updated during the training process; the instruction can be defined as: "Please identify the entities in the text and categorize the minto the following three categories: person; location; organization."; the concatenation of the K texts in the knowledge enhancement sample and the target sample text is: " text1 NE1; text2 NE2; ...; target_text", It is a special symbol used to represent images.
[0053] Add the multimodal information fusion model MoE cross attention in front of each decoding layer of the large model. Specifically, when entering the first layer, input the text features and image features obtained previously, use the text features as query, and the image features as key and value to input into the MoE cross attention of the first layer, and then input the output result into the decoding layer of the large model to obtain the latent vector, which is input into the next MoE-cross attention as query input, and the image features are still used as key and value, and so on. The latent vector output after multiple layers of MoE cross attention and decoding layers is input into the linear layer to obtain the predicted output, and the predicted output is decoded into a dictionary form, and the multimodal named entity recognition task is completed accordingly.
[0054] Among them: The prediction output formula is:
[0055]
[0056] Where: T(Instruction,d text ,T text ) represents the text information composed of instructions, K texts in knowledge retrieval enhancement, and target sample texts, I(d image ,T image ) represents the concatenation of K images in knowledge retrieval enhancement and the target sample image, θ represents the model parameters to be trained, y represents the label, Indicates the generated result.
[0057] The training goal is to minimize the language model loss L, and the formula is designed as follows:
[0058]
[0059] Among them, y represents the label, Indicates the final generation result of the model.
[0060] The above are only preferred embodiments of the present invention. It should be understood that the description of the above embodiments is only used to help understand the method and core ideas of the present invention, and is not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, etc. made within the ideas and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. An enhanced perceptual named entity recognition method based on a large model and mixed experts, characterized in that: The design has four task unit architectures, including: data augmentation task based on text-based graph model, image-text retrieval task, multimodal information fusion task based on hybrid expert model, and generative named entity recognition task based on large model, among which: The data augmentation task based on the Wensheng graph model: It is used to enhance the effect of named entity recognition tasks. By introducing image information, the text information and the image information can be mutually verified, reducing the risk of information loss and misrecognition under a single modality. The text-graph model is used to construct images that are more consistent with the text semantics and have less noise information, and the generated image-text pairs are used as a database. The image and text retrieval task: Perform feature calculation on the image information and text information of the currently input target image-text pair sample to obtain corresponding image features and text features, then traverse each image-text pair in the database, calculate the features of its image information and text information respectively, and then calculate the cosine similarity between the image features and text features of the target image-text pair sample and the target image-text pair sample respectively, and take the weighted average as the final image-text pair similarity; The multimodal information fusion task based on the hybrid expert model: It is used to ensure that the features of each input text information and image information can be optimally processed to enhance the processing of image and text features; the input image and text features are processed through the cross-attention mechanism to obtain mixed features, which are then passed through one or more linear layers of the router to obtain the expert scores, and then routed to each mixed expert model for further processing; The generative named entity recognition task based on the large model: First, for the knowledge-enhanced samples, the Vision Encoder and Language Encoder are used to encode them respectively, and the image information and text information of the samples are combined with the target image and text. The text information needs to be concatenated with the instructions before encoding. The image and text features obtained after encoding and combining are input into the multimodal information fusion model for processing. Then, the traditional named entity recognition classification task is improved into a generation task, and the multimodal information fusion task based on the hybrid expert model is combined with the decoding of the large model; for each decoding layer, the input image and text features are first processed by the third unit architecture, and then further processed by the decoding layer; Finally, entities are generated from the image and text features, and the model is trained with the goal of minimizing the language model loss L.
2. According to claim 1, the method for enhanced perceptual named entity recognition based on large models and mixed experts is characterized in that: Extracting high-level features from the mixture of the text information and image information can enhance the understanding of complex entities by the text graph model.
3. According to claim 1, the method for enhanced perceptual named entity recognition based on large models and mixed experts is characterized in that: The cosine similarity is designed as follows: Simularity j =λcosine(E i (T image ),AND i (d j_image ))+little things(And t (T text ),AND t (d j_text )) Among them: λ is a hyperparameter, generally λ∈R, a real number between 0 and 1, used to weigh the importance of the two modes, cosine represents the calculation of the cosine similarity between the two, E i Represents Vision Encoder, used to encode images, E t Represents a LanguageEncoder, used to encode text.
4. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 1, characterized in that: By searching the database, K image-text pairs with the highest similarity are extracted to improve the information completeness of relevant image-text features.
5. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 4, characterized in that: The K image-text pairs with the highest similarity are used as knowledge enhancement samples.
6. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 1, characterized in that: The multimodal information fusion task based on the hybrid expert model introduces a hybrid expert mechanism in the multimodal feature fusion stage. The hybrid features are distributed to each hybrid expert model according to the calculated scores through the router to expand the parameter scale and improve the robustness of the multimodal information fusion model, so that it can better cope with noise interference.
7. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 1, characterized in that: The image-text retrieval task and the multimodal information fusion task based on the hybrid expert model are both improved based on the multimodal model open-flamingo. The CLIP model is used to extract features of text information and image information pairs, and then similarity retrieval is performed with other image-text pairs in the database.
8. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 1, characterized in that: The introduction of the large model improves the output method of the traditional named entity recognition task. Through the generative output method, the generalization ability of the multimodal named entity recognition model on the unseen classes is optimized, and the knowledge base of the multimodal named entity recognition model is deepened.
9. The method for enhanced perceptual named entity recognition based on a large model and mixed experts according to claim 1, characterized in that: The formula for minimizing the language model loss L is designed as: Among them, y represents the label, Indicates the final generation result of the model.