Named entity recognition method based on multi-modal large model fine-grained knowledge generation

Abstract

The invention provides a named entity recognition method based on multi-modal large model fine-grained knowledge generation, and belongs to the technical field of information extraction in natural language processing. According to the method, a multi-modal large model MLLM and a large language model LLM are combined to jointly generate fine-grained auxiliary knowledge, the LLM uses a context learning mode to guide the LLM to generate auxiliary knowledge related to a sample, and the MLLM uses manual annotation data to perform fine tuning on the LLM, so that the MLLM outputs the related auxiliary knowledge according to input text and image information; and combining the obtained auxiliary knowledge with the original text, entering a downstream sequence labeling model for training and reasoning, and completing named entity recognition. According to the method, world knowledge and multi-modal information which are beneficial to information extraction are taken into consideration, the performance of named entity recognition of the downstream sequence marking model is improved, and the entities in the text can be recognized more accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of information extraction in natural language processing, and specifically relates to a named entity recognition technology based on a multimodal large model. Background Art

[0002] The named entity recognition (NER) task is a fundamental natural language processing (NLP) task and an important technology for constructing knowledge graphs. The task aims to identify entities with specific meanings from text and classify them into predefined categories, such as person names, locations, organizations, time expressions, quantity expressions, etc. The NER task can be divided into two main aspects: boundary recognition and entity classification. The goal of boundary recognition is to determine the starting and ending positions of entities in the text, while entity classification is to classify these recognized entities into predefined types.

[0003] Traditional NER tasks only focus on text. In recent years, with the rapid development of Internet multimedia information, especially the rise and popularity of social platforms such as Weibo and Twitter, a lot of corpus exists along with multimodal information such as pictures and videos. At the same time, due to the characteristics of social media, the text therein often has the characteristics of being short, lacking context, and having a lot of noise, which brings great difficulties to information extraction work such as entity recognition solely from the text modality. In some cases, the pictures corresponding to the text can be used as information supplements for the text to improve the accuracy of information extraction. The multimodal named entity recognition (MNER) task thus emerges. This task provides pairs of text and images, and the model can choose to utilize the information contained in the images to perform named entity recognition on the text.

[0004] Some previous work has focused on designing interaction mechanisms between images and text to integrate visual information into the text modality. Lu et al. encoded the entire picture and designed an attention mechanism to capture visual information related to the text. To avoid the negative impact of irrelevant visual objects on text reasoning, Chen et al. proposed a hierarchical visual fusion mechanism, introducing hierarchical multi-scale visual features into the self-attention layer of the language model. Yuan et al. encoded text and images into graph structures respectively and guided the alignment of visual objects and text entities through graph alignment. However, the differences in distributions between modalities limit the performance of the above methods.

[0005] Another type of method aids entity recognition in the model by introducing external knowledge. Wang et al. constructed a knowledge base using Wikipedia as the data source and adopted a multimodal retrieval method to retrieve knowledge associated with text and images respectively, improving the performance of entity recognition. However, due to reasons such as homonyms and ambiguities, it is impossible to guarantee that the knowledge retrieved by this method is relevant to the current example. In recent years, with the rapid development of pre-trained large language models (LLMs), many LLMs have demonstrated quite high reasoning and few-shot context learning capabilities, and relevant research has shown that LLMs have the ability to store knowledge and retrieve knowledge from it. Based on this, Li et al. proposed to generate external knowledge by finding similar samples as examples and utilizing the few-shot context learning ability of the large language model to assist the downstream model in named entity recognition. Although the above work of this type of method has achieved certain success, this type of method fails to make good use of the information in the visual modality. For example, in the work of Wang et al., the pictures are only used for retrieval, while Li et al. utilize the pictures by generating picture descriptions, which will inevitably cause loss of visual information. Therefore, it is of great significance to study a method that uses large language models and fully utilizes multimodal information in named entity recognition. Summary of the Invention

[0006] In view of the problems existing in the above prior art, the present invention proposes a method for combining a multimodal large model (MLLM) and a large language model (LLM) to jointly generate fine-grained auxiliary knowledge and guide the downstream sequence labeling model to complete named entity recognition. This method introduces a pre-trained multimodal large model, synthesizes text information and visual information to generate fine-grained auxiliary knowledge, and fully utilizes multimodal information in named entity recognition.

[0007] The technical solution of the present invention is as follows:

[0008] A named entity recognition method based on the generation of fine-grained knowledge of a multimodal large model, characterized by comprising the following steps:

[0009] (1) Select samples from the training set and manually annotate the knowledge to construct a set of manually annotated knowledge samples; use the multimodal named entity recognition (MNER) model to perform multimodal information fusion on all samples in the training set and the test set to obtain vector encodings;

[0010] (2) Based on the set of manually annotated knowledge samples, construct a supervised fine-tuning SFT dataset D SFT , and perform supervised fine-tuning on the multimodal large model (MLLM);

[0011] (3) For any sample in the training set and the test set without manually labeled knowledge, construct a prompt, and input the prompt and the image into the multi-modal large model MLLM fine-tuned in step (2) to obtain the auxiliary knowledge of this sample from MLLM;

[0012] (4) For any sample in the training set and the test set without manually labeled knowledge, generate auxiliary knowledge using a large language model (LLM). Specifically: Calculate the vector similarity between this sample and the manually labeled knowledge samples, and based on the calculation results, select the N manually labeled knowledge samples with the highest similarity for this sample to form a context learning example set; Construct a prompt according to the example set and input it into the LLM to output the auxiliary knowledge of this sample from the LLM;

[0013] (5) For any sample in the training set and the test set without manually labeled knowledge, splice the auxiliary knowledge output by the MLLM fine-tuned in step (3) and the auxiliary knowledge generated by the LLM in step (4) with the original text of this sample respectively and input them into the sequence labeling model; The sequence labeling model completes named entity recognition through a Transformer encoder and a conditional random field (CRF) layer.

[0014] Further, the manually labeled knowledge in step (1) includes: entities existing in the text; domain knowledge associated with the entities; analysis of the correspondence between image visual features and text entities.

[0015] Further, in step (2), the SFT dataset where P' represents the prompt input to the multi-modal large model, I is the image of the sample, Z is the manually labeled knowledge of the sample, and M represents the number of samples in the manually labeled sample set; The specific method of supervised fine-tuning is: construct a prompt containing the task description and instructions, and the input sample, and input it into the MLLM; Fine-tune the MLLM with the goal of minimizing the loss function, and the loss function is: where θ represents the parameters of the multi-modal large model.

[0016] Further, the vector similarity calculation in step (4) uses the cosine similarity method, and the formula is: where H1 and H2 represent the vectors encoded by the multi-modal named entity recognition model for two different samples.

[0017] Further, the LLM prompt constructed in step (4) includes:

[0018] Task description and instructions; Similar example set; Text and image information of the current input sample.

[0019] Further, the splicing method of the auxiliary knowledge and the original text in step (5) is: appending the knowledge sequence to the end of the text to form an extended input sequence.

[0020] Further, the training method of the sequence annotation model is as follows:

[0021] For each in the training set, the auxiliary knowledge generated by the LLM and the auxiliary knowledge generated by the MLLM are respectively spliced with the original text to construct multiple groups of training data, all of which are used for training the sequence annotation model; for the samples in the validation set and the test set, one of the knowledge generated by the LLM and the MLLM is selected for inference.

[0022] Further, the Transformer encoder encodes the spliced sequence to extract character-level features; the CRF layer predicts the probability distribution of the label sequence based on the encoded features to complete entity boundary recognition and classification.

[0023] The technical effects of the present invention are as follows:

[0024] A named entity recognition method based on fine-grained knowledge generation of a multi-modal large model according to the present invention comprehensively generates fine-grained auxiliary knowledge from a multi-modal large model and a large language model, taking into account both world knowledge and multi-modal information that are beneficial to information extraction, thereby further improving the performance of the downstream sequence tagging model for named entity recognition and being able to more accurately identify entities in the text. Through experiments, the test results on two public data sets show that this method has a significant performance improvement compared with existing methods. Description of the Drawings

[0025] Figure 1 is the overall flowchart of the named entity recognition method based on fine-grained knowledge generation of a multi-modal large model according to the present invention;

[0026] Figure 2 is a multi-modal named entity recognition example in a specific embodiment of the present invention. Detailed Embodiments

[0027] The present invention will be further clearly and completely described below with reference to the drawings through specific embodiments.

[0028] The overall framework diagram / flowchart of the present invention is as Figure 1As shown. First, the present invention randomly selects about 300 samples and performs manual annotation of knowledge. Subsequently, for: (1) large language models (LLMs), in the way of in-context learning, it guides them to generate auxiliary knowledge related to the samples; (2) multi-modal large models (MLLMs), it uses the manually annotated data to perform supervised fine-tuning on them, so that they can output relevant auxiliary knowledge according to the input text and image information. The obtained auxiliary knowledge is combined with the original text and enters the downstream sequence labeling model for training and inference. For knowledge from different sources, adopting the idea of data augmentation, they are jointly used as the training set to train the sequence labeling model, while the validation set and the test set only contain one of the sources.

[0029] The following combines the attached Figure 1 , elaborates on the named entity recognition method based on the generation of fine-grained knowledge of multi-modal large models, and the steps for named entity recognition of this text:

[0030] (1) First, randomly select a small part of samples from the training set. Based on the original text-image pairs, manually annotate relevant knowledge to obtain a sample set of manually annotated knowledge Among them, T i is the text of the i-th sample, I i is the image of the i-th sample, Z i is the manually annotated knowledge of the i-th sample, and M represents the number of samples in the sample set of manually annotated knowledge. The knowledge includes: (1) entities existing in the text; (2) relevant domain knowledge associated with the entities; (3) analysis of the correspondence between the visual features of the image and the text entities. Then, use an existing multi-modal named entity recognition (MNER) model to complete the encoding of all samples in the training set and the test set, that is, the original text-image pairs undergo multi-modal information fusion through this model, and finally obtain the fused vector encoding: H = MNER(T, I), where T is the text of the sample and I is the image of the sample.

[0031] (2) Construct a supervised fine-tuning (SFT) data set according to the sample set of manually annotated knowledge obtained in step (1) Among them, P' i represents the prompt words input to the multi-modal large model, including: task description and instructions, input samples. Minimize the following loss function as the goal to fine-tune the multi-modal large model: where θ represents the parameters of the multi-modal large model.

[0032] (3) For any sample (T, I) in the training set and the test set that has not been manually annotated with knowledge, construct a prompt P' and input the image I into the multi-modal large model MLLM fine-tuned in step (2) to obtain the auxiliary knowledge of this sample from MLLM: Z MLLM = MLLM(P', I). Compared with existing methods, this method innovatively uses a multi-modal large model to integrate the visual information of the picture into the generated auxiliary knowledge.

[0033] (4) For any sample (T, I) in the training set and the test set that has not been manually annotated with knowledge, calculate the cosine similarity between vectors with all the samples that have been manually annotated with knowledge in step (1). The calculation formula is: where H1 and H2 represent the vectors encoded by the multi-modal named entity recognition model for two different samples. According to the calculation results, select the N samples of manually annotated knowledge with the highest similarity for this sample to form the context learning example set E for this sample. Construct a prompt P for this sample to be input into the large language model, including: (a) task description and instructions; (b) similar example set; (c) current input sample. Take P as the input of the large language model LLM and output the auxiliary knowledge of this sample from LLM: Z LLM = LLM(P) = {z1, … z m} where m represents the length of the auxiliary knowledge.

[0034] (5) For any sample (T, I) in the training set and the test set that has not been manually annotated with knowledge, assume that a certain piece of auxiliary knowledge generated by LLM or MLLM in steps (3) and (4) is Z = {z1, …, z m} where m represents the length of this auxiliary knowledge. Concatenate it with the original text in the sample to get [T; Z], and then input it into the Transformer encoder (Transformer-based Encoder) to obtain the encoded output of each position character: h1, …, h n , …, h n+m = embed([T; Z]), where n represents the length of the original text T. Then, take the encoded outputs h1, … h n corresponding to the characters of the original text T and send them into the conditional random field (CRF) layer to predict the probability distribution of the label sequence: where represents the label sequence, y iThe tag representing the i-th character position contains the named entity classification and boundary information at that location. For each sample in the training set, all the multiple auxiliary knowledge generated by multiple independent LLMs and MLLMs in steps (3) and (4) is used for training; for the samples in the validation set and the test set, one piece of the auxiliary knowledge generated by the LLMs and MLLMs is selected for inference.

[0035] Through the above steps, the present invention comprehensively utilizes large language models and multimodal large models to achieve multi-modal named entity recognition assisted by fine-grained knowledge. Compared with the existing methods that only utilize large language models, the present invention first introduces multimodal large models in this task to directly utilize the visual information contained in images, so as to more accurately achieve multi-modal named entity recognition.

[0036] Figure 2 As an example of multi-modal named entity recognition, using the method of the present invention, multi-modal named entity recognition is performed on the sample pictures and texts in Figure 2 to obtain the results: Leonardo DiCaprio (person's name), Kate Moss (person's name), New York City (place name). When tested on the public datasets Twitter-2015 and Twitter-2017 datasets, the accuracies of 78.09% and 90.63% are respectively achieved, which are 1.03% and 0.83% higher than the existing methods.

[0037] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention. However, those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection claimed by the present invention is subject to the scope defined by the claims.

Claims

1. A named entity recognition method based on fine-grained knowledge generation of multi-modal large models, characterized in that, It includes the following steps: (1) Select samples from the training set and manually annotate the knowledge to construct a set of manually annotated knowledge samples; use the multi-modal named entity recognition (MNER) model to perform multi-modal information fusion on all samples in the training set and the test set to obtain vector encodings; (2) Construct a supervised fine-tuning SFT dataset D based on the set of manually annotated knowledge samples SFT and perform supervised fine-tuning on the multi-modal large model MLLM; (3) For any sample in the training set and the test set without manually annotated knowledge, construct a prompt, and input the prompt and the image into the multi-modal large language model (MLLM) fine-tuned in step (2) to obtain the auxiliary knowledge of this sample from the MLLM; (4) For any sample in the training set and the test set without manually annotated knowledge, use the large language model (LLM) to generate auxiliary knowledge. Specifically, calculate the vector similarity between this sample and the manually annotated knowledge samples, and according to the calculation result, select the N manually annotated knowledge samples with the highest similarity for this sample to form a context learning example set; construct a prompt according to the example set and input it into the LLM to output the auxiliary knowledge of this sample from the LLM; (5) For any sample in the training set and the test set without manually annotated knowledge, splice the auxiliary knowledge output by the MLLM fine-tuned in step (3) and the auxiliary knowledge generated by the LLM in step (4) with the original text of this sample respectively and input them into the sequence labeling model; the sequence labeling model completes named entity recognition through a Transformer encoder and a conditional random field (CRF) layer.

2. The method according to claim 1, characterized in that The manually annotated knowledge in step (1) includes: entities existing in the text; domain knowledge associated with the entities; analysis of the correspondence between image visual features and text entities.

3. The method according to claim 1, wherein In the said step (2), the SFT dataset where P' represents the prompt input to the multi-modal large model, I is the image of the sample, Z is the manually annotated knowledge of the sample, and M represents the number of samples in the set of manually annotated samples; the specific method of supervised fine-tuning is: construct a prompt containing the task description and instructions, the input sample, and input it into the MLLM; fine-tune the MLLM with the goal of minimizing the loss function, and the loss function is: where θ represents the parameters of the multi-modal large model.

4. The method according to claim 1, wherein The vector similarity calculation in the step (4) adopts the cosine similarity method, and the formula is as follows: where H1 and H2 represent the vectors encoded by the multi-modal named entity recognition model for two different samples.

5. The method according to claim 1, wherein The LLM prompts constructed in step (4) include: Task description and instructions; similar example set; text and image information of the current input sample.

6. The method according to claim 1, characterized in that, The splicing method of the auxiliary knowledge and the original text in step (5) is: append the knowledge sequence to the end of the text to form an extended input sequence.

7. The method according to claim 1, wherein The training method of the sequence labeling model is: For each in the training set, splice the auxiliary knowledge generated by the LLM and the auxiliary knowledge generated by the MLLM with the original text respectively to construct multiple groups of training data, all of which are used for training the sequence labeling model; for the samples in the validation set and the test set, select one of the knowledge generated by the LLM and the MLLM for inference.

8. The method according to claim 1, wherein The Transformer encoder encodes the spliced sequence to extract character-level features; the conditional random field (CRF) layer predicts the probability distribution of the label sequence based on the encoded features to complete entity boundary recognition and classification.

Images