Multi-modal named entity recognition method based on multi-modal large model context learning
Through the context learning method of multimodal large model, the dependence of multimodal named entity recognition on labeled data and model fine-tuning in a small sample scenario is solved, and efficient multimodal information utilization and recognition performance is achieved, which is suitable for social media analysis and public opinion monitoring.
Patent Information
- Application Number
- CN202510409149.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-02
- Publication Date
- 2025-07-29
AI Technical Summary
The existing multimodal named entity recognition technology requires a large amount of labeled data and model fine-tuning in a small sample scenario, and there is information loss during the conversion of image information into text, resulting in limited recognition performance.
The context learning method based on multimodal large model is adopted to calculate text and image similarity, build a context prompt template, and input a multimodal large model for inference, so as to achieve efficient multimodal information utilization without large-scale annotation data and model fine-tuning.
It significantly reduces the demand for labeled data volume, improves the generalization of the model and the feasibility of practical applications, reduces information loss during modal conversion, and improves the accuracy and stability of multimodal named entity recognition.
Smart Images
Figure CN120387450A_ABST
Abstract
Description
Technical Field
[0001] It relates to the technology of multimodal information fusion and named entity recognition under few-shot conditions, and specifically relates to a multimodal named entity recognition method based on in-context learning of a multimodal large model. Background Art
[0002] Multimodal Named Entity Recognition (MNER) is a task of recognizing named entities in text by simultaneously using multiple information modalities such as text and images. In recent years, with the explosive growth of multimodal data such as social media and e-commerce platforms, MNER technology has gradually become an important research direction in the field of natural language processing because it can effectively combine information from different modalities and improve the accuracy of entity recognition, and has been widely applied in many fields such as information extraction, knowledge graph construction, and intelligent interaction systems.
[0003] The existing research on MNER technology mainly focuses on better fusing text and image information through different cross-modal interaction mechanisms. For example, the UMT model proposes to achieve deep interaction between text and visual information through a unified multimodal Transformer architecture to improve entity recognition performance; UMGF designs a unified multimodal graph structure to help identify and annotate entities by capturing the semantic relationships between text tokens and visual objects. In addition, the HVPNeT model constructs a hierarchical visual prefix network to effectively fuse visual features and text information, improving the effect of vision-enhanced entity recognition.
[0004] Although the above methods have improved the effect of MNER to a certain extent, these methods usually rely on a large amount of manually annotated data and require large-scale parameter fine-tuning of the model to obtain better performance, which causes great limitations in practical applications, especially in few-shot scenarios. At the same time, in order to adapt to existing text models, many methods often directly convert image information into text through OCR or image description generation technology. This approach inevitably leads to the loss of image information, thus reducing the effective utilization of multimodal information by the model and the accuracy of recognition. For example, although the ITA method avoids the difficulty of direct cross-modal feature alignment, it sacrifices the richness and diversity of the original visual features and fails to achieve the best recognition effect.
[0005] In summary, the main technical problems existing in the existing multimodal named entity recognition technology include: (1) a large amount of labeled data and large-scale model fine-tuning are required, and it is not applicable to few-shot scenarios; (2) key content is lost during the conversion of image information into text, resulting in insufficient multimodal information fusion and limiting the improvement of recognition performance.
[0006] Therefore, there is an urgent need for a new multi-modal named entity recognition technology that can efficiently utilize multi-modal information without large-scale labeled data and model fine-tuning, especially showing excellent recognition effects in few-shot learning scenarios, to overcome the above technical problems. Summary of the Invention
[0007] To solve the technical defects in the prior art, where there is an urgent need for a new multi-modal named entity recognition technology that can efficiently utilize multi-modal information without large-scale labeled data and model fine-tuning, especially showing excellent recognition effects in few-shot learning scenarios, the technical solution provided by the present invention is as follows:
[0008] A multi-modal named entity recognition method based on in-context learning of a multi-modal large model, including:
[0009] Steps of calculating the text similarity and image similarity between the query to be recognized and each case in the candidate case pool respectively, and obtaining the text similarity ranking and image similarity ranking results;
[0010] Steps of performing fusion processing on the text similarity ranking and image similarity ranking results to obtain a comprehensive ranking result, and selecting the top k most relevant cases as context examples from it;
[0011] Steps of constructing a context prompt template including task description, entity type definition, and context examples based on the context examples;
[0012] Steps of forming a complete input prompt with the context prompt template and the case to be recognized, inputting it into the multi-modal large model for in-context reasoning, and obtaining the feedback of the large model as the named entity recognition result of the case to be recognized.
[0013] Further, a preferred implementation is provided. The steps of calculating the text similarity and image similarity specifically include:
[0014] Using a pre-trained text encoding model to perform embedding encoding on the query text of the case to be recognized and the texts in the candidate case pool respectively, and then calculating the text similarity;
[0015] Using a pre-trained image encoding model to perform embedding encoding on the query image of the case to be recognized and the images in the candidate case pool respectively, and then calculating the image similarity.
[0016] Further, a preferred implementation is provided. The cosine similarity is used for both the calculation of the text similarity and the image similarity.
[0017] Further, a preferred embodiment is provided, where the fusion process specifically includes: summing the text similarity ranking numbers and the image similarity ranking numbers of each case in the candidate case pool, and using the ranking after summation as the overall ranking of the multimodal examples to obtain the comprehensive ranking result.
[0018] Further, a preferred embodiment is provided, where the definitions of the four entity types in the context prompt template include person, organization, location, and others.
[0019] Further, a preferred embodiment is provided, where the inference result of the multimodal large model is output in a structured JSON format.
[0020] Based on the same inventive concept, the present invention also provides a multimodal named entity recognition device based on context learning of a multimodal large model, including:
[0021] A module for calculating the text similarity and image similarity between the case to be recognized and each case in the candidate case pool respectively, and obtaining the text similarity ranking and the image similarity ranking results;
[0022] A module for fusing the text similarity ranking and the image similarity ranking results to obtain a comprehensive ranking result, and selecting the top k most relevant cases as context examples from it;
[0023] A module for constructing a context prompt template including a task description, entity type definitions, and context examples based on the context examples;
[0024] A module for forming a complete input prompt with the context prompt template and the case to be recognized, inputting it into the multimodal large model for context inference, and obtaining the named entity recognition result of the case to be recognized.
[0025] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computing program, and when the computer program is read by a computer, the computer executes the method.
[0026] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium, and when the processor reads the computer program stored in the storage medium, the computer executes the method.
[0027] Based on the same inventive concept, the present invention also provides a computer program product, which is a computer program, and when the computer program is executed, the method is implemented.
[0028] Compared with the prior art, the beneficial effects of the technical solution provided by the present invention are as follows:
[0029] A context learning method based on a multimodal large model is adopted, without the need for model fine-tuning. That is, through a carefully designed context prompt construction strategy, the powerful context awareness and learning capabilities of the large model are effectively utilized. Compared with traditional methods (such as UMT, UMGF, etc.) that rely on a large amount of supervised data and model fine-tuning, this method significantly reduces the demand for the amount of labeled data under few-shot conditions, effectively improving the generalization of the model and the feasibility of practical applications.
[0030] By calculating the similarity ranking of text and image separately and combining the two for comprehensive ranking, the adaptive selection of context examples is realized. This strategy of fusing text and image similarity significantly reduces the information loss in the modality conversion process compared with traditional methods (such as ITA, etc.) that rely only on the single mode of converting images to text, and more comprehensively and accurately retains and utilizes the original information of images and text, ultimately significantly improving the accuracy and stability of multimodal named entity recognition.
[0031] Experimental verification shows that on mainstream datasets such as Twitter-15 and Twitter-17, this embodiment has achieved the best performance indicators superior to existing methods, fully verifying the effectiveness and robustness of the proposed context learning and example selection strategies in multimodal few-shot scenarios. Compared with few-shot learning methods such as ProtoBERT and StructShot, this solution can quickly achieve excellent performance without additional training, significantly improving the model's fast adaptation ability to new fields or new entity categories and enhancing the model's applicability in actual complex scenarios.
[0032] It can be widely applied to multimodal text understanding related work such as social media analysis, public opinion monitoring, and intelligent question answering. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 It is a system diagram of the multimodal named entity recognition method;
[0034] Figure 2 It is a schematic diagram of the prompt template for context examples;
[0035] Figure 3 It is a schematic diagram of the inference template;
[0036] Figure 4 It is a schematic diagram of the experimental results;
[0037] Figure 5 It is a schematic diagram of template construction. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0038] To make the advantages and beneficial effects of the technical solutions provided by the present invention more clearly manifested, the technical solutions provided by the present invention are further described in detail below with reference to the accompanying drawings. Specifically:
[0039] Embodiment 1. This embodiment provides a multimodal named entity recognition method based on in-context learning of a multimodal large model, including:
[0040] The step of calculating the text similarity and image similarity between the query to be recognized and each case in the candidate case pool respectively, and obtaining the text similarity ranking and image similarity ranking results;
[0041] The step of performing fusion processing on the text similarity ranking and image similarity ranking results to obtain a comprehensive ranking result, and selecting the top k most relevant cases as context examples from it;
[0042] The step of constructing a context prompt template including task description, entity type definition, and context examples based on the context examples;
[0043] The step of forming a complete input prompt with the context prompt template and the case to be recognized, inputting it into the multimodal large model for context reasoning, and obtaining the feedback of the large model as the named entity recognition result of the case to be recognized.
[0044] The step of calculating the text similarity and image similarity respectively specifically includes:
[0045] Using a pre-trained text encoding model to perform embedding encoding on the query text of the case to be recognized and the texts in the candidate case pool respectively, and then calculating the text similarity;
[0046] Using a pre-trained image encoding model to perform embedding encoding on the query image of the case to be recognized and the images in the candidate case pool respectively, and then calculating the image similarity.
[0047] The cosine similarity is used for both the calculation of the text similarity and the image similarity.
[0048] The fusion processing is specifically: summing the text similarity ranking serial numbers and image similarity ranking serial numbers of each case in the candidate case pool, and using the ranking after summation as the overall ranking of the multimodal examples to obtain the comprehensive ranking result.
[0049] The four entity type definitions in the context prompt template include Person (PER), Organization (ORG), Location (LOC), and Other Entities (MISC).
[0050] The inference result of the multimodal large model is output in a structured JSON format.
[0051] The JSON format is specifically defined as:
[0052] {'PER': [List of person entities], 'ORG': [List of organization entities], 'LOC': [List of location entities], 'MISC': [List of other entities]}.
[0053] Embodiment 2. This embodiment further explains the above-provided technical solution. Specifically:
[0054] This embodiment provides a multi-modal named entity recognition method based on in-context learning of a multi-modal large model. Without the need for a large amount of labeled data and model fine-tuning, it uses the in-context learning paradigm and an adaptive in-context example selection strategy to achieve high-precision multi-modal entity recognition under few-shot conditions. The specific implementation steps are as follows:
[0055] First, briefly, the method retrieves multi-modal in-context examples, selects the examples most relevant to the case to be recognized to construct an in-context prompt; then uses the in-context prompt to construct a complete input prompt template; finally, provides the input prompt template to the multi-modal large model for reasoning to obtain the named entity recognition result of the case to be recognized.
[0056] The specific detailed implementation is as follows:
[0057] Step 1. Multi-modal in-context example retrieval:
[0058] To construct an effective in-context learning input, this embodiment designs an in-context example retrieval strategy based on multi-modal information. It specifically includes three sub-steps: text similarity ranking, image similarity ranking, and comprehensive ranking.
[0059] (1) Text similarity ranking
[0060] First, obtain the query text of the case to be recognized, use a pre-trained text encoding model (such as ALL-MPNet-base-v2) to encode the query text and all texts in the candidate case pool to obtain text embedding representation vectors. Then, calculate the similarity scores between the query text embedding and each case text embedding in the case pool based on cosine similarity, and sort them from high to low according to similarity to obtain the text similarity ranking result.
[0061] (2) Image similarity ranking
[0062] Meanwhile, for the query image of the case to be recognized, use a pre-trained image encoding model (such as CLIP-ViT-Large-patch14) to perform feature encoding on the query image and all images in the candidate case pool to obtain image embedding representation vectors. Then, use cosine similarity to calculate the similarity scores between the query image embedding and each image embedding in the case pool respectively, and sort them from high to low based on the similarity to obtain the image similarity sorting result.
[0063] (3) Comprehensive sorting
[0064] Then, fuse the ranking positions of the above-obtained text similarity sorting and image similarity sorting, that is, after adding the ranking numbers of the two modalities respectively, re-sort them from low to high according to the total ranking number to obtain the comprehensive similarity sorting result. The top k cases in the comprehensive sorting are selected as the most relevant multi-modal context examples for the subsequent prompt construction step.
[0065] Step 2: Construction of context prompt template:
[0066] Based on the multi-modal context examples obtained in the above steps, this step constructs a complete context prompt template to effectively activate the context learning ability of the multi-modal large model. The context prompt template includes a task description (TaskDescription), a definition of the entity types to be recognized, and the retrieved top-k context examples.
[0067] First, construct the task description part. The task description clarifies the goals and requirements of the current multi-modal named entity recognition task, that is, to identify the named entities existing in the text-image combination cases, and clearly gives the types of entities to be recognized (such as PER, ORG, LOC, MISC).
[0068] Secondly, according to the standard definition of the multi-modal entity recognition task, clearly list and define the meanings of different categories of entities to help the multi-modal large model accurately understand the recognition target.
[0069] Finally, add the top-k context examples obtained from the comprehensive sorting in Step 1 to the context prompt template in the form of "example - entity list" in turn. Each example is expressed in a unified form, clearly showing the corresponding case text, image features, and the entities labeled in each entity category, forming a complete context prompt template.
[0070] Step 3: Query inference based on the multi-modal large model:
[0071] Based on the complete context prompt template constructed in the above steps, form the final input by combining the case to be recognized with the context prompt template in the same format as the context examples, and send it into the multi-modal large model for context learning and inference processing.
[0072] In specific implementations, the input form of the case to be recognized is kept completely consistent with the context example format, so that the model can accurately understand the relationship between the context and the current query case and the task requirements. In addition, to ensure the consistency and parsability of the output results of the multimodal large model, the output format of the model is strictly limited to a predefined structured JSON form, for example:
[0073] {'PER': [list of entities], 'ORG': [list of entities], 'LOC': [list of entities], 'MISC': [list of entities]}
[0074] Based on the above predefined structure, the multimodal large model infers and judges the input case, and finally outputs JSON-structured data containing the specific recognition results of various types of entities, forming the final named entity recognition result of the current query case.
[0075] Through the above steps, this implementation scheme can effectively complete the multimodal named entity recognition task in the few-shot scenario without large-scale manually labeled data and complex model fine-tuning, and achieve better recognition performance and generalization ability than existing traditional methods.
[0076] In the above technical solution, a context learning method based on a multimodal large model is adopted without model fine-tuning. Compared with the traditional method that relies on a large amount of supervised data and model fine-tuning, the requirement for the amount of labeled data is significantly reduced under the few-shot condition, effectively improving the generalization of the model and the feasibility of practical applications.
[0077] By calculating the similarity rankings of the text and the image separately and combining the two for comprehensive ranking, adaptive context example selection is realized. Compared with the traditional single mode that only relies on converting the image to text, the information loss in the modality conversion process is significantly reduced, and the original information of the image and the text is retained and utilized more comprehensively and accurately, ultimately significantly improving the accuracy and stability of multimodal named entity recognition.
[0078] Embodiment 3. Combination Figures 1-4 To illustrate this embodiment, this embodiment further describes the above-provided technical solution in detail through specific embodiments. Specifically:
[0079] With the popularity of multi-modal data on social media platforms, multi-modal named entity recognition (MNER) has become an important research direction. MNER helps the NER model identify entities more accurately by taking images as auxiliary inputs. Previous studies have tried to introduce visual information into the NER task and adopted various cross-modal attention mechanisms to promote the interaction between text and images. However, it is pointed out that the performance limitations of such methods mainly stem from the distribution differences between different modalities. Although existing technologies have tried to alleviate this problem by further optimizing the cross-modal attention mechanism, end-to-end training of cross-modal Transformer architectures requires a large amount of computing resources, posing challenges to the practical application of the model. ITA and MoRe adopted a new paradigm to solve the MNER task. ITA avoided the problem of multi-modal alignment by giving up the direct use of original visual features and instead choosing OCR and image caption generation technologies to transmit image information. MoRe assisted prediction by retrieving additional knowledge related to text and images from an explicit knowledge base. However, none of these methods can fully meet the knowledge requirements for the model to understand text. UMT uses a transformer layer with a multi-modal interaction module to capture the modal dynamics between tokens and images in MNER; UMGF adopts a unified multi-modal graph method to capture the semantic relationships between tokens and visual objects and perform entity tagging; HVPNeT uses a hierarchical visual prefix fusion network for visually enhanced entities;
[0080] DebiasCL proposed an MNER method based on de-biased contrastive learning and studied the modal consistency enhanced by cross-modal contrastive learning. For few-shot learning methods, ProtoBERT uses a prototype network with a BERT encoder backbone; StructShot uses token-level nearest neighbor classification and structured reasoning. Although they have achieved certain results, they usually rely on a large amount of annotated data, which is insufficient for generalizing the ability to localize and identify entities to other entity categories.
[0081] In recent years, with the rise of large language models, these models are pre-trained on large-scale corpora and demonstrate powerful context learning capabilities by aligning with human preferences, providing new ideas for solving the above problems.
[0082] Combined with Figure 1 ;
[0083] In multi-modal context example retrieval work:
[0084] Text similarity ranking
[0085] In multi-modal named entity recognition, the input includes text and images. We perform similarity retrieval and ranking on the two modalities separately. The main purpose of this step is to find case texts in the candidate case pool that are similar to the text of the query case for subsequent selection of context examples.
[0086] 1. Given a candidate case pool D containing N text-image example pairs, we use a pre-trained language model, such as ALL-MPNet, to encode the query text and case texts. The encoding process is as follows:
[0087] H query = MPNet(T query )
[0088] H D = MPNet(D t )
[0089] Among them, H query and H D represent the embedding representations of the query text T query and the case text D t respectively;
[0090] 2. Then, use cosine to calculate the similarity scores between the query text and the texts in the case pool;
[0091] S t = Cosine(H query , H D )
[0092] The similarity ranking of the texts in the case pool and the query text is obtained according to the scores.
[0093] R t = rank(S t )
[0094] Image similarity ranking
[0095] The main purpose of this step is to find case texts in the candidate case pool that are similar to the image of the query case for subsequent selection of context examples.
[0096] 1. In this embodiment, a pre-trained image encoder model, such as CLIP, is used to encode the query image and case images. The encoding process is as follows:
[0097] V query = CLIP(V query )
[0098] V D = VLIP(D v )
[0099] Among them, Vquery and V D respectively represent the embedded representations of the query image V query and the case image D v ;
[0100] 2. Then use cosine to calculate the similarity scores between the query image and the images in the case pool;
[0101] S v = Cosine(V query , V D )
[0102] 3. Finally, obtain the similarity ranking of the images in the case pool and the query image according to the scores.
[0103] R v = rank(S v )
[0104] Comprehensive ranking
[0105] In multi-modal named entity recognition, the input includes text and images. According to the similarity ranking results of the image and text modalities Rv and Rt, add the two rankings and sort them to obtain the final ranking result, and select the top k cases as the context examples of the large model.
[0106] R = rank(R t + R v )
[0107] σ = Top-K(R)
[0108] The context learning paradigm consists of two parts: task instruction construction and example prompt construction. In this implementation, the definition of the multi-modal command entity recognition task is used as the instruction to help the multi-modal large model understand the current task. At the same time, the entity type localization and examples in the dataset are used as prompts. Finally, the top-k example results in the multi-modal context example retrieval are used as example prompts to form a complete context, as Figure 2 shown.
[0109] Query inference
[0110] The format of the query is consistent with the format of the context examples, which ensures that both the query and the examples follow the same structure, facilitating seamless integration and comparison during the context learning process. In addition, although the entity types are specified, the output format may vary. To ensure the consistency of the output format, this implementation uses the JSON format and clearly defines the output structure as {'PER': [list of existing entities], 'ORG': [list of existing entities], 'LOC': [list of existing entities], 'MISC': [list of existing entities]}. The inference template is asFigure 3 as shown
[0111] Finally, the complete input prompt is input to the multi-modal large model to obtain its feedback as the result of the task.
[0112] Experimental settings
[0113] Datasets: This embodiment is evaluated on two benchmark MNER datasets, including Twitter-15 and Twitter-17. The Twitter-15 dataset contains 4,000 training samples and 3,357 test samples. The Twitter-17 dataset consists of 3,373 training samples and 723 test samples.
[0114] Meanwhile, for few-shot MNER related experiments, this embodiment follows the settings of existing research and divides the dataset into D 10 , D 50 , D 100 , D all which contain 10, 50, 100 and all training samples respectively.
[0115] Experimental settings: During the example retrieval process, this embodiment uses All-MPNet-base-v2 to calculate text similarity; for image similarity and self-correlation calculation, this embodiment adopts CLIP-ViT-Large-patch14. In the inference stage, this embodiment uses InternVL2, specifically the 8B version, which is one of the open-source models with the best performance on the OpenCompass leaderboard. All experiments are run on an NVIDIA A100 GPU with 40GB of memory.
[0116] Figure 4 The experimental results shown above demonstrate the effectiveness of the method of this embodiment.
[0117] Experimental verification shows that on mainstream datasets such as Twitter-15 and Twitter-17, this embodiment achieves the best performance metrics superior to existing methods, fully verifying the effectiveness and robustness of the proposed context learning and example selection strategies in multi-modal few-shot scenarios. Compared with few-shot learning methods such as ProtoBERT and StructShot, this solution can quickly achieve excellent performance without additional training, significantly improving the model's ability to quickly adapt to new domains or new entity categories and enhancing the model's applicability in actual complex scenarios.
[0118] As Figure 5As shown, context example retrieval is performed and used to construct a template, and a complete prompt is constructed, including the task description, entity type definition, retrieved context examples, and the query itself. The constructed prompt is used to apply a multimodal large language model (MLLM) to obtain feedback and prediction results without additional training.
[0119] The above further describes the technical solutions provided by the present invention through several specific embodiments to highlight the advantages and benefits of the technical solutions provided by the present invention. However, the above several specific embodiments are not used as limitations to the present invention. Any reasonable modifications and improvements to the present invention, combinations of implementation manners, equivalent replacements, etc. within the spirit and principle of the present invention should be included within the protection scope of the present invention.
Claims
1. A multimodal named entity recognition method based on in-context learning of a multimodal large model, characterized in that, Including: Steps of respectively calculating the text similarity and image similarity between the query to be recognized and each case in the candidate case pool, and obtaining the text similarity ranking and image similarity ranking results; Steps of performing fusion processing on the text similarity ranking and image similarity ranking results to obtain a comprehensive ranking result, and selecting the top k most relevant cases therefrom as context examples; Steps of constructing a context prompt template including a task description, entity type definitions, and context examples based on the context examples; Steps of forming a complete input prompt with the context prompt template and the case to be recognized, inputting it into a multi-modal large model for context reasoning, and obtaining the feedback of the large model as the named entity recognition result of the case to be recognized.
2. The multimodal named entity recognition method based on in-context learning of a multimodal large model according to claim 1, wherein, The steps of respectively calculating the text similarity and image similarity specifically include: Using a pre-trained text encoding model to perform embedding encoding on the query text of the case to be recognized and the texts in the candidate case pool respectively, and then calculating the text similarity; Using a pre-trained image encoding model to perform embedding encoding on the query image of the case to be recognized and the images in the candidate case pool respectively, and then calculating the image similarity.
3. A multimodal named entity recognition method based on in-context learning of a multimodal large model according to claim 1, characterized in that, The cosine similarity is used for both the calculation of the text similarity and the image similarity.
4. A multimodal named entity recognition method based on in-context learning of a multimodal large model according to claim 1, characterized in that, The fusion processing is specifically: summing the text similarity ranking serial numbers and image similarity ranking serial numbers of each case in the candidate case pool, and using the ranking after summation as the overall ranking of the multi-modal examples to obtain a comprehensive ranking result.
5. A multimodal named entity recognition method based on in-context learning of a multimodal large model according to claim 1, characterized in that The four entity type definitions in the context prompt template include: person, organization, location, and others.
6. A multimodal named entity recognition method based on in-context learning of a multimodal large model according to claim 1, characterized in that, The inference result of the multi-modal large model is output in a structured JSON format.
7. A multimodal named entity recognition device based on in-context learning of multimodal large models, characterized in that, Including: A module for respectively calculating the text similarity and image similarity between the case to be recognized and each case in the candidate case pool, and obtaining the text similarity ranking and image similarity ranking results; A module for performing fusion processing on the text similarity ranking and image similarity ranking results to obtain a comprehensive ranking result, and selecting the top k most relevant cases therefrom as context examples; A module for constructing a context prompt template including a task description, entity type definitions, and context examples based on the context examples; Forming a complete input prompt with the context prompt template and the case to be recognized, inputting it into a multi-modal large model for context reasoning, and obtaining the feedback of the large model as the named entity recognition result of the case to be recognized.
8. A computer storage medium for storing a computing program, characterized in that, When the computer program is read by a computer, the computer executes the method described in claim 1.
9. A computer, comprising a processor and a storage medium, characterized in that, When the processor reads the computer program stored in the storage medium, the computer executes the method described in claim 1.
10. A computer program product, as a computer program, characterized in that, When the computer program is executed, the method described in claim 1 is implemented.